search

e-alfred / flowupload

Flowupload app compatible with Nextcloud
GNU Affero General Public License v3.0
61 stars 20 forks source link

Unable to adjust chunkSize and simultaneousUploads #243

Closed AdamVenn closed 3 years ago

AdamVenn commented 3 years ago

First, thank you for the invaluable and unique contribution to Nextcloud.

I am unable to modify the chunkSize and simultaneousUploads parameters by following the instructions in the readme. This may be an omission in the readme, something wrong with the src files or my lack of knowledge in Javascript/Node.js.

Steps to reproduce

Expected behaviour

The readme said the speed would increase!

Actual behaviour

Many errors!

gyp ERR! configure error 
gyp ERR! stack Error: EACCES: permission denied, mkdir '/var/www/nextcloud/apps/flowupload/node_modules/fsevents/build'
gyp ERR! System Linux 5.4.0-54-generic
gyp ERR! command "/usr/bin/node" "/usr/bin/node-gyp" "rebuild"
gyp ERR! cwd /var/www/nextcloud/apps/flowupload/node_modules/fsevents
gyp ERR! node -v v10.19.0
gyp ERR! node-gyp -v v6.1.0
gyp ERR! not ok

> iltorb@2.4.5 install /var/www/nextcloud/apps/flowupload/node_modules/iltorb
> node ./scripts/install.js || node-gyp rebuild

internal/modules/cjs/loader.js:638
    throw err;
    ^

Error: Cannot find module '/var/www/nextcloud/apps/flowupload/node_modules/iltorb/scripts/install.js'
    at Function.Module._resolveFilename (internal/modules/cjs/loader.js:636:15)
    at Function.Module._load (internal/modules/cjs/loader.js:562:25)
    at Function.Module.runMain (internal/modules/cjs/loader.js:831:12)
    at startup (internal/bootstrap/node.js:283:19)
    at bootstrapNodeJSCore (internal/bootstrap/node.js:623:3)
gyp ERR! configure error 
gyp ERR! stack Error: EACCES: permission denied, mkdir '/var/www/nextcloud/apps/flowupload/node_modules/iltorb/build'
gyp ERR! System Linux 5.4.0-54-generic
gyp ERR! command "/usr/bin/node" "/usr/bin/node-gyp" "rebuild"
gyp ERR! cwd /var/www/nextcloud/apps/flowupload/node_modules/iltorb
gyp ERR! node -v v10.19.0
gyp ERR! node-gyp -v v6.1.0
gyp ERR! not ok 
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! iltorb@2.4.5 install: `node ./scripts/install.js || node-gyp rebuild`
npm ERR! Exit status 1
npm ERR! 
npm ERR! Failed at the iltorb@2.4.5 install script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     /root/.npm/_logs/2020-11-26T21_05_57_633Z-debug.log

Server configuration

Operating system: Ubuntu 20.04.1 LTS

Web server: Apache/2.4.46

Database: mysql Ver 15.1 Distrib 10.3.25-MariaDB, for debian-linux-gnu (x86_64) using readline 5.2

PHP version: Version: 7.4.3, Memory Limit: 512 MB, Max Execution Time: 3600, Upload max size: 700 GB

Nextcloud version: 19.0.5 (Latest stable at time of writing)

Updated from an older Nextcloud/ownCloud or fresh install: Updated from 19.0.4 today!

Where did you install Nextcloud from: wget https://download.nextcloud.com/server/releases/nextcloud-18.0.4.tar.bz2

Signing status:

Signing status No errors have been found.

List of activated apps:

App list Enabled: - activity: 2.12.1 - admin_audit: 1.9.0 - bruteforcesettings: 2.0.1 - cloud_federation_api: 1.2.0 - contactsinteraction: 1.0.0 - dav: 1.15.0 - documentserver_community: 0.1.8 - extract: 1.2.5 - federatedfilesharing: 1.9.0 - files: 1.14.0 - files_downloadactivity: 1.8.0 - files_external: 1.10.0 - files_pdfviewer: 1.8.0 - files_rightclick: 0.16.0 - files_sharing: 1.11.0 - files_trashbin: 1.9.0 - files_videoplayer: 1.8.0 - flowupload: 1.1.2 - group_everyone: 0.1.6 - logreader: 2.4.0 - lookup_server_connector: 1.7.0 - metadata: 0.12.0 - oauth2: 1.7.0 - onlyoffice: 6.1.0 - password_policy: 1.9.1 - privacy: 1.3.0 - provisioning_api: 1.9.0 - serverinfo: 1.9.0 - settings: 1.1.0 - sharebymail: 1.9.0 - text: 3.0.1 - theming: 1.10.0 - twofactor_backupcodes: 1.8.0 - updatenotification: 1.9.0 - user_ldap: 1.9.1 - viewer: 1.3.0 - workflowengine: 2.1.0 Disabled: - accessibility - comments - encryption - federation - files_versions - firstrunwizard - nextcloud_announcements - notifications - photos - recommendations - support - survey_client - systemtags

Nextcloud configuration:

Config report ``` { "system": { "instanceid": "***REMOVED SENSITIVE VALUE***", "passwordsalt": "***REMOVED SENSITIVE VALUE***", "secret": "***REMOVED SENSITIVE VALUE***", "trusted_domains": [ "***REMOVED SENSITIVE VALUE***", "***REMOVED SENSITIVE VALUE***" ], "datadirectory": "***REMOVED SENSITIVE VALUE***", "dbtype": "mysql", "version": "19.0.5.2", "overwrite.cli.url": "***REMOVED SENSITIVE VALUE***", "htaccess.RewriteBase": "\/", "dbname": "***REMOVED SENSITIVE VALUE***", "dbhost": "***REMOVED SENSITIVE VALUE***", "dbport": "", "dbtableprefix": "oc_", "mysql.utf8mb4": true, "dbuser": "***REMOVED SENSITIVE VALUE***", "dbpassword": "***REMOVED SENSITIVE VALUE***", "installed": true, "memcache.local": "\\OC\\Memcache\\APCu", "ldapIgnoreNamingRules": false, "ldapProviderFactory": "OCA\\User_LDAP\\LDAPProviderFactory", "log_type": "syslog", "logfile": "", "loglevel": 3, "log.condition": { "apps": [ "admin_audit" ] }, "trashbin_retention_obligation": "1, 2", "ldapUserCleanupInterval": "30", "maintenance": false, "theme": "", "app_install_overwrite": [ "files_frommail" ], "updater.secret": "***REMOVED SENSITIVE VALUE***" }, "apps": { "accessibility": { "enabled": "no", "installed_version": "1.4.0", "types": "" }, "activity": { "enabled": "yes", "installed_version": "2.12.1", "types": "filesystem" }, "admin_audit": { "enabled": "yes", "installed_version": "1.9.0", "types": "logging" }, "backgroundjob": { "lastjob": "3651" }, "bruteforcesettings": { "enabled": "yes", "installed_version": "2.0.1", "types": "" }, "calendar": { "enabled": "no", "installed_version": "2.0.3", "types": "" }, "cloud_federation_api": { "enabled": "yes", "installed_version": "1.2.0", "types": "filesystem" }, "comments": { "enabled": "no", "installed_version": "1.8.0", "types": "logging" }, "contactsinteraction": { "enabled": "yes", "installed_version": "1.0.0", "types": "dav" }, "core": { "backgroundjobs_mode": "ajax", "installed.bundles": "[\"CoreBundle\"]", "installedat": "1589901242.7554", "lastcron": "1606425363", "lastupdateResult": "[]", "lastupdatedat": "1606423576", "oc.integritycheck.checker": "[]", "public_files": "files_sharing\/public.php", "public_webdav": "dav\/appinfo\/v1\/publicwebdav.php", "theming.variables": "b494f9fe4e205ef1e2a61df9ff5f797d", "updater.secret.created": "1606413618", "vendor": "nextcloud" }, "dav": { "buildCalendarReminderIndex": "yes", "buildCalendarSearchIndex": "yes", "chunks_migrated": "1", "enabled": "yes", "generateBirthdayCalendar": "no", "installed_version": "1.15.0", "regeneratedBirthdayCalendarsForYearFix": "yes", "sendEventReminders": "no", "sendInvitations": "no", "types": "filesystem" }, "documentserver_community": { "enabled": "yes", "installed_version": "0.1.8", "types": "filesystem" }, "extract": { "enabled": "yes", "installed_version": "1.2.5", "types": "" }, "federatedfilesharing": { "enabled": "yes", "installed_version": "1.9.0", "types": "" }, "federation": { "enabled": "no", "installed_version": "1.8.0", "types": "authentication" }, "files": { "cronjob_scan_files": "500", "enabled": "yes", "installed_version": "1.14.0", "types": "filesystem" }, "files_antivirus": { "av_cmd_options": "", "av_host": "", "av_infected_action": "delete", "av_max_file_size": "-1", "av_mode": "socket", "av_path": "\/usr\/bin\/clamscan", "av_port": "0", "av_socket": "\/var\/run\/clamav\/clamd.ctl", "av_stream_max_length": "26214400", "enabled": "no", "installed_version": "2.4.1", "types": "filesystem,dav" }, "files_downloadactivity": { "enabled": "yes", "installed_version": "1.8.0", "types": "filesystem" }, "files_external": { "enabled": "yes", "installed_version": "1.10.0", "types": "filesystem" }, "files_pdfviewer": { "enabled": "yes", "installed_version": "1.8.0", "types": "" }, "files_rightclick": { "enabled": "yes", "installed_version": "0.16.0", "types": "" }, "files_sharing": { "enabled": "yes", "incoming_server2server_share_enabled": "no", "installed_version": "1.11.0", "lookupServerEnabled": "no", "lookupServerUploadEnabled": "no", "outgoing_server2server_share_enabled": "yes", "types": "filesystem" }, "files_trashbin": { "enabled": "yes", "installed_version": "1.9.0", "types": "filesystem,dav" }, "files_versions": { "enabled": "no", "installed_version": "1.11.0", "types": "filesystem,dav" }, "files_videoplayer": { "enabled": "yes", "installed_version": "1.8.0", "types": "" }, "firstrunwizard": { "enabled": "no", "installed_version": "2.7.0", "types": "logging" }, "flowupload": { "enabled": "yes", "installed_version": "1.1.2", "types": "" }, "group_everyone": { "enabled": "yes", "installed_version": "0.1.6", "types": "authentication" }, "logreader": { "enabled": "yes", "installed_version": "2.4.0", "types": "" }, "lookup_server_connector": { "enabled": "yes", "installed_version": "1.7.0", "types": "authentication" }, "metadata": { "enabled": "yes", "installed_version": "0.12.0", "types": "" }, "nextcloud_announcements": { "enabled": "no", "installed_version": "1.7.0", "types": "logging" }, "notifications": { "enabled": "no", "installed_version": "2.6.0", "types": "logging" }, "oauth2": { "enabled": "yes", "installed_version": "1.7.0", "types": "authentication" }, "onlyoffice": { "DocumentServerUrl": "***REMOVED SENSITIVE VALUE***/index.php\/apps\/documentserver_community\/", "defFormats": "{\"docx\":true,\"pptx\":true,\"xlsx\":true,\"odp\":true,\"ods\":true,\"odt\":true,\"doc\":true,\"ppt\":true,\"xls\":true}", "editFormats": "{\"csv\":true,\"docx\":true,\"pptx\":true,\"txt\":true,\"xlsx\":true,\"odp\":true,\"ods\":true,\"odt\":true,\"rtf\":true}", "enabled": "yes", "installed_version": "6.1.0", "sameTab": "true", "types": "filesystem" }, "password_policy": { "enabled": "yes", "enforceHaveIBeenPwned": "1", "installed_version": "1.9.1", "types": "authentication" }, "photos": { "enabled": "no", "installed_version": "1.0.0", "types": "" }, "privacy": { "enabled": "yes", "installed_version": "1.3.0", "types": "" }, "provisioning_api": { "enabled": "yes", "installed_version": "1.9.0", "types": "prevent_group_restriction" }, "recommendations": { "enabled": "no", "installed_version": "0.6.0", "types": "" }, "serverinfo": { "enabled": "yes", "installed_version": "1.9.0", "types": "" }, "settings": { "enabled": "yes", "installed_version": "1.1.0", "types": "" }, "sharebymail": { "enabled": "yes", "installed_version": "1.9.0", "types": "filesystem" }, "support": { "enabled": "no", "installed_version": "1.1.0", "types": "session" }, "survey_client": { "enabled": "no", "installed_version": "1.6.0", "types": "" }, "systemtags": { "enabled": "no", "installed_version": "1.8.0", "types": "logging" }, "text": { "enabled": "yes", "installed_version": "3.0.1", "types": "dav" }, "theming": { "backgroundMime": "image\/png", "cachebuster": "7", "color": "#002F7F", "enabled": "yes", "faviconMime": "image\/png", "installed_version": "1.10.0", "logoMime": "image\/png", "name": "***REMOVED SENSITIVE VALUE***", "slogan": "***REMOVED SENSITIVE VALUE***", "types": "logging", "url": "***REMOVED SENSITIVE VALUE***" }, "twofactor_backupcodes": { "enabled": "yes", "installed_version": "1.8.0", "types": "" }, "updatenotification": { "bruteforcesettings": "2.0.1", "core": "19.0.2.2", "documentserver_community": "0.1.7", "enabled": "yes", "extract": "1.2.5", "installed_version": "1.9.0", "metadata": "0.12.0", "onlyoffice": "6.0.0", "types": "", "update_check_errors": "0" }, "user_ldap": { "cleanUpJobOffset": "0", "enabled": "yes", "installed_version": "1.9.1", "s01_lastChange": "1606413302", "s01has_memberof_filter_support": "1", "s01home_folder_naming_rule": "", "s01last_jpegPhoto_lookup": "0", "s01ldap_agent_password": "***REMOVED SENSITIVE VALUE***", "s01ldap_attributes_for_group_search": "", "s01ldap_attributes_for_user_search": "", "s01ldap_backup_host": "", "s01ldap_backup_port": "", "s01ldap_base": "***REMOVED SENSITIVE VALUE***", "s01ldap_base_groups": "***REMOVED SENSITIVE VALUE***", "s01ldap_base_users": "***REMOVED SENSITIVE VALUE***", "s01ldap_cache_ttl": "600", "s01ldap_configuration_active": "1", "s01ldap_default_ppolicy_dn": "", "s01ldap_display_name": "displayname", "s01ldap_dn": "***REMOVED SENSITIVE VALUE***", "s01ldap_dynamic_group_member_url": "", "s01ldap_email_attr": "", "s01ldap_experienced_admin": "0", "s01ldap_expert_username_attr": "", "s01ldap_expert_uuid_group_attr": "", "s01ldap_expert_uuid_user_attr": "", "s01ldap_ext_storage_home_attribute": "", "s01ldap_gid_number": "gidNumber", "s01ldap_group_display_name": "cn", "s01ldap_group_filter": "***REMOVED SENSITIVE VALUE***", "s01ldap_group_filter_mode": "0", "s01ldap_group_member_assoc_attribute": "member", "s01ldap_groupfilter_groups": "***REMOVED SENSITIVE VALUE***", "s01ldap_groupfilter_objectclass": "group", "s01ldap_host": "***REMOVED SENSITIVE VALUE***", "s01ldap_login_filter": "***REMOVED SENSITIVE VALUE***", "s01ldap_login_filter_mode": "0", "s01ldap_loginfilter_attributes": "", "s01ldap_loginfilter_email": "0", "s01ldap_loginfilter_username": "1", "s01ldap_nested_groups": "0", "s01ldap_override_main_server": "", "s01ldap_paging_size": "500", "s01ldap_port": "389", "s01ldap_quota_attr": "", "s01ldap_quota_def": "", "s01ldap_tls": "0", "s01ldap_turn_off_cert_check": "0", "s01ldap_turn_on_pwd_change": "0", "s01ldap_user_avatar_rule": "default", "s01ldap_user_display_name_2": "", "s01ldap_user_filter_mode": "0", "s01ldap_userfilter_groups": "***REMOVED SENSITIVE VALUE***", "s01ldap_userfilter_objectclass": "person", "s01ldap_userlist_filter": "***REMOVED SENSITIVE VALUE***", "s01use_memberof_to_detect_membership": "1", "types": "authentication" }, "viewer": { "enabled": "yes", "installed_version": "1.3.0", "types": "" }, "workflowengine": { "enabled": "yes", "installed_version": "2.1.0", "types": "filesystem" } } } ```

Are you using external storage, if yes which one: local, but have not yet tried uploading to it using Flow Upload

Are you using encryption: no

Are you using an external user-backend, if yes which one: LDAP, connecting to a Samba 4 AD

LDAP configuration (delete this part if not used)

LDAP config ``` +-------------------------------+-------------+ | Configuration | s01 | +-------------------------------+-------------+ | hasMemberOfFilterSupport | 1 | | homeFolderNamingRule | | | lastJpegPhotoLookup | 0 | | ldapAgentName | *** | | ldapAgentPassword | *** | | ldapAttributesForGroupSearch | | | ldapAttributesForUserSearch | | | ldapBackupHost | | | ldapBackupPort | | | ldapBase | *** | | ldapBaseGroups | *** | | ldapBaseUsers | *** | | ldapCacheTTL | 600 | | ldapConfigurationActive | 1 | | ldapDefaultPPolicyDN | | | ldapDynamicGroupMemberURL | | | ldapEmailAttribute | | | ldapExperiencedAdmin | 0 | | ldapExpertUUIDGroupAttr | | | ldapExpertUUIDUserAttr | | | ldapExpertUsernameAttr | | | ldapExtStorageHomeAttribute | | | ldapGidNumber | gidNumber | | ldapGroupDisplayName | cn | | ldapGroupFilter | *** | | ldapGroupFilterGroups | *** | | ldapGroupFilterMode | 0 | | ldapGroupFilterObjectclass | group | | ldapGroupMemberAssocAttr | member | | ldapHost | *** | | ldapIgnoreNamingRules | | | ldapLoginFilter | *** | | ldapLoginFilterAttributes | | | ldapLoginFilterEmail | 0 | | ldapLoginFilterMode | 0 | | ldapLoginFilterUsername | 1 | | ldapMatchingRuleInChainState | unknown | | ldapNestedGroups | 0 | | ldapOverrideMainServer | | | ldapPagingSize | 500 | | ldapPort | 389 | | ldapQuotaAttribute | | | ldapQuotaDefault | | | ldapTLS | 0 | | ldapUserAvatarRule | default | | ldapUserDisplayName | displayname | | ldapUserDisplayName2 | | | ldapUserFilter | *** | | ldapUserFilterGroups | *** | | ldapUserFilterMode | 0 | | ldapUserFilterObjectclass | person | | ldapUuidGroupAttribute | auto | | ldapUuidUserAttribute | auto | | turnOffCertCheck | 0 | | turnOnPasswordChange | 0 | | useMemberOfToDetectMembership | 1 | +-------------------------------+-------------+ ```

Client configuration

Browser: Firefox 83

Operating system: Pop OS

Logs

Web server error log

Web server error log ``` Available on request :) ```

Nextcloud log (data/nextcloud.log)

Nextcloud log ``` Available on request :) ```

Browser log

Browser log ``` Available on request :) ```
JonathanTreffler commented 3 years ago

This is very likely a permissions issue. Please try running npm ci and npm run build with sudo or login with a root user (with sudo su for example)

AdamVenn commented 3 years ago

Thanks for the reply! I'm sorry, I should have mentioned that all commands were run as root (using sudo su).

AdamVenn commented 3 years ago

Does anyone have any ideas about this?

JonathanTreffler commented 3 years ago

So sorry, i completely forgot about this issue. What is the output of "node -v" ? Older versions of nodejs often cause problems.

AdamVenn commented 3 years ago

No worries, any help appreciated! node -v v10.19.0

JonathanTreffler commented 3 years ago

Error: EACCES: permission denied, mkdir '/var/www/nextcloud/apps/flowupload/node_modules/iltorb/build'

I think thats the concrete error.

JonathanTreffler commented 3 years ago

But i am not sure why root wouldn't have permission to create a folder.

AdamVenn commented 3 years ago

Yeah I was wondering that too. Could it be that it is/was in use? Already exists?

tlinna commented 3 years ago

I bumped into quite similar problem also with Ubuntu Server 20.04 LTS. It seems that running sudo su --login, which provides environment as if you were just logged in as root, fixes the problem with permissions. Something wrong with the existing environment with plain su, I reckon? (Disclaimer: I'm not that expert of a linux user)

The default chunk size seems too low to me, the browser consumes way way too much CPU with speeds of only a few megabytes per second even making the flowupload UI sluggish. Upped chunk size to 10 megabytes and everything works much faster with reasonable CPU usage.

AdamVenn commented 3 years ago

Oh wow, thank you for the tip! I will try it out and post the results.

AdamVenn commented 3 years ago

It worked! Thank you @tlinna! I did some testing, balancing the chunkSize, PHP max_upload, max_post_size, and memory_limit and got to a good place - around 60-70MB/s over LAN. Faster than Cyberduck! However, I found a new problem - near the end of the upload, a request is made which exceeds the chunkSize. So I could not set my chunkSize too close to my max_post_size because then this larger request would go over the limit and get an error.

Should I post this as a separate bug or is this expected behaviour?

JonathanTreffler commented 3 years ago

I found a new problem - near the end of the upload, a request is made which exceeds the chunkSize. So I could not set my chunkSize too close to my max_post_size because then this larger request would go over the limit and get an error.

Should I post this as a separate bug or is this expected behavior?

This is kind of expected behavior. It is the standard way the Flow.js library we use for the uploads handles the situation, as you can see in their documentation. But there is a solution. You can set the "forceChunkSize" variable in the Flow.js condiguration to true.

forceChunkSize Force all chunks to be less or equal than chunkSize. Otherwise, the last chunk will be greater than or equal to chunkSize. (Default: false)

Maybe we should add that to the Flow.js configuration by default. In testing we didn't modify the chunkSize configuration parameter, so this issue never appeared. I don't know why Flow.js opted for this weird default behavior.

You seem to be the first to have this issue, but it is definitely not you fault :) You can certainly create a issue where we can discuss this further.

JonathanTreffler commented 3 years ago

It seems that running sudo su --login, which provides environment as if you were just logged in as root, fixes the problem with permissions

Thank you very much for sharing your solution 👍

JonathanTreffler commented 3 years ago

The default chunk size seems too low to me, the browser consumes way way too much CPU with speeds of only a few megabytes per second even making the flowupload UI sluggish. Upped chunk size to 10 megabytes and everything works much faster with reasonable CPU usage.

I agree, because especially the high client side cpu usage is a problem, but i don't know what a good default value would be. I am a bit worried that increasing the values too much would hurt the user experience of users on a slow server, like a Raspberry.

AdamVenn commented 3 years ago

But there is a solution. You can set the "forceChunkSize" variable in the Flow.js condiguration to true.

I will do that! Thanks very much! I will leave it to you to consider the default behaviour for your app, I don't think I have much more to say on the matter!

AdamVenn commented 3 years ago

I agree, because especially the high client side cpu usage is a problem, but i don't know what a good default value would be. I am a bit worried that increasing the values too much would hurt the user experience of users on a slow server, like a Raspberry.

I noticed that the normal file uploader within Nextcloud goes in 10MB chunks.