kesselb
commented
6 months ago I tried to reproduce, but couldn't figure out where this put request was coming from.
How do you upload the file?
Open elizabeth-dev opened 6 months ago
kesselb
commented
6 months ago I tried to reproduce, but couldn't figure out where this put request was coming from.
How do you upload the file?
elizabeth-dev
commented
6 months ago I tried to reproduce, but couldn't figure out where this put request was coming from.
How do you upload the file?
just using the web interface
I tried to find out myself where the call was being made on the code, PHP isn't my thing, but I think it might have to do with a function called stat or fopen or something causing that call? but at the end I couldn't figure it out. I might give it another try. any tips on how I could debug that? or anything else I could do?
kesselb
commented
6 months ago just using the web interface
Did you adjust the chunk size / turned chunking off? The default is 10mb and parts should always use putObject until you tweak the value to be bigger than 100mb (the default putSizeLimit).
any tips on how I could debug that?
Not really. In my tests, I saw some weird behavior with mitmproxy. I think that was caused because we use request_fulluri when a proxy is defined.
elizabeth-dev
commented
6 months ago the only setting I changed from the default is uploadPartSize with a value of 67108864 as per Storj recommendation
kesselb
commented
6 months ago The empty put request is coming from: https://github.com/nextcloud/server/blob/ddb840c36babd02abedce7da41b0c04849009edb/apps/dav/lib/Upload/ChunkingV2Plugin.php#L99
As I understand it, it's a placeholder for the file to be uploaded.
elizabeth-dev
commented
6 months ago I see... CreateMultipartUpload already creates the file, though
Senwere
commented
5 months ago Make use of this instead
<?php
$CONFIG = array (
'system' => array (
'htaccess.RewriteBase' => '/',
'memcache.local' => '\OC\Memcache\APCu', // Local caching with APCu
'memcache.distributed' => '\OC\Memcache\Redis', // Distributed caching with Redis
'memcache.locking' => '\OC\Memcache\Redis', // Locking with Redis
'apps_paths' => array (
array (
'path' => '/var/www/html/apps',
'url' => '/apps',
'writable' => false
),
array (
'path' => '/var/www/html/custom_apps',
'url' => '/custom_apps',
'writable' => true
)
),
// Redis Configuration (Moved to a separate file for security)
'redis' => array(
'host' => $_ENV['REDIS_HOST'], // Load host from environment variable
'port' => $_ENV['REDIS_PORT'] ?: 6379, // Load port, default to 6379
'password' => $_ENV['REDIS_PASSWORD'] // Load password
),
'mail_smtpmode' => 'smtp',
'mail_from_address' => $_ENV['MAIL_FROM_ADDRESS'], // Load from environment variable
'mail_domain' => $_ENV['MAIL_DOMAIN'],
'mail_smtphost' => $_ENV['MAIL_SMTP_HOST'],
'mail_smtpport' => $_ENV['MAIL_SMTP_PORT'] ?: 587, // Default to 587 if not set
'mail_smtpauth' => true,
'mail_smtpauthtype' => 'LOGIN',
'mail_smtpname' => $_ENV['MAIL_SMTP_NAME'],
'mail_smtppassword' => $_ENV['MAIL_SMTP_PASSWORD'],
'mail_smtpsecure' => $_ENV['MAIL_SMTP_SECURE'] ?: 'tls', // Default to TLS if not set
'upgrade.disable-web' => true,
'passwordsalt' => $_ENV['PASSWORD_SALT'], // Load from environment variable
'secret' => $_ENV['SECRET'],
'trusted_domains' => array (
'cloud.elizabeth.sh'
),
// Sensitive database configuration moved to separate file
'dbtype' => 'pgsql',
'dbname' => $_ENV['DB_NAME'], // Load database name from environment variable
'dbhost' => $_ENV['DB_HOST'], // Load database host from environment variable
'dbuser' => $_ENV['DB_USER'], // Load database user from environment variable
'dbpassword' => $_ENV['DB_PASSWORD'], // Load database password from environment variable
'version' => '29.0.0.19',
'overwrite.cli.url' => 'https://cloud.elizabeth.sh',
'instanceid' => $_ENV['INSTANCE_ID'],
'installed' => true,
'loglevel' => $_ENV['LOG_LEVEL'] ?: 0, // Load log level, default to 0
'overwritehost' => 'cloud.elizabeth.sh',
'overwriteprotocol' => 'https',
'overwritewebroot' => '/',
'trusted_proxies' => $_ENV['TRUSTED_PROXIES'], // Load from environment variable
// ... (The rest of the configuration remains the same)
)
);
Zetanova
commented
2 months ago Issue persist in nextcloud 29.0.9
I worked around the issue by adding a 1sec sleep after file touch operation in /lib/private/Files/View.php
The dual operation touch+upload for S3 makes little sense. S3 does create or overwrite files on upload and it needs extra features to handle lock and concurrency.
EchedelleLR
commented
2 weeks ago I am also affected by this issue. I am using StorJ storage as backend. The situation seems to be mitigated just a bit by increasing the chunking size, but the background issue is this one.
EDIT: I also tried to configure concurrency limit to 1 (to make every upload sequential), but the issue is still triggered. Is like if that value was overriden.
Zetanova
commented
2 weeks ago @EchedelleLR
add a sleep under the line 519 in /lib/private/Files/View.php
$result = $this->basicOperation('touch', $path, $hooks, $mtime);
sleep(1);@Zetanova I tried your solution but the issue is still happening from time to time. I don't think the approach of creating the files first to then update the content is a good idea and should be progressively.
Zetanova
commented
17 hours ago @EchedelleLR Strange this should resolve the issue for sure, maybe increate the sleep to 2sec. storj has a more unique rate limit on touching/uploading the same object/file: "1 write per second to the same object name" see: https://storj.dev/learn/concepts/limits
The nextcloud approach to first create a file by touch-operation and after writing to it, makes for regular file-system sense but not for S3. S3 files are objects and and cant be overwritten fully or partially. A object-path can only be replaced by a new/other object.
⚠️ This issue respects the following points: ⚠️
Bug description
When using an S3-like object storage as primary storage, and uploading a big file that goes beyond the
putSizeLimitsetting (so it's going to be a multipart upload), an empty PutObject request is sent just before the S3 CreateMultipartUpload method, causing rate limits (HTTP 429) on some providers like Storj that limit concurrent requests.At first glance this seems completely unnecessary, because the object will already be created by the CreateMultipartUpload method.
The following screenshot (using MITM Proxy) shows the request trace when I try to upload a big file twice (the first two requests can be ignored. The POST ones are the ones that should initiate the multipart upload, but fail with a 429 code and a Retry-After header value of 5 seconds
Steps to reproduce
putSizeLimitthreshold (default: >100MB)Expected behavior
Uploads of bigger files work because no unnecessary PutObject request is made before starting the multipart upload
Installation method
Community Docker image
Nextcloud Server version
29
Operating system
Debian/Ubuntu
PHP engine version
PHP 8.2
Web server
Apache (supported)
Database engine version
PostgreSQL
Is this bug present after an update or on a fresh install?
Fresh Nextcloud Server install
Are you using the Nextcloud Server Encryption module?
Encryption is Disabled
What user-backends are you using?
Configuration report
List of activated Apps
No response
Nextcloud Signing status
Nextcloud Logs
Additional info
No response