Bug: S3 + chunking uploads stop working past a certain file size threshold

kerichdev commented 5 months ago

What happened?

While using S3 as a datasource, and chunking on cloudflare's free plan, certain size files never end up on the S3 bucket after upload, and create ghost database entries with 0B size.

The tested WORKING max size so far is 400mb. Anything beyond is (probably) bound to break. The tested NOT working size is 2gb.

Version

latest (ghcr.io/diced/zipline or ghcr.io/diced/zipline:latest)

What browser(s) are you seeing the problem on?

Chromium-based (Chrome, Edge, Brave, Opera, mobile chrome/chromium based, etc)

Zipline Logs

zipline_1   | 2024-03-31 11:12:07,434 AM info  [datasource] using S3(kericloudstorage) datasource
zipline_1   | 2024-03-31 11:12:07,441 AM debug [server] Starting server
zipline_1   | 2024-03-31 11:12:07,450 AM error [server] Found temporary files in Zipline's temp directory.
zipline_1   | 2024-03-31 11:12:07,450 AM error [server] This can happen if Zipline crashes or is stopped while chunking a file.
zipline_1   | 2024-03-31 11:12:07,451 AM error [server] If you are sure that no files are currently being processed, you can delete the files in the temp directory.
zipline_1   | 2024-03-31 11:12:07,451 AM error [server] The temp directory is located at: /tmp/zipline
zipline_1   | 2024-03-31 11:12:07,451 AM error [server] If you are unsure, you can safely ignore this message.

Browser Logs

No response

Additional Info

The errors in the logs happen only after the file upload, even though it does claim to start the server. Maybe it crashes?

diced commented 4 months ago

I tested with a 1gb file, and although my upload speeds aren't fast it did upload the file in the end to my s3 bucket. Not sure what's going on here.. You can try to view network activity on htop or something like that to see if zipline is actually uploading the files. Unfortunately zipline doesn't really provide useful logs when it uploads to s3 but what I said previously should be a good indicator.

kerichdev commented 4 months ago

I did monitor the activity, and while it was uploading the chunks (to the server), the end file never gets to the S3 bucket - it frees up the space taken by the chunks without any upload activity. The 1 Gb might also be a working threshold, I only confirmed it to not work with 2 Gb files so far.

diced commented 4 months ago

Interesting... Did you use AWS S3 or something else? I tested with AWS.

kerichdev commented 4 months ago

I used Yandex Cloud, it's AWS-compliant, and seems to work just fine with everything under 400Mb, and possibly above, until 2Gb.

kashalls commented 4 months ago

Can you provide specs on how you are using and what the specs are of the system is hosting Zipline?

kerichdev commented 4 months ago

Sure, the specs are: Ryzen 9 7950X3D (2 threads only) 2.5Gb DDR5 35Gb NVMe 1Gbps bandwidth

I use it only for an nginx server and a few other tiny things, along with Zipline with an S3 bucket.

cswimr commented 2 months ago

2.5Gb DDR5

This is probably your problem. As far as I'm aware, Zipline can only upload files that the system can fully load into memory. For a system with that amount of memory, your limit would probably be somewhere between 1.5 Gb and 2 Gb, depending on what else is running on the system.

diced commented 1 month ago

I'm going to close this issue, but I will say that I'm working on a way in v4 where it should just directly upload multipart uploads using s3's MultipartUpload stuff, either way like @SeaswimmerTheFsh the 2.5GB memory might be the problem here. You can try to change your temp directory to somewhere other than /tmp since /tmp will be in ram.

diced / zipline