Closed kerichdev closed 1 month ago
I tested with a 1gb file, and although my upload speeds aren't fast it did upload the file in the end to my s3 bucket. Not sure what's going on here.. You can try to view network activity on htop or something like that to see if zipline is actually uploading the files. Unfortunately zipline doesn't really provide useful logs when it uploads to s3 but what I said previously should be a good indicator.
I did monitor the activity, and while it was uploading the chunks (to the server), the end file never gets to the S3 bucket - it frees up the space taken by the chunks without any upload activity. The 1 Gb might also be a working threshold, I only confirmed it to not work with 2 Gb files so far.
Interesting... Did you use AWS S3 or something else? I tested with AWS.
I used Yandex Cloud, it's AWS-compliant, and seems to work just fine with everything under 400Mb, and possibly above, until 2Gb.
Can you provide specs on how you are using and what the specs are of the system is hosting Zipline?
Sure, the specs are: Ryzen 9 7950X3D (2 threads only) 2.5Gb DDR5 35Gb NVMe 1Gbps bandwidth
I use it only for an nginx server and a few other tiny things, along with Zipline with an S3 bucket.
2.5Gb DDR5
This is probably your problem. As far as I'm aware, Zipline can only upload files that the system can fully load into memory. For a system with that amount of memory, your limit would probably be somewhere between 1.5 Gb and 2 Gb, depending on what else is running on the system.
I'm going to close this issue, but I will say that I'm working on a way in v4 where it should just directly upload multipart uploads using s3's MultipartUpload stuff, either way like @SeaswimmerTheFsh the 2.5GB memory might be the problem here. You can try to change your temp directory to somewhere other than /tmp
since /tmp
will be in ram.
What happened?
While using S3 as a datasource, and chunking on cloudflare's free plan, certain size files never end up on the S3 bucket after upload, and create ghost database entries with 0B size.
The tested WORKING max size so far is 400mb. Anything beyond is (probably) bound to break. The tested NOT working size is 2gb.
Version
latest (ghcr.io/diced/zipline or ghcr.io/diced/zipline:latest)
What browser(s) are you seeing the problem on?
Chromium-based (Chrome, Edge, Brave, Opera, mobile chrome/chromium based, etc)
Zipline Logs
Browser Logs
No response
Additional Info
The errors in the logs happen only after the file upload, even though it does claim to start the server. Maybe it crashes?