Download of pictures fails, if the zip file is too large

[loulou91]

The download of large number of pictures looks limited by the size of the zip file (about 100-140Mo according to my trials). Below this threshold zip file is correctly downloaded, above nothing happens. I tried to reproduce it with "demo" base but there is not enough pictures in it !!!

In addition, no progress indication is provided. In my case for a file of about 100Mo it takes about 30s to be downloaded.

[derneuere]

The function that needs to be worked on is the following: https://github.com/LibrePhotos/librephotos/blob/de428fc8408f7fce50c7f3266eff482f88719429/api/views/views.py#L1545-L1574

I would guess, that this is a timeout issue.

[marcobehler]

Seconding the bug. The download image POST gets cancelled after 30 secs. Is there a quick way to increase that timeout?

[derneuere]

Sadly there is no env value for it. It's defined here:

https://github.com/LibrePhotos/librephotos-frontend/blob/3124224cc199054a9fdc55d629d133a2e4b9c820/src/api_client/apiClient.js#L23-L30

[dotanm]

Can this value be set as higher? I would say it needs to be at least 4 times higher, for full album downloads to succeed.

It's needed in case a user wants to save photos to a local folder for uploading to another location, such as an album printing service, or even just for some local backup.

It would be best if this can become a LibrePhotos Admin setting, and some progress will be displayed.

The error I'm getting in the browser console is: Uncaught (in promise) TypeError: e.response is undefined

And:

Object { message: "Request failed with status code 404", name: "AxiosError", code: "ERR_BAD_REQUEST", config: {…}, request: XMLHttpRequest, response: {…}, stack: "" }
​code: "ERR_BAD_REQUEST"
​config: Object { timeout: 30000, xsrfCookieName: "csrftoken", xsrfHeaderName: "X-CSRFToken", … }
​message: "Request failed with status code 404"
​name: "AxiosError"
​request: XMLHttpRequest { readyState: 4, timeout: 30000, withCredentials: true, … }
​response: Object { data: Blob, status: 404, statusText: "Not Found", … }
​stack: ""
​<prototype>: Object { constructor: o(e, t, n, r, o), toJSON: toJSON(), stack: "", … }```

[mrbc42]

Can this value be set as higher? I would say it needs to be at least 4 times higher, for full album downloads to succeed.

It's needed in case a user wants to save photos to a local folder for uploading to another location, such as an album printing service, or even just for some local backup.

Seconding this, is it possible for the devs just to change this 30s timeout and nothing else? I think it would really help alot of people, myself included. right now, if you want to download anything over 30 or 40 images or ~130 mb, its not possible.

[AnkurPrabhu]

The function that needs to be worked on is the following:

https://github.com/LibrePhotos/librephotos/blob/de428fc8408f7fce50c7f3266eff482f88719429/api/views/views.py#L1545-L1574

I would guess, that this is a timeout issue.

the function inside ZipListPhotosView should be an async task right ? and the data that is to be sent should be in chunks is what i feel let me know if i am missing something

[derneuere]

@AnkurPrabhu Not sure about sending it in chunks, but there needs to be a bit of scaffolding around it. The zipping can take a while, so maybe it makes sense to add this as a long-running job. The worst case is somebody wanting to export everything, and the average library is like 100 GB. Some checks are probably needed to not hog all the space and make the system unresponsive. It probably needs to be saved to a folder and be deleted once the zip is downloaded or after a certain time. Then you need a way to tell the client, that he can download the zip.

Should look something like this: Click download, zipping starts, user gets a window with how far along it is and then starts downloading once the zip is available.

I think adding a new model for the zips could make sense. In that way, asking if there is a zip you can download, should be easy. Also makes creating unique download links for downloading a zip possible.

[AnkurPrabhu]

@derneuere i have already started working on this issue would love it if you could assign it to me. also the approach i have used is just to make an async task but the async task class does not have any property where it can tell us if its still running or is finished it only has attributes result and success. i saw our existing async task functions and the use of long running jobs can you please give me some context how can i refactor my code and make it as long running job and things i should be aware of ....below is the code for my async task for making zip file

[derneuere]

Assigned it to you!

The logic looks good, but there should be a check for edge cases like if there is enough space :) Just create a couple of test to look for possible edge cases.

Adding a new type works like this:

1. Add a new type in this class: https://github.com/LibrePhotos/librephotos/blob/dev/api/models/long_running_job.py
2. Django should complain that changes happened, but the migration did not happen yet.
3. Change the entrypoint.sh to have a "python manage.py makemigrations | tee /logs/makemigrations.log"
4. After another restart you should have a new migration file in api
5. Create a similar boilerplate like in here: https://github.com/LibrePhotos/librephotos/blob/dev/api/batch_jobs.py

There is in general a {"progress": {"current": 0, "target": 0}} json object, which correspond with the values you see in the frontend.

Also you have a typo in output_directory :smile:

[AnkurPrabhu]

@derneuere should we close this now ?

[derneuere]

Yes you are right :)

[mrbc42]

Hi, I just updated my system to the latest Docker version running on Debian and the problem still persists. Tried to download 379 photos and under "top" in the Linux console, I could see "gunicorn" start but then disappear after less than a minute and no download available. Let me know if I can give you more info.

Debian 10.3
Docker version 24.0.5, build ced0996
Librephotos Latest

[mrbc42]

Hi, I just updated my system to the latest Docker version running on Debian and the problem still persists. Tried to download 379 photos and under "top" in the Linux console, I could see "gunicorn" start but then disappear after less than a minute and no download available. Let me know if I can give you more info.

Debian 10.3
Docker version 24.0.5, build ced0996
Librephotos Latest

Just did some more testing and looks like the maximum size file I can download is in the 150mb range so it has definitely improved. For me that is only about 50 photos. Is there some other file limit setting that I can alter to get a larger number of files downloaded?

[derneuere]

The new download mechanism is available on dev and will ship in the next release in december :)

[mrbc42]

The new download mechanism is available on dev and will ship in the next release in december :)

Perfect, thank you.

[loulou91]

I just made a pull on dev to make my own test... I'm no more able to download any group of pictures even one. I see this pop-up "Download started Download Starting ..." for some seconds, that's all, and nothing in my chrome "download" folder. Does anyone made some tests?

[derneuere]

Works fine in brave on android and with the demo server.

[mrbc42]

Trying to test dev in docker and changed the Tag in the .env file

tag=dev

but still loads latest after doing

docker-compose down
docker-compose pull
docker-compose up -d

is there something else I need to do to get the dev image?

[loulou91]

Works fine in brave on android and with the demo server.

your'e right, it also works in my testbase with same build (I tested only with a few pictures!). But with my ops base nothing happens. I had a look to logs of my ops db: nothing happears. in my test db logs only a line related to zip file deletion. 2023-11-25 08:17:51,976 : all_tasks.py : delete_zip_file : 85 : INFO : file deleted sucessfully at path : /protected_media/zip/e6534c8b-8fb9-4635-8a1e-121e0c8dbd1e1.zip

What could I do to investigate?

[loulou91]

I run LP without -d and I discovered a line "backend | Insufficient Storage: /api/photos/download" when I requested some pictures (3) download. Of course I checked I have enough place in my proxmox VM.

[derneuere]

Could you download your server stats? There should be a value called "free_storage_in_mb". If it reports the wrong storage size, I will have to recheck the package we use to check for free storage :)

[loulou91]

"image_tag": "dev", "available_ram_in_mb": 7942, "gpu_name": "", "gpu_memory_in_mb": "", "total_storage_in_mb": 49080, "used_storage_in_mb": 35464, "free_storage_in_mb": 11197, "number_of_users": 1,

[loulou91]

Even "small fix" can be efficient... ;-)

[loulou91]

Now looks working but... zip building is considered in worker log and it a good point as far as process may be a bit long! First test (450 pictures) was fine but zip file has been received some time after job completion... Second test was not good, zip file has correctly been built but not received. Moreover it stayed in "/protected_media/zip/" folder... Further trials (with 400 to 500 pictures) often finish with blocking of the VM... and nothing special in logs after reboot.

So, not so easy to see what happens!

[derneuere]

Hmm, this could be maybe a timeout on the client side. The zip usually gets deleted as soon as the download starts, but will also get auto deleted after a certain timeframe.

@AnkurPrabhu want to take a look at this?

[AnkurPrabhu]

Hmm, this could be maybe a timeout on the client side. The zip usually gets deleted as soon as the download starts, but will also get auto deleted after a certain timeframe.

@AnkurPrabhu want to take a look at this?

Yes will definitely look into this might need some guidance as in what environment is this being tested in and what should be my initial steps for debugging

[derneuere]

He test this on his production system, which is the same as dev.

I think you could check how well timeouts in the client work, by just waiting 30 seconds / minute or so in the view to reproduce the issue

[AnkurPrabhu]

@derneuere i had ran some tests so on local everything works for me so i tested it on demo everything worked for me on demo but the issue is the content download part of a request takes time so it is possible that the content download part times out not sure for me it went up to more than 1 min and still downloaded the file .... also we need to put a lot of images on demo server so that we can test things there or do we have something like dev server to test it

also can we get some more information of the network tab while running the tests from @loulou91

[derneuere]

Hmm, then it could possibly only be this issue #715 @loulou91 Did you use "Select All" for selecting the images?

[loulou91]

I restarted some tests this morning after a new pull (dev). In these tests I used "select all" button or "click" then "shift click" on the last picture. I didn't see any difference... It worked till I tried a collection of ("thing/37/") of 3534 pictures and then nothing happened even in worker log.

I founded "insufficient storage" in gunicom_django.log...

gunicorn_django.log

[AnkurPrabhu]

@loulou91 if you are receiving this message then maybe ur zip file is too large and you have no space for it ? did u check ur storage size ?

[miniMUSK]

I am also having issues with downloading pictures. I am trying to download just one album worth of pictures (120 photos) however, when I select the album and select all photos, then download, nothing happens. If i go to the photos tab and select all it does download a ZIP file but the download only takes about 5 seconds and there are only 18 photos in the download when I selected all my photos (over 4k). When I go to my worker log it does not show anything for Download Selected Photos like the picture above from @loulou91, I am very new to LibrePhotos only started using it a couple months ago. I love it but besides making it accessible publicly (not a simple way?), this has been my first obstacle since starting to use it and now I am seeking guidance...

[derneuere]

@miniMUSK It seems to be a regression, can you open a bug ticket?