lovasoa / dezoomify-rs

Zoomable image downloader for Google Arts & Culture, Zoomify, IIIF, and others
https://dezoomify-rs.ophir.dev
GNU General Public License v3.0
736 stars 65 forks source link

dezoomify-rs crashes when it goes out of available memory #3

Closed Arekkusu1998 closed 4 years ago

Arekkusu1998 commented 5 years ago

Forgive me @lovasoa, but I've a problem. I want to dezoomify the cadastral map of Capergnanica. Following your instructions, I got the tiles.yaml file; I put it in the same folder as the .exe file; I started the .exe and it appeared on a black background: "Enter a URL or a path to a tiles.yaml file: ..."; I pasted the tiles.yaml file and started. On the verge of finishing and creating the .jpg file, dezoomify-rs unexpectedly stopped with THIS SCREEN.

Errore

What does it mean, please? Point out that:

Thank you in advance.

lovasoa commented 5 years ago

Hello, and thank you for opening this issue. It looks like dezoomify-rs stops functioning when it tries to allocate memory for the final image canvas. What is the size of the image you are trying to download, and much RAM do you have? Also, can you post the tiles.yaml file you created?

rigel71 commented 5 years ago

Hi, with the first version of dezoomify-rs.exe I dezoomify some maps of as milano. In .jpg I open all image, in .png I have some problem. Now I test the new version, thanks

Arekkusu1998 commented 5 years ago

Thank you for your concern, @lovasoa.

Here is the information you requested:

However, two strange facts have occurred πŸ€”: the day before yesterday, Chrome strangely signaled me the download of dezoomify-rs as dangerous, trying in any way to obstacle it. Yesterday morning, I deleted it and re-downloaded it, but from Firefox, without warnings and/or advices. I tried to dezoomify the Capergnanica map and this time it all went right. At this point, I can't really explain to myself the origin of the repeated lock-downs of ... So far I've successfully dezoomed 7 maps, all supported the IIIF protocol. Furthermore, I cannot provide an explanation to the difficulty encountered by @rigel71: once the dezoomified.jpg file is downloaded, if I change the extension to .png, my pc (old!) not only displays it, but displays it with a few tenths of a second ahead of time to the .jpg format (which, however, appears perfect, at my opinion).

I hope I have been helpful.

lovasoa commented 5 years ago

Can you provide a screenshot of the warning messages chrome shows ?

I tried your tiles.yaml on my computer, where I have much more available memory, and it works fine. This image requires around 3Gb of memory to be dezoomed with the current version of dezoomify, so make sure you close all the other running applications that may consume memory when you dezoom it.

I am going to keep this issue open, and use it to keep track of dezoomify-rs memory usage. It can still be improved: currently both the individual tiles and the final image are kept in memory at the same time.

Arekkusu1998 commented 5 years ago

Here's the Chrome's screenshot during the download (locked), as you requested @lovasoa. warning Yes, I admit my pc is shamefully old (2007) and has a reduced memory 😣: not surprisingly, dezoomify-rs worked with Capergnanica cadastral map in the early morning, just turned on the computer.

Thank you very much: if you keep this issue open, I'll refer to it for any future questions (especially if I can't create the /info.json string or the tiles.yaml file).

lovasoa commented 5 years ago

You can open as many new issues as you want. Dealing with one issue per thread helps keeping things organized.

I am also thinking of adding direct support for IIIF to dezoomify-rs, so that you can directly paste the url of the info.json file and let the tool do the rest.

lovasoa commented 5 years ago

Starting with version 1.2.0, dezoomify-rs supports IIIF directly. This means you can simply give it the url of an info.json file, and it will start dezooming immediately. You don't need to create tiles.yaml files for asmilano.it anymore, for instance. You can simply give dezoomify-rs an url such as http://www.asmilano.it/fast/iipsrv.fcgi?IIIF=/opt/divenire/files/./tifs/05/38/538110.tif/info.json and it will do the rest automatically. Giving the URL instead of using a tiles.yaml file also allows you to choose which zoom level to download. If you don't have enough memory for the highest zoom level, you can still download a smaller, less zoomed-in version.

rigel71 commented 5 years ago

wonderful! It is very simply. I dezoomed in various zoom level. All is perfect

lovasoa commented 5 years ago

Hi all ! I just released version 1.3.0 that is faster and uses less memory :) I'd love to hear your comments: are there still images that are too large to dezoom with the memory you have ?

Arekkusu1998 commented 5 years ago

Thank U very much for Ur precious work, @lovisoa😊! I haven't definitively switched to a more advanced version because in the black panel of the .exe file I CAN'T use the Copy-Paste function for the URLs (.../info.json). So, if I make a single typo, I've to start all over again. I know how to inhabit Copy-Paste in the prompt, but in dezoomify-rs it seems not to be possible the same thingπŸ˜₯.

lovasoa commented 5 years ago

I don't have a windows machine to test, but I'm pretty sure you can use copy and paste in the windows terminal. What you call the "black panel" is your operating system terminal, it is not managed by dezoomify-rs itself. So you should look up on the internet how to copy-and-paste to your terminal. See for example : https://www.laptopmag.com/articles/how-to-windows-10-command-prompt-copy

lovasoa commented 5 years ago

And if you say you know how to copy-and-paste in the prompt, you can launch dezoomify-rs from the prompt.

rigel71 commented 5 years ago

It is simple you have to open the prompt, then... immagine

immagine

Arekkusu1998 commented 5 years ago

I sincerely thank both, @lovasoa and @rigel71, for the help offered to me: with your instructions, I've solved the Copy-Paste Command issue on the black panel of dezoomify-rs v.1.3.1😊!

MrChrisWin commented 4 years ago

Thank you. You raised good points.

lovasoa commented 4 years ago

Streaming image encoders

Help wanted

Hello everyone ! After such a long time, I finally implemented the first streaming image writer. A streaming image writer streams the image to the disk, without ever needing to hold the full image in memory. This requires a complex buffering system, and this is a big code change, so I need beta-testers before I can publish the release for everyone. Currently, we have a single streaming encoder: the one for PNG images (jpeg images are limited in size, so they usually fit in memory anyway).

So, if you want to help, please beta-test the new code, saving your own huge images to PNG, and report any bug, performance regression, or memory usage regression you find !

Download the beta at: https://send.firefox.com/download/7764d0b4c712eb99/#-TFS_ZES-tcgZpz-odvaYw

(the link is temporary and the files will be deleted after 7 days or 100 downloads, whichever comes first)

@asoftbird, @KempWatson, mentioning you because this might be of interest to you too.

MrChrisWin commented 4 years ago

Streaming image encoders

Help wanted

Hello everyone ! After such a long time, I finally implemented the first streaming image writer. A streaming image writer streams the image to the disk, without ever needing to hold the full image in memory. This requires a complex buffering system, and this is a big code change, so I need beta-testers before I can publish the release for everyone. Currently, we have a single streaming encoder: the one for PNG images (jpeg images are limited in size, so they usually fit in memory anyway).

So, if you want to help, please beta-test the new code, saving your own huge images to PNG, and report any bug, performance regression, or memory usage regression you find !

Download the beta at: https://send.firefox.com/download/7764d0b4c712eb99/#-TFS_ZES-tcgZpz-odvaYw

(the link is temporary and the files will be deleted after 7 days or 100 downloads, whichever comes first)

@asoftbird, @KempWatson, mentioning you because this might be of interest to you too.

So I tried to download this at level 8 and output to PNG: https://artsandculture.google.com/asset/the-tower-of-babel-pieter-bruegel-the-elder/bAGKOdJfvfAhYQ

I have been running several iterations and noticed the following:

KempWatson commented 4 years ago

Ditto that. Hangs on tile 176 for me too.

Kemp

On Sun, May 17, 2020 at 9:38 PM Chris Win notifications@github.com wrote:

Streaming image encoders Help wanted

Hello everyone ! After such a long time, I finally implemented the first streaming image writer. A streaming image writer streams the image to the disk, without ever needing to hold the full image in memory. This requires a complex buffering system, and this is a big code change, so I need beta-testers before I can publish the release for everyone. Currently, we have a single streaming encoder: the one for PNG images (jpeg images are limited in size, so they usually fit in memory anyway).

So, if you want to help, please beta-test the new code, saving your own huge images to PNG, and report any bug, performance regression, or memory usage regression you find !

Download the beta at: https://send.firefox.com/download/7764d0b4c712eb99/#-TFS_ZES-tcgZpz-odvaYw

(the link is temporary and the files will be deleted after 7 days or 100 downloads, whichever comes first)

@asoftbird https://github.com/asoftbird, @KempWatson https://github.com/KempWatson, mentioning you because this might be of interest to you too.

So I tried to download this at level 8 and output to PNG: https://artsandculture.google.com/asset/the-tower-of-babel-pieter-bruegel-the-elder/bAGKOdJfvfAhYQ

It seems to hang around 176 to 180 tiles.

β€” You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/lovasoa/dezoomify-rs/issues/3#issuecomment-629900428, or unsubscribe https://github.com/notifications/unsubscribe-auth/AC2OM5OGR7DTL3KCU2PEH2TRSCGTFANCNFSM4IK4GGMQ .

lovasoa commented 4 years ago

Thank you for the feedbacks ! Here is a new beta for today, taking them into account:

Pauses in the downloads may still happen, but they should be shorter and less frequent. If you have a good internet connection, tiles can be downloaded faster than they can be encoded and written to disk: since the whole goal of the streaming encoder is to be able to limit the memory pressure, it has to stop the downloads if it can't keep up with the incoming data. But in all cases, the total process should be faster: the time you waste during the pauses is time you would have spent encoding the image in the end in the old architecture. If the total time taken is (significantly) larger than before, it's a bug, please report it here ! πŸš€πŸš€πŸš€ https://send.firefox.com/download/7b82c2588a756d48/#yVeSihB6lH_EexJgfG-wBQ

P.S. I just realized that the previous binaries I send were compiled with the wrong flags, they were development builds, with lower performance.

MrChrisWin commented 4 years ago

lovasoa, thank you for trying something new and improving the software.

Using the same google art & culture link above,

Several thoughts on improving speed for consideration:

FYI: I have 100mpbs internet connection with 32GB of RAM, i7-4790K, Win10. Dezoomify-rs outputs to a 10GB RAMdisk which in theory should be much faster than SSD.

lovasoa commented 4 years ago

Hello and thank you for testing ! I tried your google arts link, and could not reproduce your issue :

     Running `target/release/dezoomify-rs 'https://artsandculture.google.com/asset/the-tower-of-babel-pieter-bruegel-the-elder/bAGKOdJfvfAhYQ' huge.png -w 100000`
[ETA:0s] ######################################## 22704/22704 Finished tile download
Image successfully saved to 'huge.png' (current working directory: /Users/olojkine/Developpement/dezoomify-rs)

real    13m41.241s
user    20m50.717s
sys     7m20.902s

It takes 13 minutes to download the 22704 tiles and create the 8.4 Gb PNG file. Are you using the latest beta I published, and exporting to PNG ?

KempWatson commented 4 years ago

With second code drop, Tower of Babel (on level 8, the largest) hangs for me consistently between 1200-1210 tiles... but not always the same exact tile. The hang is very long, but not forever... after, many WARNs, and a corrupt PNG is generated. Windows, 4C/8T, 32 GB. While it hangs, CPU is 12.5%, so fully occupying 1 core. At 1200 tiles, that's nowhere near memory limit, so that's not the issue... sounds like maybe hanging on write?

Better, but no cigar yet!

Kemp

On Tue, May 19, 2020 at 5:18 AM Ophir LOJKINE notifications@github.com wrote:

Hello and thank you for testing ! I tried your google arts link, and could not reproduce your issue :

 Running `target/release/dezoomify-rs 'https://artsandculture.google.com/asset/the-tower-of-babel-pieter-bruegel-the-elder/bAGKOdJfvfAhYQ' huge.png -w 100000`

[ETA:0s] ######################################## 22704/22704 Finished tile download Image successfully saved to 'huge.png' (current working directory: /Users/olojkine/Developpement/dezoomify-rs)

real 13m41.241s user 20m50.717s sys 7m20.902s

It takes 13 minutes to create the 8.4 Gb PNG file.

β€” You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/lovasoa/dezoomify-rs/issues/3#issuecomment-630697694, or unsubscribe https://github.com/notifications/unsubscribe-auth/AC2OM5PRRDF74HWHR7A6VYDRSJFH5ANCNFSM4IK4GGMQ .

MrChrisWin commented 4 years ago

With second code drop, Tower of Babel (on level 8, the largest) hangs for me consistently between 1200-1210 tiles... but not always the same exact tile. The hang is very long, but not forever... after, many WARNs, and a corrupt PNG is generated. Windows, 4C/8T, 32 GB. While it hangs, CPU is 12.5%, so fully occupying 1 core. At 1200 tiles, that's nowhere near memory limit, so that's not the issue... sounds like maybe hanging on write? Better, but no cigar yet! Kemp … On Tue, May 19, 2020 at 5:18 AM Ophir LOJKINE @.***> wrote: Hello and thank you for testing ! I tried your google arts link, and could not reproduce your issue : Running target/release/dezoomify-rs 'https://artsandculture.google.com/asset/the-tower-of-babel-pieter-bruegel-the-elder/bAGKOdJfvfAhYQ' huge.png -w 100000 [ETA:0s] ######################################## 22704/22704 Finished tile download Image successfully saved to 'huge.png' (current working directory: /Users/olojkine/Developpement/dezoomify-rs) real 13m41.241s user 20m50.717s sys 7m20.902s It takes 13 minutes to create the 8.4 Gb PNG file. β€” You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#3 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AC2OM5PRRDF74HWHR7A6VYDRSJFH5ANCNFSM4IK4GGMQ .

How long did it take you to download the full file?

KempWatson commented 4 years ago

No idea... overnight, and I didn't check timing. Will do again. The hang was at least an hour.

K.

On Tue, May 19, 2020 at 9:51 AM Chris Win notifications@github.com wrote:

With second code drop, Tower of Babel (on level 8, the largest) hangs for me consistently between 1200-1210 tiles... but not always the same exact tile. The hang is very long, but not forever... after, many WARNs, and a corrupt PNG is generated. Windows, 4C/8T, 32 GB. While it hangs, CPU is 12.5%, so fully occupying 1 core. At 1200 tiles, that's nowhere near memory limit, so that's not the issue... sounds like maybe hanging on write? Better, but no cigar yet! Kemp … <#m6010351234432101993> On Tue, May 19, 2020 at 5:18 AM Ophir LOJKINE @.***> wrote: Hello and thank you for testing ! I tried your google arts link, and could not reproduce your issue : Running target/release/dezoomify-rs ' https://artsandculture.google.com/asset/the-tower-of-babel-pieter-bruegel-the-elder/bAGKOdJfvfAhYQ' huge.png -w 100000 [ETA:0s] ######################################## 22704/22704 Finished tile download Image successfully saved to 'huge.png' (current working directory: /Users/olojkine/Developpement/dezoomify-rs) real 13m41.241s user 20m50.717s sys 7m20.902s It takes 13 minutes to create the 8.4 Gb PNG file. β€” You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#3 (comment) https://github.com/lovasoa/dezoomify-rs/issues/3#issuecomment-630697694>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AC2OM5PRRDF74HWHR7A6VYDRSJFH5ANCNFSM4IK4GGMQ .

How long did it take you to download the full file?

β€” You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/lovasoa/dezoomify-rs/issues/3#issuecomment-630832873, or unsubscribe https://github.com/notifications/unsubscribe-auth/AC2OM5K4ULSF4QJKZ65WSHDRSKFHVANCNFSM4IK4GGMQ .

lovasoa commented 4 years ago

Warnings

When you see warnings, can you please report them ? What were they about ?

Corrupt PNG

Also, how are the PNG files corrupt ? Not many desktop applications can open huge images so they can fail on a valid (but large) PNG file. I understand that a file that just sits there without being able to be opened by anything isn't of great use, but I do not have control over the image viewers themselves.

Potential new output format

Fortunately, the new architecture that allows for multiple encoders is also a good first step towards the implementation of #26. It should now be possible to create a specialized encoder that saves the tiles to disk, in, say, the IIIF format, in order to make it practical to view the image locally, without having to resort to creating impractically large files. But one thing at a time :) I first want to make sure the new version does not introduce any regressions, and ship it. Once we are sure that the new version is faster than the old one (or as fast in the worst cases), uses less memory (or as much in the worst cases), and can dezoom all the images that the old one can, I'll release it and start focusing on the next things.

KempWatson commented 4 years ago

"Failed to load {tile}". It's happening even on level 6, and the CPU is 1 core pegged, otherwise no network, memory, or disk pressure. I've tried it from 12 locations globally, 2 Google images, different zoom levels. Have not tried other image sources yet though, so it's certainly possible it's only Google.

Kemp

On Tue, May 19, 2020 at 10:15 AM Ophir LOJKINE notifications@github.com wrote:

Warnings

When you see warnings, can you please report them ? What were they about ? Corrupt PNG

Also, how are the PNG files corrupt ? Not many desktop applications can open huge images so they can fail on a valid (but large) PNG file. I understand that a file that just sits there without being able to be opened by anything isn't of great use, but I do not have control over the image viewers themselves. Potential new output format

Fortunately, the new architecture that allows for multiple encoders is also a good first step towards the implementation of #26 https://github.com/lovasoa/dezoomify-rs/issues/26. It should now be possible to create a specialized encoder that saves the tiles to disk, in, say, the IIIF format, in order to make it practical to view the image locally, without having to resort to creating impractically large files. But one thing at a time :) I first want to make sure the new version does not introduce any regressions, and ship it. Once we are sure that the new version is faster as the old one (or as fast in the worst cases), uses less memory (or as much in the worst cases), and can dezoom all the images that the old one can, I'll release it.

β€” You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/lovasoa/dezoomify-rs/issues/3#issuecomment-630847964, or unsubscribe https://github.com/notifications/unsubscribe-auth/AC2OM5O5C3PIJN75SZWW333RSKIARANCNFSM4IK4GGMQ .

lovasoa commented 4 years ago

@KempWatson : It's strange, because the tile downloading and decoding hasn't changed in this version. Doesn't this error also happen with the latest stable release of dezoomify ? I'm going to make the warning more verbose anyway, to help debugging it. And failed tile downloads are systematically retried by default, you can disable that by setting --retries 0 to speed of the process if you know there are missing tiles (but I guess it's not the case with google).

lovasoa commented 4 years ago

@dancorpse, I forgot to mention you, but you may be interested in trying the new beta too. I think you are one of the oldest dezoomusers out there ;)

KempWatson commented 4 years ago

Nope, stable works perfectly every time, no WARNS, takes about 20 seconds for Babel level 6. PNG verifies as ok. Redid new codebase, level 6, gave many WARNs for tile load failures, then crashed on:

ERROR Input/Output error: wrong data size, got 237 bytes too many

Did it a second time, still going after 5 minutes, many errors, both WARN tile load, and a second type of error, with no WARN, no timestamp, just a "network error: error sending request for url". Also a third type, just once "network error: request or response b".

Still finalizing file after 20 minutes, so at least 60 times slower than original, and with errors :-(

Kemp

On Tue, May 19, 2020 at 11:15 AM Ophir LOJKINE notifications@github.com wrote:

@KempWatson https://github.com/KempWatson : It's strange, because the tile downloading and decoding hasn't changed in this version. Doesn't this error also happen with the latest stable release of dezoomify ? I'm going to make the warning more verbose anyway, to help debugging it.

β€” You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/lovasoa/dezoomify-rs/issues/3#issuecomment-630889850, or unsubscribe https://github.com/notifications/unsubscribe-auth/AC2OM5LFVQQTDUGIEIIE7DDRSKPCLANCNFSM4IK4GGMQ .

lovasoa commented 4 years ago

I fixed the wrong data size, got X bytes too many yesterday in 769b68a13946b19968913adc6baffed25a0ca249. I updated the download link after fixing it, but you must have downloaded it before. I should have made it explicit I uploaded a new version a few minutes after the first one. And perhaps more importantly: the builds were still slow debug builds, because I had misconfigured the CD pipeline. Sorry about that !

Anyway, here is a new version, which is a fast release build, and where the warnings about failed tile downloads are more verbose.

https://send.firefox.com/download/28b9e96a8e6d00bf/#EELQvO0e0_v1dco8BP4HnA

KempWatson commented 4 years ago

Good news... downloads level 6, 7, 8 all without any issues at all. I was getting around 5 MB/s on network, and 13 MB/sec to disk, which makes sense. No weirdness at all now :-) so far so good.

Kemp

On Tue, May 19, 2020 at 1:29 PM Ophir LOJKINE notifications@github.com wrote:

I fixed the wrong data size, got X bytes too many yesterday in 769b68a https://github.com/lovasoa/dezoomify-rs/commit/769b68a13946b19968913adc6baffed25a0ca249. I updated the download link after fixing it, but you must have downloaded it before. I should have made it explicit I uploaded a new version a few minutes after the first one. And perhaps more importantly: the builds were still slow debug builds, because I had misconfigured the CD pipeline. Sorry about that !

Anyway, here is a new version, which is a fast release build, and where the warnings about failed tile downloads are more verbose.

https://send.firefox.com/download/28b9e96a8e6d00bf/#EELQvO0e0_v1dco8BP4HnA

β€” You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/lovasoa/dezoomify-rs/issues/3#issuecomment-630966380, or unsubscribe https://github.com/notifications/unsubscribe-auth/AC2OM5MAI4RAINCV5VVQOLTRSK6WPANCNFSM4IK4GGMQ .

lovasoa commented 4 years ago

I made a few more improvements , and you can download the latest build here : https://github.com/lovasoa/dezoomify-rs/releases/tag/build-25e9c8d18aa0196c59d5e2050bd6fb3443b9fcd5

If no one reports any regression, then I'll probably release it this week.

MrChrisWin commented 4 years ago

I made a few more improvements , and you can download the latest build here : https://github.com/lovasoa/dezoomify-rs/releases/tag/untagged-7390da2b90bad1bc3e9a

If no one reports any regression, then I'll probably release it this week.

The link returns a 404 not found.

lovasoa commented 4 years ago

I updated the link βœ…

MrChrisWin commented 4 years ago

Works well. lol my computer can't handle the 8GB tower of babel png file

lovasoa commented 4 years ago

Thank you everyone for beta-testing. dezoomify-rs v2.0.0 is now published. I can finally proudly close this issue.

KempWatson commented 4 years ago

Congrats! Looking forward to beating on this more!

On Sat, May 23, 2020 at 5:21 PM Ophir LOJKINE notifications@github.com wrote:

Closed #3 https://github.com/lovasoa/dezoomify-rs/issues/3.

β€” You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/lovasoa/dezoomify-rs/issues/3#event-3367434806, or unsubscribe https://github.com/notifications/unsubscribe-auth/AC2OM5JDX66PXTYYHDTVTGLRTA45LANCNFSM4IK4GGMQ .

rigel71 commented 4 years ago

Sorry. I read only now. It is a terrible time. I will try the new one

Il 23.05.2020 23:21 Ophir LOJKINE ha scritto:

Thank you everyone for beta-testing. dezoomify-rs v2.0.0 is now published [1]. I can finally proudly close this issue.

-- You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub [2], or unsubscribe [3].

Links:

[1] https://github.com/lovasoa/dezoomify-rs/releases/tag/v2.0.0 [2] https://github.com/lovasoa/dezoomify-rs/issues/3#issuecomment-633142095 [3]

https://github.com/notifications/unsubscribe-auth/AL2CGSWITDUAKCUQ7EVBFNDRTA45LANCNFSM4IK4GGMQ

rigel71 commented 4 years ago

Hi Do you use the same way as the previous one?

lovasoa commented 4 years ago

What do you mean exactly ? If you are having issues with the new version, then can you please open a new issue ? Each message in this thread notifies all participants.