search

tryashtar / image-map

Converts image to maps for Minecraft Java and Bedrock
244 stars 36 forks source link

Make huge amount converting more efficient #70

Closed Militalex closed 2 years ago

Militalex commented 4 years ago

When converting a huge amount of Images (e.g. 100) the program will crash.

tryashtar commented 4 years ago

Could you post the crash message please?

Militalex commented 4 years ago

Unfortunately, there is now crash message because the program stops immediately running. I assume it is memory problem, because every time when the memory of the program hits more than around 1,500 MB the program stops running.

(I have 8 GB RAM and I have Intel(R) Core(TM) i5-6400 CPU @ 2,70GHz)

Here you find a video where I am showing the problem.

https://www.youtube.com/watch?v=VUWHprlwfPg

By the way, you see in this video also the problem mentioned in #69.

tryashtar commented 4 years ago

Oh dear. Yeah, definitely loading 4*5*110=2200 images is going to be pretty memory-intensive. The image previews take up way more native memory than they should; the process memory is usually 50 times the heap size. Unfortunately, there's not much I can do about the images in the previews taking up memory. They are only 128x128, so not too bad though. I did notice that I accidentally removed the disposal of the original image during some refactoring, so extremely large opened files would sit around in memory pointlessly. This was even worse if they were split into many maps, since the image gets scaled up before being split. So I did fix that -- the image gets reclaimed after the green bar finishes loading. Consequentially, it's probably best to avoid the "confirm all" button when dealing with huge images or huge amounts of them, since it pulls in everything at once and has the bars going at the exact same time (most of them can't even start anyway because the CPU gets hogged). So try this version out and try spamming the confirm button (or waiting for the bars to complete) instead of doing confirm all. Not super ideal but I'm interested in the results: ImageMap3.zip

Militalex commented 4 years ago

"Unfortunately, there's not much I can do about the images in the previews taking up memory." There is one thing you could do. You could buffer finished maps in a folder and load only the visible maps into memory.

"So try this version out and try spamming the confirm button..." Spamming the confirm button does not avoid this problem #69. Its correct that waiting avoids the problem, but if I wait, the converting will take too much time, because I can only convert 4 pictures at once without #69.

If I was you, I would limit the simultaneously calculated maps (to a specific number being adjustable in settings), so that the CPU doesn't get hogged.

tryashtar commented 4 years ago

You're right. I'll look into both of those things.

tryashtar commented 2 years ago

4.0 should handle this fine!