search

Flameeyes / scan2pdf

GNU General Public License v3.0
8 stars 4 forks source link

Random failures with multi-page scans #1

Closed chuckb closed 12 years ago

chuckb commented 12 years ago

Using debian squeeze, armel build, for Marvel Kirkwood processor. When I run 3 or more page scans with an ADF, I get random failures of the conversion from unpaper to the next step.

chuckb commented 12 years ago

Turns out that my embedded system, with 128MB of ram, runs out of memory with more than 2 concurrent unpaper processes concurrently run, if the processes all simultaneously run the "converting to qpixels" step at or around the same time. Resident memory for each process goes from ~ 9MB to 41MB. Not sure if there is a switch in unpaper to tamp down memory buffer utilization...looking. I am using the unpaper package from debian squeeze (version 0.3).

chuckb commented 12 years ago

Running unpaper with --no-qpixels keeps memory utilization level...not sure the impact of resulting deskewing image processing. Will run tests and report back.

Flameeyes commented 12 years ago

(About pspawn, you probably noticed that I don't use spawn directly but rather their system() reimplementation, which is using waitpid properly.)

I guess it's time to find a way to limit the number of concurrent jobs to run at once, I'll see if I can find a way around this. Would help if I found the time to make unpaper use OpenMP or other multithread method so that instead of running N processes it can simply split up a single process...

chuckb commented 12 years ago

Right, I realized this once I dug more in the code (and I updated my comment when I realized it). I put in some logic to look at the amount of memory on the system and then switched on the --no-qpixels switch. Random failures have gone away. This fix will probably be needed, in combination with a process limiter to make large number of pages work with ADFs. I'll give you a pull request with my --no-qpixels hack later today.

chuckb commented 12 years ago

Used crude but workable method to watch free memory and only start enough concurrent image processing forks.