Tenrec-Builders / pi-scan

Pi Scan is a simple, robust capture appliance for book scanners. It runs on a Raspberry Pi 2.
BSD 2-Clause "Simplified" License
264 stars 37 forks source link

Performance Improvements #11

Closed meelash closed 6 years ago

meelash commented 6 years ago

Now that I have a complete workflow functioning for scanning books and have some experience with it, I would like to increase the efficiency. There are two glaring bottlenecks, currently, one being errors that stop the flow and, the other, general slow performance. I'll be posting another issue to try to isolate the cause of those errors that I get and fix them. Here, I'd like to address the performance problems I'm finding.

Ideally, I think, one would expect the mechanical ability of the operator to be the limiting factor in maximum speed of scanning as opposed to the software. But I find that I am limited by software when I'm scanning.

When I hit the Scan button, almost instantly I hear the cameras fire, and I can begin moving to the next page. But then the majority of the time I can easily complete turning the page and lifting the book back up to the platen well before Pi Scan is ready to capture the next scan. The interesting thing is there seems to be quite a bit of variability in the time spent on the "Capturing..." screen, sometimes it matches my mechanical speed, and sometimes, I am waiting for some seconds.

The performance hit I am talking about is I would estimate that I could easily do upward of 1000 pages per hour physically, but currently I am maybe getting 600-700 even not counting the errors that I alluded to.

I guess the first question I have is, are my experiences typical? Or is there something I am doing wrong?

My system is a raspberry pi2 with touch screen, powershot IXUS 160 cameras, cheap class 4 memory cards (I thought these didn't matter? If necessary I can upgrade them to test).

duerig commented 6 years ago

The biggest thing to impact performance is the speed of the external storage device. Either your memory card or USB drive. This has a noticeable and immediate impact. With a fast memory card for external storage, I find that the processing time is almost always less than the time to flip the page.

The SD cards on the cameras themselves may have an impact as well, though I have not tested this extensively. So if you try speeding them up, let me know what if any impact you see.

The SD card on the Raspberry Pi does not have an impact on the speed at all. The only thing this might affect is startup time (how long it takes to boot into Pi Scan).

The typical setup that I use for this involves two cheap class 4 SD cards for the cameras, one cheap class 4 card for the Raspberry Pi. And one expensive Class 10 U3 card for external storage. Note that All of this goes over USB 2. So that will be the upper speed limit regardless.

Just a quick note regarding camera failure handling. A bug has crept into the most recent build which impedes fast failure handling in some cases. After a failure it will sometimes see the cameras as gphoto (DSLR) cameras instead of CHDK (ELPH 160) cameras. I am working on fixing this and will release version 1.6 with a fix when it is ready. My clever code for auto-detecting this was not as clever as I thought.

meelash commented 6 years ago

Yes, I did notice that issue with mistakenly detecting the cameras as gphoto cameras. Requires a restart and it will recognize them correctly.

My external storage is a 32GB USB drive (USB 2). I purchased one Class 10 card so I might try to use that as external storage and see if it makes a difference. No idea what the cards are in the camera, but I'm guessing they are Class 4. I'll experiment with this and see if I can get an improvement.

meelash commented 6 years ago

It is useful to know that my experience is not typical. I did a short test using my class 10 card plugged into a USB2 hub and it seemed to have a marked improvement, although technically, my USB 2 flash drive should have better performance than a class 10. Maybe it is a defective flash drive?

I will explore further, now that I know it is something in my setup that it is causing the software to be the bottleneck.

meelash commented 6 years ago

According to this app: https://itunes.apple.com/us/app/id425264550?mt=12, speeds on the USB2 flash drive are about a third of what they are using the class 10 sd card... 5.9MB/s vs 14.3MB/s for write speeds. So definitely need to look into a faster USB drive.

Dmole commented 6 years ago

Why is the rpi recommended, when you could have a USB3 Odroid or rockpro64 ?

meelash commented 6 years ago

Am I missing something? Doesn't seem like either of those devices are even being manufactured yet.

duerig commented 6 years ago

There is not that much that is specifically dependent on a Raspberry Pi. Comment out the gpio pin code and it will likely work on most any Linux installation with the right library dependencies installed.

In general, the Raspberry Pi is less expensive, better supported, with a larger community. There are SBC systems that have better specs. But it is unclear how better specs will be much of an improvement. Using a Raspberry Pi and a decently fast SD card for external storage, you can scan as fast as you turn the page safely on an Archivist or Archivist Quill.

If you need a much faster solution, you might try TwoCamControl which runs on a laptop or desktop computer and therefore maximizes speed in a way that no SBC will do. @meelash just commented on another thread that TwoCamControl definitely seems to run faster.

Dmole commented 6 years ago

@meelash the XU3/XU4 has been available since 2014.

@duerig The Raspberry Pi is by far the most popular but I found Odroid to be the 2nd most popular and for about 2x the price one gets about 10x the speed, so I started using them a few years ago and never looked back.

I only brought this up because the demo video seemed slow, yes I'd use a laptop, but for those who want a dedicated solution I thought I'd mention RPi alternatives I have had positive experiences with, like the RK3399 is a popular choice for upcoming SBCs and it has been used in laptops in the past (~30x faster than a pi for some things like crypto).

duerig commented 6 years ago

The thing that is most tempting about these other models is the increased RAM. The biggest limitation of the Pi as it stands is the 1 GB memory limit. Increasing the memory is what I keep wanting to see from each Pi release. If and when I ever move to one of the competitors, it would be because of the memory issue.

On Tue, Jun 12, 2018, at 10:35 AM, Tim wrote:

@meelash[1] the XU3/XU4 has been available since 2014.

@duerig[2] The Raspberry Pi is by far the most popular but I found Odroid to be the 2nd most popular and for about double the price one gets about 10 times the speed, so I started using them a few years ago and never looked back.> I only brought this up because the demo video seemed slow, yes I'd use a laptop, but for those who want a dedicated solution I thought I'm mention RPi alternatives I have had positive expereances with, like the RK3399 is a popular choice for upcoming SBCs and it has been used in laptops in the past (~30x faster than a pi for some things like crypto).> — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub[3], or mute the thread[4].

Links:

  1. https://github.com/meelash
  2. https://github.com/duerig
  3. https://github.com/Tenrec-Builders/pi-scan/issues/11#issuecomment-396654169
  4. https://github.com/notifications/unsubscribe-auth/AA1sTLTBw3dPblm_Z-orUwLf8xmj2ETlks5t7-3JgaJpZM4Rdqax