search

DIYBookScanner / spreads

Modular workflow assistant for book digitization
GNU Affero General Public License v3.0
127 stars 53 forks source link

SpreadPi and Powershot SX160 #190

Open niram8 opened 9 years ago

niram8 commented 9 years ago

I set up SpreadPi with two PowerShot However, the downloaded images (JPEG) are ot readible neither on windows or linux. I also got the error message "Thumbnail is not in JPEG format".The SX160 works well with CHDKPTP (on Windows 7).Can I set up or modify SpreadPi to work with the SX160.

markvdb commented 9 years ago

Please, in order to provide a somewhat useful bug report, include at least:

(Also note that this is not a support forum for spreadpi, but a bug tracker for errors in the spreads software. I will gladly help you determine if this is a spreads bug.)

niram8 commented 9 years ago

This is my setup:

The error logs are on the raspberry pi. I will have to figure out how to transfer them

niram8 commented 9 years ago

Unfortunately I do not know how to upload the log files to this thread. Apparently only images files (jpeg, etc) can be uploaded. Can send them by email? Thanks

adongy commented 9 years ago

Drop them on a pastebin and paste the link here.

On May 7, 2015 8:15:04 PM CEST, niram8 notifications@github.com wrote:

Unfortunately I do not know how to upload the log files to this thread. Apparently only images files (jpeg, etc) can be uploaded. Can send them by email? Thanks

Reply to this email directly or view it on GitHub: https://github.com/DIYBookScanner/spreads/issues/190#issuecomment-99965452

Sent from my Android device with K-9 Mail. Please excuse my brevity.

niram8 commented 9 years ago

See the logs of a freshly installed image (2015-04-26) and triggering session: spreadslog.json: http://pastebin.com/k6gBiJp4 spreads.log: http://pastebin.com/w8TaEk9x

adongy commented 9 years ago

As mark said, can you paste a 'corrupted' jpeg file?

adongy commented 9 years ago

Can you also try to take a picture using chdkptp (the lua shell), it's available over at assembla.

niram8 commented 9 years ago

Here are the two files: 001.jpg: by spreadpi (not readible) test.jpg: directly from chdkptp shell with the rs command (readible on Windows) Hope the upload works 001 test jpg

jbaiter commented 9 years ago

Thank you, that clears some things up! The root cause seems to lie in the Python<->chdkptp library. I just checked the corrupted file with a hex editor, and this is what seems to happen:

The beginning marker of the JPEG file (0xFFD8) only occurs at the very end of the file, but it should be at position 0. Curiously enough, it occurs right after the JPEG end marker (0xFFD9). So it seems to be the case that when assembling the chunks that are streamed from the device, the first chunk is written last.

After some poking around, it seems that I naively assumed that I received the chunks in order, which does not seem to be the case for your file. So I just pushed a change that sorts the chunks by their ascending offset before assembling them.

Could you try the new version on your SpreadPi and see if it helps? To update, SSH onto your Pi (user: spreads, password: spreads) and run the following:

$ sudo su
# Enter 'spreads' when asked for a password
$ pip install http://jbaiter.de/files/chdkptp.py-0.1.4beta.tar.gz
$ /etc/init.d/spread restart
# or `systemctl restart spreads` if you are running the very latest image
niram8 commented 9 years ago

Unfortunately it is not working. After the first update to my current image the cameras did not shoot at all. With a fresh image (2015-04-26) and the update I get the error: [string""]:4:attempt to compare nil with number

adongy commented 9 years ago

Most chunks do not have an offset... that's why you get this error.

jbaiter commented 9 years ago

Thanks for the hints, @adongy, I think I have found a better and more robust way to assemble the chunks that arrive from the cameras: https://github.com/jbaiter/chdkptp.py/commit/4ff8e538de80a00602922b11271a95323db20322

Here is a new beta release that you should try: http://jbaiter.de/files/chdkptp.py-0.1.4beta2.tar.gz

niram8 commented 9 years ago

With "chdkptp.py-0.1.4beta2.tar.gz" it is working. The images are readible and the thumbnails visible in the web interace. Shooting is not quite robust. May be still a way.

adongy commented 9 years ago

Can you give us logs for every issue you encounter?

adongy commented 9 years ago

as for zoom, it may be that the sx160 has many zoom step. try to put a high number (eg 60 or more) and try again.

niram8 commented 9 years ago

Sorry. The zoom apparently is a parameter issue. With about 30 it is working fine. Any hints how can I extract relevant parts of the log files. I am afraid the complete log files will become to clumpy.

markvdb commented 9 years ago

paste it to pastie.org or pastebin.com?

2015-05-12 0:03 GMT+02:00 niram8 notifications@github.com:

Sorry. The zoom apparently is a parameter issue. With about 30 it is working fine. Any hints how can I extract relevant parts of the log files. I am afraid the complete log files will become to clumpy.

— Reply to this email directly or view it on GitHub https://github.com/DIYBookScanner/spreads/issues/190#issuecomment-101061392 .

Mark Van den Borre Hogestraat 16 3000 Leuven, België +32 486 961726

jbaiter commented 9 years ago

I just found another set of issues in the chdkptp.py library that were probably the cause for the tuple index out of range errors you were seeing. Please install the new beta release:

# pip install http://jbaiter.de/files/chdkptp.py-0.1.4beta3.tar.gz

System/camera crashes in manual focus mode

Do you mean that the system completely freezes up? Or is it just the camera that becomes unresponsive?

niram8 commented 9 years ago

After install of "chdkptp.py-0.1.4beta3.tar.gz" spreads does not start any more. See the error capture after install: http://pastebin.com/zvEgZaQV

After reinstalling "chdkptp.py-0.1.4beta2.tar.gz" the system works again.

Here is a fresh log with some capture sessions (foot pedal not working) with "chdkptp.py-0.1.4beta3.tar.gz" : http://pastebin.com/CREygn87 The last session is with manual-focus. One camera crashes (shuts down) and the system reports the tulip error. Spreads does not freeze. After clicking OK on the tulip error the Web-Interface is still running.

Occasionally the Web-Interace freezes, i.e. does not come come back from shooting. I did not identify it reproducible. I assume that it happens when at least one camera is not reponding any more. I`ll try to capture a log for this.

niram8 commented 9 years ago

Sorry, The log is with "chdkptp.py-0.1.4beta2.tar.gz" (NOT chdkptp.py-0.1.4beta3.tar.gz): http://pastebin.com/CREygn87

jbaiter commented 9 years ago

Ugh, I just noticed that I have a horrible typo in beta3, please try this new one:

# pip install http://jbaiter.de/files/chdkptp.py-0.1.4beta4.tar.gz
nafraf commented 9 years ago

Not related to the bug, but to check in the future. Probably, the third method to set focus, where left is pressed during shot_half pressed, does not work for SX cameras.

jbaiter commented 9 years ago

@nafraf: Is there a way to set a manual focus value on the SX cameras over PTP?

niram8 commented 9 years ago

The image http://jbaiter.de/files/chdkptp.py-0.1.4beta4.tar. works. However foot pedal still is not functioning. Also "Tulip Error" still pops up

I have done the following testing sequence:

Resulting spreads.log: http://pastebin.com/jG0mDW5p

niram8 commented 9 years ago

Another Test Sequence resulting in the "tulip index out of range":

Resulting spreads.log http://pastebin.com/T2M8tmYH

nafraf commented 9 years ago

@jbaiter: For sx160, set_mf() is an option. See this test log:

connected: Canon PowerShot SX160 IS, max packet size 512
con> rec
con 1> =set_mf(1)
con 2> =set_focus(1200)
con 3> =return get_focus()
4:return:1248
con 4> =set_focus(800)
con 5> =return get_focus()
6:return:797

To detect which subject distance override modes work for each camera: get_sd_over_modes()

adongy commented 9 years ago

That issue should have been fixed in the very last commit by @jbaiter. @niram8, please install latest chdkptp.py from the git repository, using pip install -e git+https://github.com/jbaiter/chdkptp.py.git

niram8 commented 9 years ago

Using "pip install -e git+https://github.com/jbaiter/chdkptp.py.git" results in the error:

root@spreadpi /home/spreads # pip install -e git+https://github.com/jbaiter/chdkptp.py.git --editable=git+https://github.com/jbaiter/chdkptp.py.git is not the right format; it must have #egg=Package Storing debug log for failure in /root/.pip/pip.log root@spreadpi /home/spreads # /root/.pip/pip.log

adongy commented 9 years ago

whoops, extra -e is not needed i think

niram8 wrote:

Using "pip install -e git+https://github.com/jbaiter/chdkptp.py.git" results in the error:

root@spreadpi /home/spreads # pip install -e git+https://github.com/jbaiter/chdkptp.py.git --editable=git+https://github.com/jbaiter/chdkptp.py.git is not the right format; it must have #egg=Package Storing debug log for failure in /root/.pip/pip.log root@spreadpi /home/spreads # /root/.pip/pip.log

— Reply to this email directly or view it on GitHub https://github.com/DIYBookScanner/spreads/issues/190#issuecomment-102094741.

niram8 commented 9 years ago

<pip install git+https://github.com/jbaiter/chdkptp.py.git> runs to install but is not working. There is no web interface and also the IP address of spreadPi is not shown on the cameras. After re-installation of http://jbaiter.de/files/chdkptp.py-0.1.4beta4.tar it worked again.

jbaiter commented 9 years ago

I just uploaded a beta5 for you, @niram8, this should work now. I checked and it seems that the tarball I created had a messed up line.

http://jbaiter.de/files/chdkptp.py-0.1.4beta5.tar.gz

niram8 commented 9 years ago

I think I am not lucky ... <http://jbaiter.de/files/chdkptp.py-0.1.4beta5.tar.gz > is not working http://jbaiter.de/files/chdkptp.py-0.1.4beta4.tar.gz is working (after re-install)

jbaiter commented 9 years ago

@niram8, for the love of god, please be more specific than just 'is not working' :-( I really want to help, but without a log or even a description of 'not working' I really can't do anything.

niram8 commented 9 years ago

@jbaiter, sorry, you are really right. For whatever reason some parts of my comments are missing. Also the reference to the file which is not running. It should read:

"https://github.com/jbaiter/chdkptp.py.git runs to install but is not working. There is no web interface and also the IP address of spreadPi is not shown on the cameras."

I did not look at the spreads.log file but immediately reinstalled "http://jbaiter.de/files/chdkptp.py-0.1.4beta4.tar.gz" which was running again (IP adress on the cameras, Web-Interface running). If the log file is helpfull I can repeat the procedure as from the current log file I cannot identify the sequence of "http://jbaiter.de/files/chdkptp.py-0.1.4beta5.tar.gz"

Last part of spreads.log is pasted here: http://pastebin.com/gF1JP1Sc My internal log of the activities (with copy of teminal output) you will find here (see last two entries on 2015-05-14): http://pastebin.com/Hn6Sj0Eu

adongy commented 9 years ago

Unfortunately these logs do not show anything suspicious, just that you ran spreads and installed various versions of chdkptp.py.

What do you mean by 'is not working'? Does spreads error out when launching ? When capturing ? Does it say anything in the logs ?

niram8 commented 9 years ago

I have tested all the versions again. Here is the sequence and the results. (As described earlier, working means: after startup cameras show the IP address of the web interface, web interface is starting up; not working means: no IP address on the cameras, web interface not available but log on to raspberry pi by putty possible)

Sequence start with image "spreadpi-20150514,img" -> working pip install http://jbaiter.de/files/chdkptp.py-0.1.4beta.tar.gz -> working pip install http://jbaiter.de/files/chdkptp.py-0.1.4beta2.tar.gz -> Working pip install http://jbaiter.de/files/chdkptp.py-0.1.4beta4.tar.gz -> working pip install git+https://github.com/jbaiter/chdkptp.py.git -> not working (started twice) pip install http://jbaiter.de/files/chdkptp.py-0.1.4beta5.tar.gz -> not working (started twice) pip install http://jbaiter.de/files/chdkptp.py-0.1.4beta4.tar.gz -> working again

Resulting spreads.log: http://pastebin.com/88EFAVxR

Hope that is clearer.

adongy commented 9 years ago

Don't bother with the beta/beta2 files, they are now outdated.

Looks like it doesn't manage to find the camera, thus it doesn't work.

Can you edit spreadsplug/dev/chdkcamera.py, and replace the yield_devices method with this one ? http://pastie.org/10193030

It should give more info.

boan-anbo commented 9 years ago

Same thing here with A800. I used an earlier version of SpreadPi before. Back then, though one of the two camera dies (automatically shutdown, not freeze) from time to time and requires restarting, it was less frequent. Now with the latest release "tuple out of index" pops up every 20-50 pages, and I need to restart the Raspberry Pi each time...I tried to use separate USB hubs for each of the camera but it didn't work either. So I am also eagerly waiting for the next update...

jbaiter commented 9 years ago

@niram8: Is this really the full log? How does it e.g. 'crash'/'not work' with beta5? From the logs it looks like everything is running fine up to the point where it tries to find cameras. Does it stop at that point? Is there any console output after that?

@gazagoal: I'm working on it, I'll release a new stable chdkptp.py version once I've got to the bottom of niram8's issue. That should then be available in the next spreadpi build.

niram8 commented 9 years ago

@adongy, I have edited "spreadsplug/dev/chdkcamera.py" here is the log (after install of spreadpi-20150514,img / chdkptp.py-0.1.4beta5.tar.gz): http://pastebin.com/awY2mBwp

@jbaiter, It should be the full log. It just stops. There is no output on the cameras. I cannot see any console output (the pi runs headless). I can still access via putty. Can I look on any other log on the pi?

jbaiter commented 9 years ago

Wow, so it just deadlocks? Can you CTRL+C or /etc/init.d/spread stop or systemctl stop spreads the spreads process? Are there any suspicious messages in dmesg?

niram8 commented 9 years ago

There is no file /etc/init.d/spread on the system (at least I do not find it in that directory). Here is the output (in putty); root@spreadpi /home/spreads # /etc/init.d/spread stop bash: /etc/init.d/spread: No such file or directory root@spreadpi /home/spreads # systemctl stop spreads root@spreadpi /home/spreads #

I cannot analyse dmesg and systemctl. Here is the output of a freshly started system: dmesg (dmesg > dmesg-output): http://pastebin.com/qupuBawV systemctl (systemctl > systemctl-output); http://pastebin.com/bx2WSv2M

jbaiter commented 9 years ago

@scannopolis, it seems your comments got deleted, but I think you were onto something!

I did some tests last weekend and it turns out that the 'mode' the camera rotary button is set to plays a role. I recommend you all set it to 'P' (or the most 'manual' mode you have on your model), this seems to minimize the errors.

I also did some more work on the error handling and logging, this should help with stability and debugging: http://jbaiter.de/files/chdkptp.py-0.1.4b7.tar.gz

scannopolis commented 9 years ago

I deleted them myself because I wasn't sure if they were useful. I did perform some more test yesterday, using the same version of CHDK (build 1.3.0 ver 4154) and the latest build from Spreads. I did the following:

  1. lights off, book far from the glass, autofocus all: got images
  2. lights on, book far from the glass, autofocus all: got images
  3. lights on, book close to the glass, autofocus all: mistake 'thumb is not in JPEG format'
  4. lights off, book close to the glass, autofocus all: got images

It seems to me that nafraf is right about this, it might be a problem with the focus of the camera rather than something else. However, I didn't have time yesterday to fix the bugs with the code that you provided, but I might be able to do it in the weekend

scannopolis commented 9 years ago

I also did:

  1. lights on, book close to the glass, manual focus: 'thumb is not in JPEG format'
jbaiter commented 9 years ago

Scann, do you have a log and an example image for the 'thumb is not in JPEG format' error?

boan-anbo commented 9 years ago

Just to follow up on the earlier "tuple out of index" report. The issue has been largely resolved with the latest built and chdk client. Much thanks!

But a new problem is that now when "tuple" problem does occur, which happens now much less frequently, it may result in a bad file for the last photo I scan. This causes the problem that, even after I restart the spreadpi, the Raspberry will crash whenever it loads the last, perhaps badly stored, photo. So either when I want to continue from where I left after the "tuple problem", or when I want to browse the last photo scanned in that workflow, the Raspberry Pi will crash and shows "the server seems to be offline".

This issue can be solved by deleting the whole workflow, or reformat the SD card. But a fix will of course be much appreciated, for it means there is a chance that if the "tuple" problem happens again, you will lose everything you've scanned under the current workflow...

adongy commented 9 years ago

Can you post the last picture saved on such an error along with the logs ? It's really hard to debug without proper input.

boan-anbo commented 9 years ago

Hi Anthony, the error just occurred again and here are the picture files (simply empty...) and the logs. Deleting the last two empty pictures solves the problem; otherwise Raspberry crushes every time I open the webpage.

https://www.dropbox.com/sh/g0w2tofb0x4szv5/AAB8NKqawOAiaUbT267vYNNHa?dl=0

Hope it helps!

adongy commented 9 years ago

Thanks, I think I got it: jpegtran does not handle very well 0 bytes image files, it just segfaults leading to the errors you mentioned. I'll try to fix it later this week.

Edit: the root cause of it all is that one of the two cameras did not seem to take a shot when asked to, maybe a little bug in CHDK or in the python wrapper. Can you try to put the camera in 'P' mode when taking shots? Usually it gives better results.