Problems with v4l2/Raspberry PI Camera and exposure times

tomn46037 commented 7 years ago

Any time I try to take a preview image with INDI (using KStars with the VM or natively installed on Windows 10) I get the following error. Live Video behaves the same way.

2017-07-04T03:07:52: Driver indi_v4l2_ccd: read message V4L2 CCD 'Failed 1.000-second manual exposure, out of device tick bounds [0,0]' 2017-07-04T03:07:52: Client 0: queuing 2017-07-04T03:07:52: Driver indi_v4l2_ccd: read setNumberVector V4L2 CCD CCD_EXPOSURE Alert CCD_EXPOSURE_VALUE='1' 2017-07-04T03:07:52: Client 0: queuing 2017-07-04T03:07:52: Client 0: sending msg copy 1 nq 2:

Further, any time I go into V4L2 CCD under Main Control and try to change "Expose" I get one of two results.. if I enter a valid exposure (.001-3600) I get the following:

2017-07-04T03:22:55: Failed 1.000-second manual exposure, out of device tick bounds [0,0]

If I enter something out of bounds, I get what looks to be a valid error:

2017-07-04T03:23:26: Requested exposure value (3601) seconds out of bounds [0.001,3600].

What's interesting to me is the bounds listed in the brackets. When I enter something not valid, it has the correct numbers. When I enter something valid.. it has the wrong numbers.

I have tried using both the daily build (http://indilib.org/download/raspberry-pi/download/6-raspberry-pi/9-indi-library-for-raspberry-pi.html) under Raspian Jessie as well as the PPA repository under Ubuntu Mate. I have tried this on a ZeroW and a Rasbperry Pi 3 with both v1.3 and v2.1 cameras.

knro commented 7 years ago

is this just limited to the Raspberry PI cameras or other V4L2 cameras?

TallFurryMan commented 7 years ago

I'd like to know too, is this a regression compared to official 1.4.1? I was chasing this as part of #242, it was crashing the driver.

tomn46037 commented 7 years ago

Interesting.. NO, it is not limited to the built in Pi camera. I didn't expect that.. (I also tried 4 seconds because I didn't want 3=3 to come up. It failed in the same way at 4 seconds.) For this test, I used a Logitech webcam.

This does seem different - in that it is showing the bounds.. [3,10000] where previously it showed [0,0]. But, it's still failing to see that the provided number of seconds is in the listed range.

2017-07-04T14:49:42: Driver indi_v4l2_ccd: read message V4L2 CCD 'Failed 3.000-second manual exposure, out of device tick bounds [3,10000]' 2017-07-04T14:49:42: Client 0: queuing 2017-07-04T14:49:42: Client 0: sending msg copy 1 nq 1:

2017-07-04T14:49:42: Driver indi_v4l2_ccd: read setNumberVector V4L2 CCD CCD_EXPOSURE Alert CCD_EXPOSURE_VALUE='3' 2017-07-04T14:49:42: Client 0: queuing 2017-07-04T14:49:42: Client 0: sending msg copy 1 nq 1:
3
2017-07-04T14:49:54: Client 0: read newNumberVector V4L2 CCD CCD_EXPOSURE CCD_EXPOSURE_VALUE='10001' 2017-07-04T14:49:54: Driver indi_v4l2_ccd: queuing responsible for 2017-07-04T14:49:54: Driver indi_v4l2_ccd: sending msg copy 1 nq 1:
10001
2017-07-04T14:49:54: Driver indi_v4l2_ccd: read message V4L2 CCD 'Requested exposure value (10001) seconds out of bounds [0.001,3600].' 2017-07-04T14:49:54: Client 0: queuing 2017-07-04T14:49:54: Client 0: sending msg copy 1 nq 1:

It looks like I'm on a daily build?? Hopefully, this will get you what you need there. Also, I don't see any other versions in the repo. This is under Ubuntu Mate.

tomn@rpi-telescope:~$ dpkg -l | grep indi-full ii indi-full 1.4~201707040148~ubuntu16.04.1 armhf Instrument-Neutral Device Interface library - Full INDI

TallFurryMan commented 7 years ago

From the code, the reason for the weird log showing the [0,0] boundaries is that the driver has no information about the absolute exposure capabilities of the device.

The absolute exposure control is named "Exposure (absolute)" and is located in the "V4L2 Control" tab. If that control is not there, there is consequently no support for absolute exposure, effectively making the driver... less useful.

One test that can be made is to start a live video, if the underlying system supports it (network bandwidth, etc). I suppose a lightweight configuration would be adjustable. If that works, that means the driver code is incorrectly thinking there is no way to configure a manual exposure.

A regression against official 1.4.1 would be interesting. I'm currently looking at PR #245 as a possible suspect, but I don't understand the code yet. @geehalel would you be able to help us? We're looking for something preventing one of the ImageAdjustNP elements from showing up.

For reference on V4L2 behavior, Raspbian Jessie has a v4.x. @tomn46037 what is the exact version?

Besides I have no explanation - yet - for your second report. Boundary [0.001,3600] is sourcing from the Primary CCD exposure, but I have not checked how it may end up there. It might all too well be a blatant bug with pointers :)

TallFurryMan commented 7 years ago

@tomn46037 Could you please in the INDI Control Panel, relevant "V4L2 CCD" tab:

Enable driver debug information by clicking "Debug" "Enable" in the "Options" panel
Check driver logs at the bottom of the "Options" panel
Disconnect the driver by clicking on "Disconnect" in the "Main Control" panel
Clear the log by clicking "Clear" at the bottom on the INDI Control Panel
Connect the driver by clicking on "Connect" in the "Main Control" panel

I'm interested in what happens when enumerating controls. This is my own log with a DMK21AU04 webcam (note it will be output most recent log first):

2017-07-05T07:20:07: xioctl: ioctl 0xC0445624/VIDIOC_QUERYCTRL received errno 22 (Invalid argument) 
2017-07-05T07:20:07: Adding Exposure (Absolute) -- min: 1 max: 36000000 step: 1 value: 0 
2017-07-05T07:20:07: Adding menu  Exposure, Auto (item 3 set) 
2017-07-05T07:20:07: Adding menu item Aperture Priority Mode MENU000_OPT001 item 1 index 3 
2017-07-05T07:20:07: xioctl: ioctl 0xC02C5625/VIDIOC_QUERYMENU received errno 22 (Invalid argument) 
2017-07-05T07:20:07: Adding menu item Manual Mode MENU000_OPT000 item 0 index 1 
2017-07-05T07:20:07: xioctl: ioctl 0xC02C5625/VIDIOC_QUERYMENU received errno 22 (Invalid argument) 
2017-07-05T07:20:07: Adding Gain -- min: 260 max: 1023 step: 1 value: 0 
2017-07-05T07:20:07: Adding Gamma -- min: 1 max: 500 step: 1 value: 0 
2017-07-05T07:20:07: Adding Brightness -- min: 0 max: 63 step: 1 value: 0 
2017-07-05T07:20:07: xioctl: ioctl 0xC0445624/VIDIOC_QUERYCTRL received errno 22 (Invalid argument) 
2017-07-05T07:20:07: xioctl: ioctl 0xC02C5625/VIDIOC_QUERYMENU received errno 22 (Invalid argument) 
2017-07-05T07:20:07: xioctl: ioctl 0xC02C5625/VIDIOC_QUERYMENU received errno 22 (Invalid argument) 
2017-07-05T07:20:07: Found intial Input "Camera 1", Format "8-bit Greyscale", Size 640x480, Frame interval 1/30s

Logs "errno 22" are not real errors in that case, but mark the end of an enumeration. In my case, the absolute exposure is available, with min/max set to [1,36000000].

geehalel commented 7 years ago

Hi, Tested with latest libindi/kstars from git, that works fine on my x86_64 ubuntu 16.04 and a logitech c270. I've looked at the #242 issue and I would suggest to change line 1473 in v4l2_driver.cpp to test for "Auto Exposure" too: if (!strcmp(Options[i].label, "Exposure, Auto") || !strcmp(Options[i].label, "Auto Exposure")) ManualExposureSP = Options + i; This may explain the ticks remaining at 0,0 as the "Exposure Absolute" v4l2 control remains read only as long as you are in autoexposure mode. We may look if this is not a uvc/kernel change or just concerns the raspberry camera. @tomn46037 your logitech webcam is limited to 1 second exposure (10000 ticks), so 3 seconds is too much.

knro commented 7 years ago

There was also an issue in reporting limits, it wasn't diving by 10000 to convert to seconds. I also applied Jean-Luc suggestion to master.

knro commented 7 years ago

Any updates on this issue?

rtolesnikov commented 7 years ago

I built the latest master branch and would like to provide a test report. It appears that the fixes provide correct reports and I'm now able to take exposures between 0 and 1 sec.

However, exposures over 1 sec are rejected with the following message: Failed 2.000-second manual exposure, out of device bounds [0.000,1.000].

This is consistent with the limits set during driver initialization: Adding Exposure Time, Absolute -- min: 1 max: 10000 step: 1 value: 1

So the remaining question is: why is V4L2 driver is reading max exposure as 1 sec instead of the real capability of the RPi CAM, which is around 6 sec?

Log attached for reference: indi_v4l2_ccd_03-45-24.txt

rtolesnikov commented 7 years ago

I performed an additional experiment by forcing the Absolute Exposure max, as found during enumeration, to be much larger. This enabled me to at least move past rejection as noted above. The interesting thing, however, is that the video driver appears to stop exposure after ~1.0 seconds, regardless of the requested duration. You may notice in the attached log that the 1-sec exposure stopped 0.030 sec before completion, as expected. however, a 2-sec was terminated 1.048 sec prior to finish; 5 sec - 4.038, etc. So when the driver reports 1 sec max, it does enforce it. indi_v4l2_ccd_05-59-40.txt

I find it interesting that I didn't see any reports on the web about successfully using Raspberry Pi CAM with INDI for long (relatively, of course) exposure setup. Maybe it's the end of that road.

That makes me curious whether an RPi CAM specific INDI driver may be needed that could perhaps be based on the picamera library, or its underlying infrastructure. I have written my own Python scripts to take long exposure captures, but INDI/EKOS integration was what I wanted to accomplish.

TallFurryMan commented 7 years ago

Here are interesting threads with interventions from users 6by9 and jamesh on the RPi forum:

6 seconds using an updated firmware of the standard cam, command line here: https://www.raspberrypi.org/forums/viewtopic.php?f=43&t=89476
the thread opened by gonzothegreat on the NOIR cam: https://www.raspberrypi.org/forums/viewtopic.php?t=82306
https://www.raspberrypi.org/forums/viewtopic.php?f=43&t=61445

knro commented 7 years ago

I take it the Pi CAM still cannot be used with INDI now?

knro commented 6 years ago

So what's the status of Pi CAM? Anyone knows?!

james-lan commented 6 years ago

When I played with it recently, it would work barely, but the v4l interface doesn't do a lot of the things it should compared to using the raspistill in terms of light gathering. Biggest limitation is 1 sec with the v4l driver.

This is by far the best one I've used, in terms of being able to see anything (granted using a 75mm f/1.5 camera, as an eypiece it'd do better): https://github.com/Gordon999/Pi-AutoGuider/blob/master/PiAGr10f4.py

knro commented 6 years ago

So is this a limitation with the V4L2 interface or with the INDI driver itself?

TallFurryMan commented 6 years ago

I believe the kernel driver taking the PiCAM in charge is probably providing "safe" information for the device. If there is a fix to do in that context, that is certainly out of scope for INDI. However, someone on the forum was recently looking for co-workers in order to build an INDI driver over raspistill. This makes sense for this small-audience device (don't get me wrong on "small-audience"!).

nlooije commented 6 years ago

@tomn46037 @knro @james-lan slightly off-topic but i hope thats ok: Is it normal behaviour the guide module is not activated in Ekos when selecting V4L2 CCD for guider in the profile for a Pi camera v2? The driver loads fine and i am able to stream from the camera, but no guide module is present in the menubar of Ekos. If this is not normal behaviour i will open an issue for it.

knro commented 6 years ago

Well, Ekos is not INDI, so this is not an INDI issue. For EKos/KStars problems, file them at https://bugs.kde.org

TallFurryMan commented 6 years ago

Hello, I believe PR #664 should be very interesting for this issue. Could you test it using your Raspberry Pi camera please? Configure "Stack" to "Additive" and try a long exposure. Code is totally not optimized, so that will probably need rework, but the results should be there.

TallFurryMan commented 6 years ago

It's possible that because that camera is actually providing an absolute exposure control, the PR would need additional work to disable the exposure. Let's see.

knro commented 5 years ago

Hello, is there any update on this?

Astro-Otter-Space commented 4 years ago

I am also interested, i just bought a camera NoIR for my RPy and i woul like tu use it with INDI/Ekos.

Astro-Otter-Space commented 4 years ago

@knro have you try https://www.indilib.org/raspberry-pi/raspberry-pi-camera.html ?

knro commented 4 years ago

Right, so this would be used instead of V4L2.

indilib / indi

Problems with v4l2/Raspberry PI Camera and exposure times #281