search

indigo-astronomy / indigo

INDIGO is a system of standards and frameworks for multiplatform and distributed astronomy software development designed to scale with your needs.
http://www.indigo-astronomy.org
Other
146 stars 64 forks source link

spurious fatal err with asi ccd running indigo in a NanoPi M4 #82

Closed agnunez closed 5 years ago

agnunez commented 5 years ago

I managed to use indigo in a NanoPi M4 for some time both with kstars and/or pyclient from a linux pc for some time without a glitch (continuous captures for two days). Suddlently, capture from kstars, or from pyclient issue an error. I can not recover from it by booting both client and server. Logs below indicate NanoPi M4 server (armv7): 21:31:45.211090 indigo_server: Driver indigo_ccd_asi 2.0.0.10 loaded 21:31:45.211398 indigo_server: Resource /ctrl (4338, text/html) added 21:31:45.211439 indigo_server: Resource /resource/angular.min.js (54437, text/javascript) added 21:31:45.211477 indigo_server: Resource /resource/bootstrap.min.js (9843, text/javascript) added 21:31:45.211507 indigo_server: Resource /resource/bootstrap.css (21056, text/css) added 21:31:45.211530 indigo_server: Resource /resource/jquery.min.js (29969, text/javascript) added 21:31:45.211552 indigo_server: Resource /fonts/glyphicons-halflings-regular.ttf (23509, application/x-font-ttf) added 21:31:45.211575 indigo_server: Resource /resource/logo.png (20757, image/png) added 21:31:45.213380 indigo_server: indigo_ccd_asi: ASI SDK v. 1, 13, 0930 21:31:45.213805 indigo_server: indigo_ccd_asi[1688, indigo_ccd_asi]: libusb_hotplug_register_callback -> OK 21:31:45.214418 indigo_server: XML Parser: parser finished 21:31:45.214486 indigo_server: Server attached 21:31:45.214606 indigo_server: Server started on 7624 21:31:46.029839 indigo_server: indigo_ccd_asi: 'ZWO ASI294MC Pro #0' attached 21:32:12.910192 indigo_server: Worker thread started socket = 20 21:32:12.910487 indigo_server: 1 clients 21:32:12.910592 indigo_server: Protocol switched to XML 21:32:14.977199 indigo_server: indigo_ccd_asi[355]: Exposure failed: dev_id = 0 exposure status = 3 21:32:14.991359 indigo_server: XML Parser: parser finished 21:32:14.991810 indigo_server: 0 clients 21:32:14.991849 indigo_server: Worker thread finished

Ubuntu pyclient (i386) log: 2018-11-19 21:32:13,486 new property UPLOAD_SETTINGS for device ZWO ASI294MC Pro #0 2018-11-19 21:32:13,487 new property CCD_MODE for device ZWO ASI294MC Pro #0 2018-11-19 21:32:13,487 new property CCD_EXPOSURE for device ZWO ASI294MC Pro #0 2018-11-19 21:32:13,487 <<<<<<<< Exposure >>>>>>>>> 2018-11-19 21:32:13,487 exposing 0.0001 sec 2018-11-19 21:32:13,490 new property CCD_STREAMING for device ZWO ASI294MC Pro #0 2018-11-19 21:32:13,490 new property CCD_ABORT_EXPOSURE for device ZWO ASI294MC Pro #0 2018-11-19 21:32:13,490 new property CCD_FRAME for device ZWO ASI294MC Pro #0 2018-11-19 21:32:13,490 new property CCD_BINNING for device ZWO ASI294MC Pro #0 2018-11-19 21:32:13,492 new property CCD_OFFSET for device ZWO ASI294MC Pro #0 2018-11-19 21:32:13,492 new property CCD_GAIN for device ZWO ASI294MC Pro #0 2018-11-19 21:32:13,492 new property CCD_FRAME_TYPE for device ZWO ASI294MC Pro #0 2018-11-19 21:32:13,493 new property CCD_IMAGE_FORMAT for device ZWO ASI294MC Pro #0 2018-11-19 21:32:13,493 new property CCD_FILE_PATH for device ZWO ASI294MC Pro #0 2018-11-19 21:32:13,493 new property CCD1 for device ZWO ASI294MC Pro #0 2018-11-19 21:32:13,493 new property CCD_COOLER for device ZWO ASI294MC Pro #0 2018-11-19 21:32:13,493 new property CCD_COOLER_POWER for device ZWO ASI294MC Pro #0 2018-11-19 21:32:13,493 new property CCD_TEMPERATURE for device ZWO ASI294MC Pro #0 2018-11-19 21:32:13,493 new property CCD_FITS_HEADERS for device ZWO ASI294MC Pro #0 2018-11-19 21:32:13,493 new Switch CONNECTION for device ZWO ASI294MC Pro #0 2018-11-19 21:32:13,587 new Number CCD_EXPOSURE for device ZWO ASI294MC Pro #0 2018-11-19 21:32:13,588 new Number CCD_EXPOSURE for device ZWO ASI294MC Pro #0 swig/python detected a memory leak of type 'std::string *', no destructor found. Traceback (most recent call last): File "./indigo.py", line 79, in newMessage self.logger.info("new Message "+ d.messageQueue(m).decode()) AttributeError: 'SwigPyObject' object has no attribute 'decode'

Any hint ? It seems that BLOB returned is not completed.. Thanks

polakovic commented 5 years ago

The problem is this line:

21:32:14.977199 indigo_server: indigo_ccd_asi[355]: Exposure failed: dev_id = 0 exposure status = 3

It is already reported to ZWO, we're waiting for solution. Pls. be patient. Peter

agnunez commented 5 years ago

thanks!

polakovic commented 5 years ago

Pls. can you try short exposure (shorter than 1s)? Thank you, Peter

agnunez commented 5 years ago

Hi, I tried the ASI295MC Pro from 30s to 0.0005s for daylight (is an allsky cam that I need to keep exposing so I am testing with a pyclient script), with this range of exptime it fails after some sub's. Failure frequency increase when I configured RAW16 instead of RAW8 format (trying to recover the 12bit dynamic range). I am also trying now with another Zwo ASI071MC (kstars and pyclient) and I can reproduce same errors. I reformat NanoPiM4 with armbian 5.65 clean with indigo from deb repository. No luck. I also tried ASICAP v1.5 in Win10 with the ASI95MC in RAW16 mode the whole morning without issues, so I guess is not camera itself but the driver. So only NanoPi or indigo_asi_ccd are suspictious now.

When the error happens, is there a way to reset the indigo_asi_ccd driver so I don't loose the night? The skycam is necessary for operation of the observatory and I need a patch solution meanwhile ZWO answer the bug report. Also, if you need any other test, please, let me know. Thanks

polakovic commented 5 years ago

This error is returned from time to time by all cameras and the reason is usually USB bandwidth. In this case no reset is necessary, both camera and software will recover from this situation. If it is not the case here, it is apparently some other reason.

  1. you can try to reduce bandwidth. With control panel (e.g. pointing the browser to http://hostname:7624 if you don't use any client software) navigate to ASI 294 > Advanced > Advanced and reduce BandWidth parameter from default value to something about 50. If it will help you can save the configuration (ASI 294 > Main > Configuration control > Save) and you can restore it anytime later (ASI 294 > Main > Configuration control > Load).

  2. ZWO recommend to check the power supply, camera is sensitive to it and PiNANO may provide not enough power for reliable operation.

Peter

agnunez commented 5 years ago

Bandwidth was 80, so I lowered to 50. Power supply to the camera is provided by an USB3.0 hub and an adittional power supply of 3A, independent of NanoPiM4 power supply (5v 5A in pin headers, not using Type-C connector). I lowered to RAW8 also, to get at least some confidence during night operations, until we find a better solution with RAW16 (preferred for science data). Is there any easy way to generate jpeg (for outreach) besides the RAW16 fits with the same exp subs? In indigo, I can select one or the other for BLOB format, but perhaps an easy library to manage RGGB into jpeg. Thanks (this is offtopic, sorry).

polakovic commented 5 years ago

Not yet, but I work on solution already, maybe you can test it in a few days. It is a pseudo driver called indigo_agent_imager and it is kind of filter between real CCD driver on one side and the client on other side (e.g. web based client). It will provide common services like generating histogram, stretched JPEG preview images, plate solving etc.

In the meantime you can look at this (but I have no personal experience):

https://github.com/jcupitt/libvips

polakovic commented 5 years ago

Any success last night with the camera?

Pls. what kind and version of OS do you use? Is it standard Raspbian?

agnunez commented 5 years ago

Yes, I didn't have any issue after bandwidth reduction. I slow the recording frame rate during the day, when exposure times where shorter, to avoid usb bus saturation. Is there a way to activate auto gain or auto exposure inside the asi driver. I currently have to manually compute histogram and guess best configuration.
I am using now standard Armbiam 5.65 stable, without any further configuration. Before I used rk3399-bionic-4.4-arm64 ubuntu and was also working flawsly until this issue. NanoPiM4 has 4 USB3.0 and 6 cores (2 more than Rpi) and I perhaps estressed too much the asi driver.

rumengb commented 5 years ago

Unfortunately we do not support auto gain. I is not hard to add though.

agnunez commented 5 years ago

Last night it hangs again the pyclient script, and I had to lower bandwidth now to 40. BTW, what's the units of this figure. Gbps,Mbps? or means anything about latency or other parameters. I solved by rebooting the NanoPiM4 as indigo is installed as a systemd service. I'm thinking to code a watchdog to reboot at 1st error.

polakovic commented 5 years ago

It is % of bandwidth. If camera itself will recover from this situation, it may be an option.

agnunez commented 5 years ago

Shorting the usb 3.0 cable corrected the situation