indigo-astronomy / indigo

INDIGO is a system of standards and frameworks for multiplatform and distributed astronomy software development designed to scale with your needs.
http://www.indigo-astronomy.org
Other
146 stars 64 forks source link

Indigo Server: Assertion 'close_nointr(fd) != -EBADF' failed at ../src/basic/fd-util.c:81 #242

Closed agnunez closed 3 years ago

agnunez commented 4 years ago

After several months of working without interruption with an ASI ASI294MC Pro as AllSky cam on a NanoPi M4, I started to get problems: Symthomps: -GAIN parameter do not change during night when activated by an set_property bash script from another computer (has been working for 6 months -The logs shows several disconnections and errors like:

 Assertion 'close_nointr(fd) != -EBADF' failed at ../src/basic/fd-util.c:81, function safe_close(). Aborting.

The python scripts that runs in main server has the following log:

Sep  2 12:20:16 orm-vastro python[2385]: INDI server skycam/7624 disconnected.
Sep  2 12:20:16 orm-vastro python[2385]: 2019-09-02 12:20:16,259 Server disconnected (exit code = -1,skycam:7624)

or

Sep  2 12:20:14 orm-vastro python[2385]: ASI_ADVANCED: newNumberVector with no valid members
Sep  2 12:20:14 orm-vastro python[2385]: 2019-09-02 12:20:14,329 new Number CCD_FRAME for device ZWO ASI294MC Pro #0
Sep  2 12:20:14 orm-vastro python[2385]: ASI_ADVANCED: newNumberVector with no valid members
Sep  2 12:20:14 orm-vastro python[2385]: 2019-09-02 12:20:14,329 new Number CCD_FRAME for device ZWO ASI294MC Pro #0
Sep  2 12:20:14 orm-vastro python[2385]: ASI_ADVANCED: newNumberVector with no valid members
Sep  2 12:20:14 orm-vastro python[2385]: 2019-09-02 12:20:14,330 new property ASI_ADVANCED for device ZWO ASI294MC Pro #0
Sep  2 12:20:14 orm-vastro python[2385]: Dispatch command error(-1):
Sep  2 12:20:14 orm-vastro python[2385]: <setSwitchVector device="ZWO ASI294MC Pro #0" name="CCD_COOLER" state="Ok">
Sep  2 12:20:14 orm-vastro python[2385]:     <oneSwitch name="COOLER_ON">
Sep  2 12:20:14 orm-vastro python[2385]: Off
Sep  2 12:20:14 orm-vastro python[2385]:     </oneSwitch>
Sep  2 12:20:14 orm-vastro python[2385]:     <oneSwitch name="COOLER_OFF">
Sep  2 12:20:14 orm-vastro python[2385]: On
Sep  2 12:20:14 orm-vastro python[2385]:     </oneSwitch>
Sep  2 12:20:14 orm-vastro python[2385]: </setSwitchVector>
Sep  2 12:20:14 orm-vastro python[2385]: Dispatch command error(-1): INDI: Could not find property CCD_TEMPERATURE in ZWO ASI294MC Pro #0
Sep  2 12:20:14 orm-vastro python[2385]: <setNumberVector device="ZWO ASI294MC Pro #0" name="CCD_TEMPERATURE" state="Ok">
Sep  2 12:20:14 orm-vastro python[2385]:     <oneNumber name="CCD_TEMPERATURE_VALUE">
Sep  2 12:20:14 orm-vastro python[2385]: 0
Sep  2 12:20:14 orm-vastro python[2385]:     </oneNumber>
Sep  2 12:20:14 orm-vastro python[2385]: </setNumberVector>
Sep  2 12:20:14 orm-vastro python[2385]: Dispatch command error(-1): INDI: Could not find property CCD_COOLER_POWER in ZWO ASI294MC Pro #0
Sep  2 12:20:14 orm-vastro python[2385]: <setNumberVector device="ZWO ASI294MC Pro #0" name="CCD_COOLER_POWER" state="Ok">
Sep  2 12:20:14 orm-vastro python[2385]:     <oneNumber name="CCD_COOLER_VALUE">
Sep  2 12:20:14 orm-vastro python[2385]: 0
Sep  2 12:20:14 orm-vastro python[2385]:     </oneNumber>
Sep  2 12:20:14 orm-vastro python[2385]: </setNumberVector>

or

Sep  2 12:20:14 orm-vastro python[2385]: No ISwitch 'ASI_HIGHEST_DR' in ZWO ASI294MC Pro #0.ASI_PRESETS
Sep  2 12:20:14 orm-vastro python[2385]: No ISwitch 'ASI_UNITY_GAIN' in ZWO ASI294MC Pro #0.ASI_PRESETS
Sep  2 12:20:14 orm-vastro python[2385]: No ISwitch 'ASI_LOWEST_RN' in ZWO ASI294MC Pro #0.ASI_PRESETS

All this is new, and didn't appears for months. I run apt update with latest indigo version today as it was frozen while working.

Any clue? Thanks

polakovic commented 4 years ago

Rumen, it may be related to recent ASI SDK update.

agnunez commented 4 years ago

Can i rollback the versión?

El lun., 2 sept. 2019 17:17, Peter Polakovic notifications@github.com escribió:

Rumen, it may be related to recent ASI SDK update.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/indigo-astronomy/indigo/issues/242?email_source=notifications&email_token=AAII4PXNNH3ZF3QWUYZCV5LQHU4DNA5CNFSM4IS45KGKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD5WGJ3A#issuecomment-527197420, or mute the thread https://github.com/notifications/unsubscribe-auth/AAII4PUPVESRPFCWRAYSXRLQHU4DNANCNFSM4IS45KGA .

polakovic commented 4 years ago

Sure, find available versions

sudo apt-cache policy indigo

and install the one you want, e.g. like this

sudo apt-get install indigo=2.0-100

agnunez commented 4 years ago

Ok. I changed the NanoPiM4 and installed a raspberry pi 3B+ with raspbian, indigo and connect the ASI294MC through USB 2.0 now (was 3.0 before). It has been 3 day without issues but this is the second night that stop working leaving this error in syslog:

Sep 13 04:02:49 raspberrypi indigo.sh[319]: 04:02:49.345212 indigo_server: indigo_ccd_asi[351, asi_read_pixels, 0x653ff3d0]: Exposure failed: dev_id = 0 exposure status = 3

I don't know if it is related to former error, that do not appear anymore, or I should open another issue.

polakovic commented 4 years ago

It is USB bandwidth related error, try to reduce it a little bit. Peter

agnunez commented 4 years ago

ok. will try with less.. but still appearing.. Sep 15 02:52:11 raspberrypi indigo.sh[346]: 02:52:11.228476 indigo_server: indigo_ccd_asi[343, asi_read_pixels, 0x643fe3d0]: ASIGetDataAfterExp(0) = 11 would be nice to detect and recover.. I have to reset usb each time..

polakovic commented 4 years ago

I doubt it is possible to recover without reconnecting :(

agnunez commented 4 years ago

Well, this is my not-very-ellegant-but-effective solution to keep the camera operational. I'm recovering automagically with a watchdog script that filter syslog and find "asi_read_pixel" error: pi@raspberrypi:~ $ cat watchdog tail -F /var/log/syslog | while read a; do echo "$a" | awk '/asi_read_pixels/ { print $1,$2,$3,"exposure failed" ; system("/home/pi/resetall") }' ; done

"resetall" scripts, stop indigo client and servers (in different cpu's in my case): `pi@raspberrypi:~ $ cat resetall

!/bin/bash

ssh sysadmin@orm-vastro "sudo systemctl stop skycam" sudo systemctl stop indigo /home/pi/usbidzwocam | xargs /home/pi/resetcam sleep 5 sudo systemctl start indigo sleep 10 ssh sysadmin@orm-vastro "sudo systemctl start skycam" `

usbidzwocam script guesses the Bus and Device usb enumerated by filtering lsusb command with usb id "03c3" of ASI294MC in my case, and pass them as argument to an standard usbreset code:

`pi@raspberrypi:~ $ cat usbidzwocam

!/bin/bash

lsusb | grep 03c3 | sed -n -e 's/Bus //p' | sed -n -e 's/Device //p' | sed -n -e 's/: ID.*$//p' pi@raspberrypi:~ $ cat resetcam

!/bin/bash

sudo ./usbreset /dev/bus/usb/$1/$2 ` the usbreset code that I use (not mine) is:

`pi@raspberrypi:~ $ cat usbreset.c / usbreset -- send a USB port reset to a USB device /

include

include

include

include

include <sys/ioctl.h>

include <linux/usbdevice_fs.h>

int main(int argc, char *argv) { const char filename; int fd; int rc;

    if (argc != 2) {
            fprintf(stderr, "Usage: usbreset device-filename\n");
            return 1;
    }
    filename = argv[1];

    fd = open(filename, O_WRONLY);
    if (fd < 0) {
            perror("Error opening output file");
            return 1;
    }

    printf("Resetting USB device %s\n", filename);
    rc = ioctl(fd, USBDEVFS_RESET, 0);
    if (rc < 0) {
            perror("Error in ioctl");
            return 1;
    }
    printf("Reset successful\n");

    close(fd);
    return 0;

} `

This solution is working and allows to keep operative my allsky camera. My proposal for the indigo asi driver, is to handle usb bandwidth errors in this way (as an option) and allows to keep working without having the user to manually unplug/plug the camera to reset , specially important for remote robotic telescopes installations.

Thanks @agnuca

rumengb commented 4 years ago

Hi this solution may be OK, but it is Linux specific. This will not work on mac.