scriptorron / indi_pylibcamera

INDI libcamera driver made in Python
MIT License
11 stars 3 forks source link

instability problem #62

Open gmke opened 7 months ago

gmke commented 7 months ago

I'm running indi_pylibcamera on a RPi Zero 2 W with Raspbian bookworm 64bit and a RPI HQ camera. Intended use is as an Allsky camera with indi-allsky. For various reasons, indi-allsky runs on a different server, the Zero 2 W only runs indiserver (v2.0.6) with the indi_pylibcamera driver. The driver was installed using pip but without a venv. Since about a week, this setup runs on my desk for test purposes. Unfortunately, the driver frequently crashes. Usually I get a run time of a few hours, the current record was about 24h. The error condition in the syslog looks always like this:

2024-02-21T21:46:54.717154+01:00 haustuer indiserver[667]: 2024-02-21T20:46:54: Driver indi_pylibcamera: indi_pylibcamera.indidevice-INFO- got exposed frame 2024-02-21T21:46:55.031060+01:00 haustuer indiserver[667]: 2024-02-21T20:46:55: Driver indi_pylibcamera: indi_pylibcamera.indidevice-INFO- preparing frame as BLOB: 24667200 bytes 2024-02-21T21:46:55.032029+01:00 haustuer indiserver[667]: 2024-02-21T20:46:55: Driver indi_pylibcamera: indi_pylibcamera.indidevice-INFO- sending BLOB 2024-02-21T21:47:09.792713+01:00 haustuer indiserver[667]: 2024-02-21T20:47:09: Driver indi_pylibcamera: indi_pylibcamera.indidevice-INFO- exposure settings: CameraSettings: FastExposure=False, DoRaw=True, ProcSize=(4056, 3040), RawMode={'label': '4056x3040 RGGB 12bit', 'size': (4056, 3040), 'true_size': (4056, 3040), 'camera_format': 'SRGGB12', 'bit_depth': 12, 'binning': (1, 1)}, CameraControls={'ExposureTime': 7827, 'AnalogueGain': 22.0, 'AeEnable': False, 'AeConstraintMode': <AeConstraintModeEnum.Normal: 0>, 'AeExposureMode': <AeExposureModeEnum.Normal: 0>, 'AeMeteringMode': <AeMeteringModeEnum.CentreWeighted: 0>, 'AwbEnable': False, 'AwbMode': <AwbModeEnum.Auto: 0>, 'Brightness': 0.0, 'ColourGains': (2.0, 2.0), 'Contrast': 1.0, 'ExposureValue': 0.0, 'NoiseReductionMode': <NoiseReductionModeEnum.Off: 0>, 'Saturation': 1.0, 'Sharpness': 0.0} 2024-02-21T21:47:10.566693+01:00 haustuer indiserver[667]: 2024-02-21T20:47:10: Driver indi_pylibcamera: indi_pylibcamera.indidevice-INFO- got exposed frame 2024-02-21T21:47:10.599487+01:00 haustuer indiserver[667]: 2024-02-21T20:47:10: Driver indi_pylibcamera: [5:28:24.978519301] [788] #033[1;31mERROR #033[1;37mCamera #033[1;34mcamera.cpp:675 #033[0mCamera in Configured state trying queueRequest() requiring state Running 2024-02-21T21:47:11.155491+01:00 haustuer indiserver[667]: 2024-02-21T20:47:11: Driver indi_pylibcamera: indi_pylibcamera.indidevice-INFO- preparing frame as BLOB: 24667200 bytes 2024-02-21T21:47:11.166201+01:00 haustuer indiserver[667]: 2024-02-21T20:47:11: Driver indi_pylibcamera: Exception during process_requests() 2024-02-21T21:47:11.166657+01:00 haustuer indiserver[667]: 2024-02-21T20:47:11: Driver indi_pylibcamera: Traceback (most recent call last): 2024-02-21T21:47:11.168027+01:00 haustuer indiserver[667]: 2024-02-21T20:47:11: Driver indi_pylibcamera: File "/usr/local/lib/python3.11/dist-packages/picamera2/previews/null_preview.py", line 85, in handle_request 2024-02-21T21:47:11.168486+01:00 haustuer indiserver[667]: 2024-02-21T20:47:11: Driver indi_pylibcamera: picam2.process_requests(self) 2024-02-21T21:47:11.168792+01:00 haustuer indiserver[667]: 2024-02-21T20:47:11: Driver indi_pylibcamera: File "/usr/local/lib/python3.11/dist-packages/picamera2/picamera2.py", line 1271, in process_requests 2024-02-21T21:47:11.169181+01:00 haustuer indiserver[667]: 2024-02-21T20:47:11: Driver indi_pylibcamera: display_request.release() 2024-02-21T21:47:11.169477+01:00 haustuer indiserver[667]: 2024-02-21T20:47:11: Driver indi_pylibcamera: File "/usr/local/lib/python3.11/dist-packages/picamera2/request.py", line 138, in release 2024-02-21T21:47:11.169795+01:00 haustuer indiserver[667]: 2024-02-21T20:47:11: Driver indi_pylibcamera: self.picam2.camera.queue_request(self.request) 2024-02-21T21:47:11.170045+01:00 haustuer indiserver[667]: 2024-02-21T20:47:11: Driver indi_pylibcamera: RuntimeError: Failed to queue request: Permission denied 2024-02-21T21:47:11.170501+01:00 haustuer indiserver[667]: 2024-02-21T20:47:11: Driver indi_pylibcamera: Exception in thread Thread-3 (thread_func): 2024-02-21T21:47:11.205746+01:00 haustuer indiserver[667]: 2024-02-21T20:47:11: Driver indi_pylibcamera: Traceback (most recent call last): 2024-02-21T21:47:11.247305+01:00 haustuer indiserver[667]: 2024-02-21T20:47:11: Driver indi_pylibcamera: File "/usr/lib/python3.11/threading.py", line 1038, in _bootstrap_inner 2024-02-21T21:47:11.248021+01:00 haustuer indiserver[667]: 2024-02-21T20:47:11: Driver indi_pylibcamera: indi_pylibcamera.indidevice-INFO- sending BLOB 2024-02-21T21:47:11.941845+01:00 haustuer indiserver[667]: 2024-02-21T20:47:11: Driver indi_pylibcamera: self.run() 2024-02-21T21:47:11.945775+01:00 haustuer indiserver[667]: 2024-02-21T20:47:11: Driver indi_pylibcamera: File "/usr/lib/python3.11/threading.py", line 975, in run 2024-02-21T21:47:11.946330+01:00 haustuer indiserver[667]: 2024-02-21T20:47:11: Driver indi_pylibcamera: self._target(*self._args, **self._kwargs) 2024-02-21T21:47:11.946625+01:00 haustuer indiserver[667]: 2024-02-21T20:47:11: Driver indi_pylibcamera: File "/usr/local/lib/python3.11/dist-packages/picamera2/previews/null_preview.py", line 29, in thread_func 2024-02-21T21:47:11.949725+01:00 haustuer indiserver[667]: 2024-02-21T20:47:11: Driver indi_pylibcamera: callback(picam2) 2024-02-21T21:47:11.950282+01:00 haustuer indiserver[667]: 2024-02-21T20:47:11: Driver indi_pylibcamera: File "/usr/local/lib/python3.11/dist-packages/picamera2/previews/null_preview.py", line 85, in handle_request 2024-02-21T21:47:11.950407+01:00 haustuer indiserver[667]: 2024-02-21T20:47:11: Driver indi_pylibcamera: picam2.process_requests(self) 2024-02-21T21:47:11.950522+01:00 haustuer indiserver[667]: 2024-02-21T20:47:11: Driver indi_pylibcamera: File "/usr/local/lib/python3.11/dist-packages/picamera2/picamera2.py", line 1271, in process_requests 2024-02-21T21:47:11.953692+01:00 haustuer indiserver[667]: 2024-02-21T20:47:11: Driver indi_pylibcamera: display_request.release() 2024-02-21T21:47:11.954203+01:00 haustuer indiserver[667]: 2024-02-21T20:47:11: Driver indi_pylibcamera: File "/usr/local/lib/python3.11/dist-packages/picamera2/request.py", line 138, in release 2024-02-21T21:47:11.954469+01:00 haustuer indiserver[667]: 2024-02-21T20:47:11: Driver indi_pylibcamera: self.picam2.camera.queue_request(self.request) 2024-02-21T21:47:11.954696+01:00 haustuer indiserver[667]: 2024-02-21T20:47:11: Driver indi_pylibcamera: RuntimeError: Failed to queue request: Permission denied 2024-02-21T21:47:23.920465+01:00 haustuer indiserver[667]: 2024-02-21T20:47:23: Driver indi_pylibcamera: indi_pylibcamera.indidevice-INFO- exposure settings: CameraSettings: FastExposure=False, DoRaw=True, ProcSize=(4056, 3040), RawMode={'label': '4056x3040 RGGB 12bit', 'size': (4056, 3040), 'true_size': (4056, 3040), 'camera_format': 'SRGGB12', 'bit_depth': 12, 'binning': (1, 1)}, CameraControls={'ExposureTime': 7827, 'AnalogueGain': 22.0, 'AeEnable': False, 'AeConstraintMode': <AeConstraintModeEnum.Normal: 0>, 'AeExposureMode': <AeExposureModeEnum.Normal: 0>, 'AeMeteringMode': <AeMeteringModeEnum.CentreWeighted: 0>, 'AwbEnable': False, 'AwbMode': <AwbModeEnum.Auto: 0>, 'Brightness': 0.0, 'ColourGains': (2.0, 2.0), 'Contrast': 1.0, 'ExposureValue': 0.0, 'NoiseReductionMode': <NoiseReductionModeEnum.Off: 0>, 'Saturation': 1.0, 'Sharpness': 0.0}

It can be resolved by restarting the indiserver systemctl --user restart indiserver.service and it will happily run again for a few hours. I know that this is difficult to diagnose and solve. Short term solution might be an automatic restart of the driver itself if he somehow diagnoses the error condition. Maybe I have to set an option in the ini file. I havn't created an ini file yet, lacking an example. If I can be of any further help, I'm willing to provide any info, reinstall or whatever needs to be done :-)

CS, Markus

scriptorron commented 7 months ago

Hi Markus,

the first error message comes from libcamera:

2024-02-21T21:47:10.599487+01:00 haustuer indiserver[667]: 2024-02-21T20:47:10: Driver indi_pylibcamera: [5:28:24.978519301] [788] #33[1;31mERROR # 33[1;37mCamera #33[1;34mcamera.cpp:675 #33[0mCamera in Configured state trying queueRequest() requiring state Running

That leaded to a RuntimeError in a thread started and hosted by the picamera2 library. It seems this exception is not raised up to the indi_pylibcamera driver (your log looks like the driver continues to start exposures). I will try to find a way to detect the exception. Than I can implement something to recover from the error.

Which version of the python-picamera2 library do you use? Please forward me the output of

apt list --installed | grep picamera2

Do you know a way to reproduce the error without waiting for hours?

Thank you, Ronald

gmke commented 7 months ago

Hi Ronald,

python3-picamera2/stable,stable,now 0.3.17-1 all [installiert]

I could uninstall it and use the libcamera build provided by Aaron with indi-allsky.

it happened this morning again, unfortunately I do not know how to force it. The Zero 2 does nothing else and sits happily on my desk with the connected PiHQ camera.

``2024-02-22T12:52:31.739573+01:00 haustuer indiserver[3521]: 2024-02-22T11:52:31: Driver indi_pylibcamera: indi_pylibcamera.indidevice-INFO- sending BLOB 2024-02-22T12:52:46.347890+01:00 haustuer indiserver[3521]: 2024-02-22T11:52:46: Driver indi_pylibcamera: indi_pylibcamera.indidevice-INFO- exposure settings: CameraSettings: FastExposure=False, DoRaw=True, ProcSize=(4056, 3040), RawMode={'label': '4056x3040 RGGB 12bit', 'size': (4056, 3040), 'true_size': (4056, 3040), 'camera_format': 'SRGGB12', 'bit_depth': 12, 'binning': (1, 1)}, CameraControls={'ExposureTime': 81176, 'AnalogueGain': 1.0, 'AeEnable': False, 'AeConstraintMode': <AeConstraintModeEnum.Normal: 0>, 'AeExposureMode': <AeExposureModeEnum.Normal: 0>, 'AeMeteringMode': <AeMeteringModeEnum.CentreWeighted: 0>, 'AwbEnable': False, 'AwbMode': <AwbModeEnum.Auto: 0>, 'Brightness': 0.0, 'ColourGains': (2.0, 2.0), 'Contrast': 1.0, 'ExposureValue': 0.0, 'NoiseReductionMode': <NoiseReductionModeEnum.Off: 0>, 'Saturation': 1.0, 'Sharpness': 0.0} 2024-02-22T12:52:47.215235+01:00 haustuer indiserver[3521]: 2024-02-22T11:52:47: Driver indi_pylibcamera: indi_pylibcamera.indidevice-INFO- got exposed frame 2024-02-22T12:52:47.244501+01:00 haustuer indiserver[3521]: 2024-02-22T11:52:47: Driver indi_pylibcamera: [20:34:01.623855968] [3544] #033[1;31mERROR #033[1;37mCamera #033[1;34mcamera.cpp:675 #033[0mCamera in Configured state trying queueRequest() requiring state Running 2024-02-22T12:52:47.792858+01:00 haustuer indiserver[3521]: 2024-02-22T11:52:47: Driver indi_pylibcamera: indi_pylibcamera.indidevice-INFO- preparing frame as BLOB: 24667200 bytes 2024-02-22T12:52:47.805494+01:00 haustuer indiserver[3521]: 2024-02-22T11:52:47: Driver indi_pylibcamera: indi_pylibcamera.indidevice-INFO- sending BLOB 2024-02-22T12:52:47.813633+01:00 haustuer indiserver[3521]: 2024-02-22T11:52:47: Driver indi_pylibcamera: Exception during process_requests() 2024-02-22T12:52:47.814132+01:00 haustuer indiserver[3521]: 2024-02-22T11:52:47: Driver indi_pylibcamera: Traceback (most recent call last): 2024-02-22T12:52:47.814253+01:00 haustuer indiserver[3521]: 2024-02-22T11:52:47: Driver indi_pylibcamera: File "/usr/local/lib/python3.11/dist-packages/picamera2/previews/null_preview.py", line 85, in handle_request 2024-02-22T12:52:47.815747+01:00 haustuer indiserver[3521]: 2024-02-22T11:52:47: Driver indi_pylibcamera: picam2.process_requests(self) 2024-02-22T12:52:47.816037+01:00 haustuer indiserver[3521]: 2024-02-22T11:52:47: Driver indi_pylibcamera: File "/usr/local/lib/python3.11/dist-packages/picamera2/picamera2.py", line 1271, in process_requests 2024-02-22T12:52:47.816365+01:00 haustuer indiserver[3521]: 2024-02-22T11:52:47: Driver indi_pylibcamera: display_request.release() 2024-02-22T12:52:47.816484+01:00 haustuer indiserver[3521]: 2024-02-22T11:52:47: Driver indi_pylibcamera: File "/usr/local/lib/python3.11/dist-packages/picamera2/request.py", line 138, in release 2024-02-22T12:52:47.819097+01:00 haustuer indiserver[3521]: 2024-02-22T11:52:47: Driver indi_pylibcamera: self.picam2.camera.queue_request(self.request) 2024-02-22T12:52:47.819363+01:00 haustuer indiserver[3521]: 2024-02-22T11:52:47: Driver indi_pylibcamera: RuntimeError: Failed to queue request: Permission denied 2024-02-22T12:52:47.820127+01:00 haustuer indiserver[3521]: 2024-02-22T11:52:47: Driver indi_pylibcamera: Exception in thread Thread-3 (thread_func): 2024-02-22T12:52:48.260608+01:00 haustuer indiserver[3521]: 2024-02-22T11:52:48: Driver indi_pylibcamera: Traceback (most recent call last): 2024-02-22T12:52:48.359300+01:00 haustuer indiserver[3521]: 2024-02-22T11:52:48: Driver indi_pylibcamera: File "/usr/lib/python3.11/threading.py", line 1038, in _bootstrap_inner 2024-02-22T12:52:48.556543+01:00 haustuer indiserver[3521]: 2024-02-22T11:52:48: Driver indi_pylibcamera: self.run() 2024-02-22T12:52:48.556922+01:00 haustuer indiserver[3521]: 2024-02-22T11:52:48: Driver indi_pylibcamera: File "/usr/lib/python3.11/threading.py", line 975, in run 2024-02-22T12:52:48.560933+01:00 haustuer indiserver[3521]: 2024-02-22T11:52:48: Driver indi_pylibcamera: self._target(*self._args, **self._kwargs) 2024-02-22T12:52:48.568943+01:00 haustuer indiserver[3521]: 2024-02-22T11:52:48: Driver indi_pylibcamera: File "/usr/local/lib/python3.11/dist-packages/picamera2/previews/null_preview.py", line 29, in thread_func 2024-02-22T12:52:48.569426+01:00 haustuer indiserver[3521]: 2024-02-22T11:52:48: Driver indi_pylibcamera: callback(picam2) 2024-02-22T12:52:48.569686+01:00 haustuer indiserver[3521]: 2024-02-22T11:52:48: Driver indi_pylibcamera: File "/usr/local/lib/python3.11/dist-packages/picamera2/previews/null_preview.py", line 85, in handle_request 2024-02-22T12:52:48.571116+01:00 haustuer indiserver[3521]: 2024-02-22T11:52:48: Driver indi_pylibcamera: picam2.process_requests(self) 2024-02-22T12:52:48.571391+01:00 haustuer indiserver[3521]: 2024-02-22T11:52:48: Driver indi_pylibcamera: File "/usr/local/lib/python3.11/dist-packages/picamera2/picamera2.py", line 1271, in process_requests 2024-02-22T12:52:48.572562+01:00 haustuer indiserver[3521]: 2024-02-22T11:52:48: Driver indi_pylibcamera: display_request.release() 2024-02-22T12:52:48.572825+01:00 haustuer indiserver[3521]: 2024-02-22T11:52:48: Driver indi_pylibcamera: File "/usr/local/lib/python3.11/dist-packages/picamera2/request.py", line 138, in release 2024-02-22T12:52:48.573637+01:00 haustuer indiserver[3521]: 2024-02-22T11:52:48: Driver indi_pylibcamera: self.picam2.camera.queue_request(self.request) 2024-02-22T12:52:48.573870+01:00 haustuer indiserver[3521]: 2024-02-22T11:52:48: Driver indi_pylibcamera: RuntimeError: Failed to queue request: Permission denied 2024-02-22T12:53:18.868336+01:00 haustuer indiserver[3521]: 2024-02-22T11:53:18: Driver indi_pylibcamera: indi_pylibcamera.indidevice-INFO- exposure settings: CameraSettings: FastExposure=False, DoRaw=True, ProcSize=(4056, 3040), RawMode={'label': '4056x3040 RGGB 12bit', 'size': (4056, 3040), 'true_size': (4056, 3040), 'camera_format': 'SRGGB12', 'bit_depth': 12, 'binning': (1, 1)}, CameraControls={'ExposureTime': 69027, 'AnalogueGain': 1.0, 'AeEnable': False, 'AeConstraintMode': <AeConstraintModeEnum.Normal: 0>, 'AeExposureMode': <AeExposureModeEnum.Normal: 0>, 'AeMeteringMode': <AeMeteringModeEnum.CentreWeighted: 0>, 'AwbEnable': False, 'AwbMode': <AwbModeEnum.Auto: 0>, 'Brightness': 0.0, 'ColourGains': (2.0, 2.0), 'Contrast': 1.0, 'ExposureValue': 0.0, 'NoiseReductionMode': <NoiseReductionModeEnum.Off: 0>, 'Saturation': 1.0, 'Sharpness': 0.0}

CS, Markus

scriptorron commented 7 months ago

I can reproduce the error with my Pi Zero WH, running 32bit Bullseye and picamera2 0.3.12-2.

scriptorron commented 7 months ago

The issue is difficult to debug. A few days ago I could reproduce the error in less than an hour. Than I added some more log messages to support debugging and now the driver runs without error since nearly a day. I wish it would fail more often.

gmke commented 7 months ago

yeah, I restarted/power cycled my test setup last Saturday and no hickup since then. Strange, the week before it stopped every few hours.

CS, Markus

ok, about an hour later after my comment above, the next failure happened...

scriptorron commented 7 months ago

I had only one error after adding more debug outputs. Maybe I am mislead by an accident, but the error happened after closing the camera. To allow an immediate aborting of a running exposure I used a stop-command which bypasses the camera event loop. Maybe that leaded to the error in the camera queue. I modified the code now to use the recommended stop-command during normal exposures. The risky immediate stop command is now used only when you abort a running exposure. It is still needed to use the risky command because it is the only way I know to abort a long term exposure without waiting until its end.

I released v2.6.0 which hopefully fixed the issue. Would be very nice when you give me feedback if it works now reliable.

Regards, Ronald

gmke commented 3 months ago

Hi Ronald, sorry for the late answer. I restarted my trials a few weeks ago with a new RPi5 (8GB) intended for another project. With this system, I was able to use indi_pycamera for now more than 10days without a problem. Encouraged by this success, I created a new image (Raspbian Lite, bookworm) for the Zero 2 W and configured it exactly the same using the build scripts in the misc folder of indi-allsky for indi and libcamera. I can use libcamera-hello or qcam without a problem. But with indi_pylibcamera, the system now stops working after one or two pictures. Symptoms are that the indiserver process runs constantly at 100% and is almost unstoppable. Memory usage is still below 400MB and only about 100MB swap space is used. I'm a bit lib camera now. Is it really necessary to use a RPi4/5 for such a trivial task. IfI can try further stuff or send logs, let me know.

CS, Markus

scriptorron commented 3 months ago

Hi Markus,

it would be a shame if it does not work on a Pi Zero! I will try to replicate your setup. On one of my telescopes I have a Pi Zero W with a HQ camera. It is not a Zero 2 W but this hopefully does not make a difference. I don't want to damage my installation and ordered a SD card (should arrive beginning of next week). When it arrives I will setup a fresh Bookworm and install indi-allsky. Do I understand right, you did misc/build_indi.sh and misc/build_libcamera.sh? Have you first installed libcamera and then build indi?

Regards, Ronald

gmke commented 2 months ago

Hi Ronald, I usually first install indi and then libcamera using the build scripts. But I have tried both the system provided libcamera packages and the self built ones and it makes no difference. Both work with libcamera-hello and qcam (you have to enable the build) without a problem. Btw I have made another test with a RPi4 (4GB) and the RPi global shutter camera, works without problems. Only my Zero W and Zero 2 W have the instability issue.

CS, Markus

scriptorron commented 2 months ago

I have seen on the indi-allsky page that the requirement for Pi Zero 2 (and Pi Zero 2) is Bullseye and not Bookworm. As far as I understand you used Bookworm. I will test Bullseye 32bit. Hopefully I tell you results tomorrow.

gmke commented 2 months ago

Actually, I have tried all variants, 32bit Bullseye and 32/64bit bookworm. Makes no difference. Currently I have installed Ubuntu 24.4 64bit and it currently slowly compiles the libcamera stuff. Did not work with the system provided packages.

CS, Markus

gmke commented 2 months ago

indi_pylibcamera does not work with ubuntu 24.4 and self build libcamera or the system packages. rpicam-hello and qcam do work.

Traceback (most recent call last): 2024-07-09T10:09:08: Driver indi_pylibcamera: File "/home/markus/venv_indi_pylibcamera/bin/indi_pylibcamera", line 5, in 2024-07-09T10:09:08: Driver indi_pylibcamera: from indi_pylibcamera.indi_pylibcamera import main 2024-07-09T10:09:08: Driver indi_pylibcamera: File "/home/markus/venv_indi_pylibcamera/lib/python3.12/site-packages/indi_pylibcamera/indi_pylibcamera.py", line 12, in 2024-07-09T10:09:08: Driver indi_pylibcamera: from picamera2 import Picamera2 2024-07-09T10:09:08: Driver indi_pylibcamera: File "/home/markus/venv_indi_pylibcamera/lib/python3.12/site-packages/picamera2/init.py", line 3, in 2024-07-09T10:09:08: Driver indi_pylibcamera: import libcamera 2024-07-09T10:09:08: Driver indi_pylibcamera: ModuleNotFoundError: No module named 'libcamera'

CS, Markus

scriptorron commented 2 months ago

I newer tested Ubuntu. But the error message looks like the libcameralibrary is missing. As far as I know this can be installed with

sudo apt install -y python3-libcamera

Hopefully this works also in Ubuntu.

I try to setup indi-allsky on my Pi Zero W. It is compiling the indi library now since 8 hours. Maybe it will finish over night.

gmke commented 2 months ago

the package is available and installed but it does not resolve the error. That was the reason I built libcamera using the build script inside the venv. I'm out of ideas. Ubuntu was an experiment. Last hope would be going back to bullseye if you are successful.

CS, Markus

gmke commented 2 months ago

ok, went back to Rasbian bullseye 32bit. Fresh install, full-upgrade, installed only indi-bin from the standard repo and python3 venv. Increased swap size to 1000MB and started the indiserver. It is running well since about a day now. Let's hope for more stability. Will keep you posted.

CS, Markus

scriptorron commented 2 months ago

I have now indi-allsky installed on my Pi Zero W. Afte reboot it runs automatically the indiserver:

/usr/local/bin/indiserver -p 7624 indi_simulator_telescope indi_simulator_ccd

I installed indi-pylibcamera system wide. Do you know how I can make indi-allsky to start the new camera driver? I do not find a setting in the GUI. I tried this: grafik but the name is invalid. I also tried "indi-pylibcamera" and "pylibcamera" with same results. I can try to find the logic in the code but I believe there must be a simpler way.

gmke commented 2 months ago

you don't need a camera name at all if there is just one camera. It automatically takes the first camera it finds. It then shows the camera as indi_pylibcamera. I once changed the name to pylibcamera_Main in indi_pylibcamera.ini and it recognised that.

CS, Markus

scriptorron commented 2 months ago

How does indi-allsky know, which camera I have? The INDI library installed many camera driver. I do not believe that indi-allsky probed all these drivers for a connected camera. It just uses indi_simulator_ccd by default. I am still searching how to change this.

scriptorron commented 2 months ago

I also run setup.sh again. There was no option to choose a camera.

gmke commented 2 months ago

argh, I misunderstood your question. Sorry my bad. You have to replace the indiserver setup. Either by rerunning setup.sh or by executing misc/change_camera.sh. After the selection it presents you a list of available camera drivers installed on the local system. grafik

CS, Markus

scriptorron commented 2 months ago

Ha, that made the trick! Indiserver uses now indi_pylibcamera. By Pi Zero W immediately crashed, likely because of no memory. I will increase swap space. Using binning 2 will also help. I come back to you when I have more results.

gmke commented 2 months ago

yep, you will need more swap as written before. I guess this is the major stability issue. If the software environment would be less wasteful of the resources, we wouldn't have a problem.

CS, Markus

scriptorron commented 2 months ago

indi-allsky is doing many memory consuming tasks on the Pi Zero. Even calculating a RGB image from the raw camera frame is pretty CPU and memory hungry. And allsky is doing much more with the images. I am surprised that it runs on my Pi Zero. But it is slow.

Since about half an hour I get images from my HQ camera. I also set binning 2 in the allsky configuration. The images look underexposed but likely this can be fixed in the settings.

gmke commented 2 months ago

Ronald, I never intended to run indi-allsky on the Pi Zero (2). The Pi Zero only delivers the pictures via your driver and indiserver to the indi-allsky instance running on my big server. Because it should/will be an unattended 24/7/365 operation, it must run very stable. I think Aaron did a lot to achieve this on the indi-allsky server side. Unfortunately, the current record with indi_pylibcamera on a Pi Zero 2 is 4 days and usually just one day (sitting on my office desk :-). I'm very thankful that you spent your time working on this issue and my special use case.

CS, Markus

scriptorron commented 2 months ago

Hi Markus,

you are welcome. I am happy that other people also use the indi_pylibcamera.

I realized that I misunderstood your setup. Am I right that your setup is like this:

I will build the same setup here. Do you think indi_allsky will run on a Pi 3B with 1GB RAM? I have 2 of them permanently running for other tasks and can install indi_allsky on one of them for test. Than I can run a camera for days

Regards, Ronald

gmke commented 2 months ago

yes, this is my setup. A Pi 3B with 1GB should be good enough. You don't need to generate time lapse videos, panoramas or store the data for a long time.

CS, Markus

gmke commented 2 months ago

3 days and still running without a hickup. I'm getting optimistic about this setup.

CS, Markus

scriptorron commented 2 months ago

I am happy to read this. Thank you for the update.

scriptorron commented 2 months ago

I have indi-allsky now running on a Pi3 getting images from an indiserver/indi_pylibcamera running on a Pi Zero W. To make the load on the Pi Zero high I did not set the binning of the HQ camera to 2. I will let it run the next days.

From next Saturday I will be in vacation for 2 weeks. I will not have access to my computer and I will not run the test unattended. After my vacation I can start the test again.

gmke commented 2 months ago

7 days and still running without a glitch. Should be ready for the roof now.

Ronald THANK YOU!!!!

CS, Markus

scriptorron commented 2 months ago

You did it without my help. Do you think the increasing of the swap space was the solution? I should make a note in the README.

gmke commented 2 months ago

yes, I think about 500MB swap is needed instead of the default 100M. But you may have possibilities to reduce the memory footprint of the driver.

CS, Markus