Driver for PlayerOne Ares-M Pro keeps failing

[gnthibault]

Describe the bug affected program: indi_playerone_ccd The following error keeps appearing very often, making the camera very hard to use: [ERROR] Error connecting to the CCD (POA_ERROR_OPERATION_FAILED).

To Reproduce

1. Run driver: indiserver indi_playerone_ccd
2. Perform action Nothing specific, it seems more or less random
3. See error In Device manager you will see: [ERROR] Error connecting to the CCD (POA_ERROR_OPERATION_FAILED).

Expected behavior I expect the driver to work properly

Screenshots If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

• OS: Description: Manjaro ARM Linux Release: 23.02
• Version: v2.0.2

[knro]

@hiro3110i can you please check?

[hiro3110i]

@gnthibault Thank you for reporting. One chance, could you please upgrade indilib and drivers to v2.0.3 which is included in the latest KStars v3.6.6? v2.0.2 uses PlayerOneCamera SDK v3.4.0 while v2.0.3 uses the latest v3.4.1. Ares-M Pro is a very new device, so it may need the latest SDK.

Upgrading KStars to v3.6.6 will install indilib v2.0.3. Alternatively, you can download the v2.0.3 indi-3rdparty source code from GitHub and build it as described in the readme.

[gnthibault]

Thank you so much for your feedback. First, this morning I finally found how to reproduce the error in a seemingly deterministic way:

1: power both usb and 12V for the camer 2: start indiserver indi_playerone_ccd 3: click the connect device button in indi device manager (launched from kstars in my case) 4: disconnect / or unpower the usb cable (which I do programmatically using pegasus ultimate powerbox) 5: try to reconnect or use any button from device manager on the camera, you will see an error (that's expected) 6: Now re-plug / repower the camera usb (which I do programmatically using pegasus ultimate powerbox) 7: You will see that you cannot reconnect to the camera though device manager, it will exhibit the poa error without exception 8: Try to restart the playerone driver from indiwebmanager, and then reconnect the camera from device manager, which is a workflow I perform routinely on other devices to programmatically reset them: you will still see the error

From there, the only way I found to reconnect to the camera, is to STOP the whole indiserver, and then start it again, then I manage to get rid of the POA error and reconnect properly. I would like to avoid restarting the whole server, or use a separate server solely for the playerone camera, as it comes with additional complexity.

I am now trying to upgrade indi to v2.0.3 to see if it fixes the issue, but I have the feeling that it's independant of indi/sdk. Thank you again for your support, I will report next experiment results here

[hiro3110i]

@gnthibault According to the changelog of SDK, v3.4.1 has fixed some bugs around USB. I hope it's fixed in the latest SDK.

[gnthibault]

@hiro3110i I updated Indi to v2.0.3 I updated indi-3rd-party libs to v2.0.3 I updated indi-3rd-party indi_playerone_ccd to v2.0.3

And now, I do not even see the device appearing in device manager, when I try to run indi_getprop for my camera, there is not return (No PlayerOne CCD Ares-M PRO.. from [...])

Return from indiserver in verbose mode is:

indiserver -vvv indi_playerone_ccd 2023-08-10T09:11:15: startup: indiserver -vvv indi_playerone_ccd 2023-08-10T09:11:15: Driver indi_playerone_ccd: pid=5013 rfd=6 wfd=6 efd=7 2023-08-10T09:11:15: listening to port 7624 on fd 5 2023-08-10T09:11:15: Local server: listening on local domain at: @/tmp/indiserver 2023-08-10T09:11:15: Driver indi_playerone_ccd: sending msg nq 1:

2023-08-10T09:11:30: Client 9: new arrival from 127.0.0.1:40876 - welcome! 2023-08-10T09:11:30: Client 9: read getProperties
2023-08-10T09:11:30: Driver indi_playerone_ccd: queuing responsible for 2023-08-10T09:11:30: Driver indi_playerone_ccd: sending msg nq 1:

I also tried with indi_playerone_single_ccd, but basically got the same result. This is much worse than expected. I am downgrading to v2.0.2 to check if the camera is back again to rule-out an hardware problem (the other camera driver seems to be working fine).

EDIT: Downgraded everything to v2.0.2, and camera cannot connect now... I really start to suspect a hardware problem now for that specific issue. will keep you up to date with further test results

[gnthibault]

Okey, so I think I found the main rootcause of the issue (not 100% sure though). The Playerone Ares-M Pro camera seems to rely on a very specific initializion/deinitialization order:

USB 3 input needs always to be powered first, Then main 12V input needs to be powered afterwards, with apparently a small delay (I use 5 seconds)

I have done some tests where 12V is powered first, and I never managed to get the camera to work properly. When the camera is in this failed state (no usb communication), then one needs to

• unpower 12V, then unpower USB, then power USB then wait ~5 seconds, then power 12V

Interestingly, the following, that seemed equivalent to me doesn't seems to work:

• unpower USB, then unpower 12V, then power USB then wait ~5 seconds, then power 12V

It took me quite some time to go back and forth between software versions, start/stop procedures, so I am not 100% sure that what I report here is fully deterministic. But to me it looks like a purely hardware/firmware camera related issue. I'd be very glad if another player one Ares-M owner or company developer could report its test report.

[gnthibault]

Ok another aspect that I didn't managed to mention earlier, is that, in order to get a successful "reset", I need to restart completely the indiserver, just restarting the driver itself (through indiwebmanager) is not enough. I reported the problem in the indiwebmanager project, but eventually I am wondering wether it wouldn't make sense to have it here: https://github.com/knro/indiwebmanager/issues/55

[gnthibault]

I have contacted playerone support to get some feedback hopefully.

[gnthibault]

Ok, so I didn't get any relevant feedback from PlayerOne, but I had the opportunity to test-out the camera under windows with their driver + Ascom driver. Interestingly Sharpcap didn't managed to connect to the camera, but PhD2 did, and managed to stream frames from the device.

So in the end it seems that, it is not a hardware problem on the camera side, I will loop with PlayerOne with this information. Now the issue can still be either on the sdk side or the indi driver side.

[gnthibault]

I have implemented a setup where I consistently restart the indiserver as well as implement a proper power-on/power-off strategy. I seeems this way the camera is working as expected, I still need to do further testing, but for me this is definitely a software issue specifically on linux either with sdk or indi driver

[hiro3110i]

@gnthibault Your deep investigations are very helpful, thank you. I'm also discussing with PlayerOne engineer. They're always cooperative and quick. I'd like to expect effective countermeasures.

[knro]

We received more reports about this failure, could it be some SDK issue?

[hiro3110i]

@knro I don't know sure at this moment. I hope that SDK will solve the stability issues around USB. On the other hand, it may be necessary to consider separately whether it is possible to avoid the problem with indi driver.

[gnthibault]

I got some feedback from playerone saying that their hardware engineers are looking into the problem, I will report any other news I receive here.

[hiro3110i]

@gnthibault I also received messages from Lei-san (PlayerOne engineer). There seems to be a little more complicated than the Saturn-C issue #824. I'd like to wait for further information.

[gnthibault]

Interestingly, I cam across this interesting post from Pegasus support: https://pegasusastro.com/forum/viewtopic.php?p=387#p387 That was for ZWO camera though, but the symptoms seems much similar to the one I am experiencing

[hiro3110i]

@gnthibault Today, I updated libplayerone and indi-playerone with the latest SDK v3.5.0. I'm not sure if your problem will be fixed, but I'd appreciate if you could test it. Thank you for your cooperation.

[gnthibault]

Thanks, this weekend I cannot, but I'll do next week

[hiro3110i]

@gnthibault I had a quick chat with Lei-san tonight. Unfortunately, this update cannot solve the problem, but engineers are still looking into it. The current best practice is to connect the USB first and then 12V power later, as you pointed out. I apologize for your inconvenience.

[gnthibault]

I have more tests to do, but it seemed like in my context this might not be an issue as problematic as I initially thought. I am working on a fully robotic observatory and I can make the startup/stop routine perfectly reproducible. My main pain point was simply to update indiweb manager to systematize indiserver restart for each camera. But it seems I am more or less here (will push my last commit to the PR today)

[glundby]

Title | Player One Saturn-C SQR driver indi-playerone-ccd v1.12 interface 6 Crashes system in many ways WDBT-002982 StellarMate StellarMate OS 19 mins ago Unassigned App not used 1.7.7 SkyHunter mount, Player One Saturn-C, ASI462MC

Player One Saturn-C SQR driver indi-playerone-ccd v1.12 interface 6 Crashes system in many ways

Sometimes it kills connection to mount, frequently it disconnects memory stick, now and then it takes down Kstars. If not it stopps in capture of second sub. This happens always when exposure is longer than 1s, shorter may get through several cakptures and downloads.To check my camera I have tested it on USB3 from PC with FireCapture and SharpCap and it works perfect.It used to work well before my last upgrade to Kstars 3.6.7 Stable

[knro]

I don't have this camera so can't test. If you have SM OS, you can file a ticket so we can do remote support to diagnose the issue further.

[glundby]

Sorry! Just found out that the RPI4b cannot handle two USB3 cameras at the same time. It overpowers the USB busses. I need to add a powered USB bus.

[hiro3110i]

@glundby Thank you for reporting. Could you please give me more information about your issue?

For your reference, I'm always using KStars/Ekos with Poseidon-C and Sedna-M which is connected to USB3.0 port on Raspberry Pi4 at the same time. Or another case, Uranus-C and Neptune-C II is working well at the same time in Ekos. Each of use case is long exposure with short exposure guiding. (not high frame rate like a video capture because RasPi4 doesn't have enough performance for video capture)

[glundby]

Hello! It boils down to not being able to do a sequence. The capture counts down to zero and hangs there with no message for a time of minutes, then it tries again. First capture often is successful including download. If exposure is sub second several may be successful and downloads. With longer exposure mostly only first is successful. The same behavior’s is seen also when capturing a single preview. Also in all other operations like focusing or pleate solve same thing happens. I have tested camera in FireCapture and SharpCap. No problems there. I have tested on 3 different RPI4b and on two different rigs. Same problem everywhere. Regards! Gustav Olav Lundby Sendt fra E-post for Windows Fra: Hiroshi SAITOSendt: mandag 6. november 2023 kl. 13:41Til: indilib/indi-3rdpartyKopi: Gustav Olav Lundby; MentionEmne: Re: [indilib/indi-3rdparty] Driver for PlayerOne Ares-M Pro keeps failing (Issue ***@***.*** Thank you for reporting. Could you please give me more information about your issue?For your reference, I'm always using KStars/Ekos with Poseidon-C and Sedna-M which is connected to USB3.0 port on Raspberry Pi4 at the same time. Or another case, Uranus-C and Neptune-C II is working well at the same time in Ekos.Each of use case is long exposure with short exposure guiding. (not high frame rate like a video capture because RasPi4 doesn't have enough performance for video capture)—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you were mentioned.Message ID: ***@***.***> 

[glundby]

Did I mention? It is a Player One Saturn-C SQR. Newest release of Kstars and all. I have successfully used the Saturn-C earler. Sendt fra E-post for Windows Fra: Hiroshi SAITOSendt: mandag 6. november 2023 kl. 13:41Til: indilib/indi-3rdpartyKopi: Gustav Olav Lundby; MentionEmne: Re: [indilib/indi-3rdparty] Driver for PlayerOne Ares-M Pro keeps failing (Issue ***@***.*** Thank you for reporting. Could you please give me more information about your issue?For your reference, I'm always using KStars/Ekos with Poseidon-C and Sedna-M which is connected to USB3.0 port on Raspberry Pi4 at the same time. Or another case, Uranus-C and Neptune-C II is working well at the same time in Ekos.Each of use case is long exposure with short exposure guiding. (not high frame rate like a video capture because RasPi4 doesn't have enough performance for video capture)—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you were mentioned.Message ID: ***@***.***> 

[hiro3110i]

@glundby One point, could you please check your device settings of Saturn-C on your Ekos? The latest version of Ekos has experimental functionality of "Fast Exposure" in the "Options" tab. It's useful for long exposure but interval of each exposure is very short. You can turn off before exposure loop. Device settings are saved in ~/.indi folder. You can try to delete old settings in this folder. Another check point is "USBBandwidthLimit" in the "Controls" tab. Less than 35 is suitable for dual camera system.

If the behavior is still unstable, I recommend rolling back to a previous stable working version.

[glundby]

Hello! I just rolled back to a version of the driver indi-playerone-ccd 1.0 and then it works!First I tried indi-playeronr-ccd 1.1.1 which did not work.The newest I had originally vas 1.1.2 which did not work I am aware of the settings. Fast Exposure I have not played with, nor changed from default. The other that is interesting for me; Sensor Mode, I have tried both options without any effect on the problem. The indi-playerone-ccd v 1.0 does not have this setting at all! I’m going to miss that. Until a fix is released I will stick to v1.0 If you need me to check something I can quickly reinsert the newest release image. If needed you may enter my system via TeamViewer, just tell me when and I will send you the password. Regards! Gustav Olav Lundby Sendt fra E-post for Windows Fra: Hiroshi SAITOSendt: tirsdag 7. november 2023 kl. 14:09Til: indilib/indi-3rdpartyKopi: Gustav Olav Lundby; MentionEmne: Re: [indilib/indi-3rdparty] Driver for PlayerOne Ares-M Pro keeps failing (Issue ***@***.*** One point, could you please check your device settings of Saturn-C on your Ekos?The latest version of Ekos has experimental functionality of "Fast Exposure" in the "Options" tab.It's useful for long exposure but interval of each exposure is very short.You can turn off before exposure loop.Device settings are saved in ~/.indi folder. You can try to delete old settings in this folder.Another check point is "USBBandwidthLimit" in the "Controls" tab. Less than 35 is suitable for dual camera system.If the behavior is still unstable, I recommend rolling back to a previous stable working version.—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you were mentioned.Message ID: ***@***.***> 

[hiro3110i]

@glundby Some new features have been added with the update, but they are not critical for astrophotography. When connecting multiple peripheral devices such as cameras to Raspberry Pi, power supply capacity is often questioned. I'm getting very stable operation using Jackery's high capacity mobile power supply. We also need to choose a high quality USB3.0 cable. On the other hand, reset problems around USB are often caused by problems with the firmware of the USB chip. I apologize for any inconvenience, but I recommend that you use a stable version for a while.

[glundby]

Thank You! I think you are right about the RPi4b usb problems. Adding to this, I'm using IOptron Skyhunter as my light rig. It is controlled by USB from RPi4b AND it also when needed, recharges it's internal battery over same cable. That seems a bad idea since it overpowers the RPi4b. The problem with hanging on second sub is a real SW problem independent from usb power!Regards! Gustav Sendt fra telefonen min-------- Opprinnelig melding --------Fra: Hiroshi SAITO @.>Dato: fre. 10. nov. 2023, 11:07Til: indilib/indi-3rdparty @.>Kopi: Gustav Olav Lundby @.>, Mention @.>Emne: Re: [indilib/indi-3rdparty] Driver for PlayerOne Ares-M Pro keeps failing (Issue #822) @glundby Some new features have been added with the update, but they are not critical for astrophotography. When connecting multiple peripheral devices such as cameras to Raspberry Pi, power supply capacity is often questioned. I'm getting very stable operation using Jackery's high capacity mobile power supply. We also need to choose a high quality USB3.0 cable. On the other hand, reset problems around USB are often caused by problems with the firmware of the USB chip. I apologize for any inconvenience, but I recommend that you use a stable version for a while.

—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you were mentioned.Message ID: @.***>

[hiro3110i]

@glundby Most of the power of Image sensor and USB bus are consumed at the read out timing. If the camera and/or bus hangs while reading out the first picture and is unable to take the second picture, we cannot deny the possibility that the power supply is the cause. Because I can shoot stably all night without any problems. I suspect it's more of a hardware issue (power, timing, noise, temperature something like that) rather than a logical software issue. Of course, there might be a software solution to avoid these problems, but it is not a fundamental.

[glundby]

Hi! First picture gets read and stored OK! I have tested on my main mount too. Even using a powered hub for PO Saturn. (Pegasus Pocket Power Box Advanced) Same. behaviour! Tested swapping cables and swapping cameras with ASI462. PO Saturn always fail, ASI not.Regards! Gustav Sendt fra telefonen min-------- Opprinnelig melding --------Fra: Hiroshi SAITO @.>Dato: fre. 10. nov. 2023, 15:23Til: indilib/indi-3rdparty @.>Kopi: Gustav Olav Lundby @.>, Mention @.>Emne: Re: [indilib/indi-3rdparty] Driver for PlayerOne Ares-M Pro keeps failing (Issue #822) @glundby Most of the power of Image sensor and USB bus are consumed at the read out timing. If the camera and/or bus hangs while reading out the first picture and is unable to take the second picture, we cannot deny the possibility that the power supply is the cause. Because I can shoot stably all night without any problems. I suspect it's more of a hardware issue (power, timing, noise, temperature something like that) rather than a logical software issue. Of course, there might be a software solution to avoid these problems, but it is not a fundamental.

—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you were mentioned.Message ID: @.***>

[hiro3110i]

@glundby I received Saturn-C camera and tested it with the latest indi_playerone_ccd driver v1.13. It works very well on my Raspberry Pi4/4GB (Ubuntu Mate 20.04). Could you try it again with the latest driver?

You should contact to PlayerOne support if you still have problem on Saturn camera.

[glundby]

Hello!

I did apt-get upgrade and currently have

[cid:40f9e88c-73b0-4cc8-b45a-f1471003a869]

That still fails! See log below. So how do I get indi_playerone_ccd 1.13?

2024-01-16T17:21:44 Download Time: 1.79 s, New Download Time Estimate: 1.84 s. 2024-01-16T17:21:44 Captured /media/stellarmate/Samsung/Pictures/Light/Light_192.fits 2024-01-16T17:21:31 Exposure timeout. Restarting exposure... 2024-01-16T17:18:22 Capturing 10.000-second image... 2024-01-16T17:18:21 Received image 5 out of 10. 2024-01-16T17:18:21 Download Time: 1.84 s, New Download Time Estimate: 1.85 s. 2024-01-16T17:18:21 Captured /media/stellarmate/Samsung/Pictures/Light/Light_191.fits 2024-01-16T17:18:08 Exposure timeout. Restarting exposure... 2024-01-16T17:14:59 Capturing 10.000-second image... 2024-01-16T17:14:58 Received image 4 out of 10. 2024-01-16T17:14:58 Download Time: 1.77 s, New Download Time Estimate: 1.86 s. 2024-01-16T17:14:58 Captured /media/stellarmate/Samsung/Pictures/Light/Light_190.fits 2024-01-16T17:14:45 Exposure timeout. Restarting exposure... 2024-01-16T17:11:36 Capturing 10.000-second image... 2024-01-16T17:11:35 Received image 3 out of 10. 2024-01-16T17:11:35 Download Time: 1.79 s, New Download Time Estimate: 1.88 s. 2024-01-16T17:11:35 Captured /media/stellarmate/Samsung/Pictures/Light/Light_189.fits 2024-01-16T17:11:22 Exposure timeout. Restarting exposure... 2024-01-16T17:08:13 Capturing 10.000-second image... 2024-01-16T17:08:12 Received image 2 out of 10. 2024-01-16T17:08:12 Download Time: 1.95 s, New Download Time Estimate: 1.93 s. 2024-01-16T17:08:12 Captured /media/stellarmate/Samsung/Pictures/Light/Light_188.fits 2024-01-16T17:07:59 Exposure timeout. Restarting exposure... 2024-01-16T17:04:50 Capturing 10.000-second image... 2024-01-16T17:04:49 Received image 1 out of 10. 2024-01-16T17:04:49 Download Time: 1.91 s, New Download Time Estimate: 1.91 s. 2024-01-16T17:04:49 Captured /media/stellarmate/Samsung/Pictures/Light/Light_187.fits 2024-01-16T17:04:36 Capturing 10.000-second image... 2024-01-16T17:04:36 Job requires 10.000-second images, has 0/10 frames captured and will be processed. 2024-01-16T17:04:36 Warning: option "Always Reset Sequence When Starting" is enabled and resets the sequence c

---

Fra: Hiroshi SAITO @.> Sendt: tirsdag 16. januar 2024 15:11 Til: indilib/indi-3rdparty @.> Kopi: Gustav Olav Lundby @.>; Mention @.> Emne: Re: [indilib/indi-3rdparty] Driver for PlayerOne Ares-M Pro keeps failing (Issue #822)

@glundbyhttps://github.com/glundby I received Saturn-C camera and tested it with the latest indi_playerone_ccd driver v1.13. It works very well on my Raspberry Pi4/4GB (Ubuntu Mate 20.04). Could you try it again with the latest driver?

You should contact to PlayerOne support if you still have problem on Saturn camera.

— Reply to this email directly, view it on GitHubhttps://github.com/indilib/indi-3rdparty/issues/822#issuecomment-1893823975, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AN4HZJSFDO5WQTJIJVDBMRLYO2C75AVCNFSM6AAAAAA3KVKTLOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQOJTHAZDGOJXGU. You are receiving this because you were mentioned.Message ID: @.***>

[glundby]

Hi!

I tested using v1.13 but no! the error still is present. With v1.1 it works OK.

Regards! Gustav Olav Lundby

---

Fra: Hiroshi SAITO @.> Sendt: tirsdag 16. januar 2024 15:11 Til: indilib/indi-3rdparty @.> Kopi: Gustav Olav Lundby @.>; Mention @.> Emne: Re: [indilib/indi-3rdparty] Driver for PlayerOne Ares-M Pro keeps failing (Issue #822)

@glundbyhttps://github.com/glundby I received Saturn-C camera and tested it with the latest indi_playerone_ccd driver v1.13. It works very well on my Raspberry Pi4/4GB (Ubuntu Mate 20.04). Could you try it again with the latest driver?

You should contact to PlayerOne support if you still have problem on Saturn camera.

— Reply to this email directly, view it on GitHubhttps://github.com/indilib/indi-3rdparty/issues/822#issuecomment-1893823975, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AN4HZJSFDO5WQTJIJVDBMRLYO2C75AVCNFSM6AAAAAA3KVKTLOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQOJTHAZDGOJXGU. You are receiving this because you were mentioned.Message ID: @.***>