indigo-astronomy / indigo

INDIGO is a system of standards and frameworks for multiplatform and distributed astronomy software development designed to scale with your needs.
http://www.indigo-astronomy.org
Other
139 stars 64 forks source link

Mount Synscan with AZeq5 hang with log and deadlock files #509

Closed wgoltz closed 7 months ago

wgoltz commented 7 months ago

A hang during with AZeq5 and Mount Synscan driver

Operations to get the hang indigo_state.txt

tail indigo.log indigo_log.txt

deadlock output indigo_deadlock.txt

rumengb commented 7 months ago

I assume you are using the web interface as there is no sky chart in the mount agent itself. After a server restart you may need to reload the page. and also web interface is just for a demo not for a real use.

The attached indigo log is not complete and probably not a trace log therefore is useless.

indigo deadlock log shows no blocked threads let alone deadlocked threads. Actually the server looks like it is completely idle.

the steps to reproduce says: "connect joystick (@ laptop) (slow)" how to connect it slow?

"click a point in Mount Agent sky display - no response" what response do you expect? the mount to slew to the target?

What operating system, what platform and what browser are you using?

wgoltz commented 7 months ago

Thank you for your comments.

I was challenged for time and not aware of what might be significant and where to start.

I am using the web interface as I thought it was a significant part of the distribution. So I am wrong there, and will use indigo_control_panel for further testing.

The joystick is attached to my laptop running linux and indigo, which just provides game-pad for hand control. indigosky runs on a Pi connected to the telescope, with everything connected by WiFi over the home router.

By slow, I meant that the response was already slow at that point. The indigo.log was a very long list of the same messages. It is just a sample to get some comments so I can get on the right track! Do I have to start a trace log before it hangs or can it be started after the response gets bad? As yet I have no clue as to what is causing my problems.

Yes, I was expecting it to pick up the coordinates ready for a slew, but there was no visible response.

rumengb commented 7 months ago

Let me see if I understand you: So you are running two Indigo servers Server 1: running on Raspberry Pi with the telescope attached to it. Server 2: running on an Intel laptop running Linux with an attached gamepad.

In that case, you need to run the indigo_server on the laptop with -r or --remote-server to connect to the Rpi. and control everything from the indigo_server running on the laptop as it will proxy the devices running in the remote server.

If this is the case please bear in mind that if you restart the remote indigo_server the connection will be lost and such behaviour can be expected.

Also please note, that we are a team of developers and astronomers, there are no clairvoyants on our team :). Please consider this handicap of ours when explaining the issue you are experiencing.

wgoltz commented 7 months ago

Thankyou for your reply,

I do appreciate that you are not clairvoyants. On the other hand, I know very little about your system and it takes me a fair while to write down my problem attempting to explain everything when I know nothing!

I have been running with the main server running on the pi with -r my_laptop. I will try the other way around.


From: Rumen Bogdanovski @.> Sent: Saturday, 18 November 2023 8:30 PM To: indigo-astronomy/indigo @.> Cc: wgoltz @.>; Author @.> Subject: Re: [indigo-astronomy/indigo] Mount Synscan with AZeq5 hang with log and deadlock files (Issue #509)

Let me see if I understand you: So you are running two Indigo servers Server 1: running on Raspberry Pi with the telescope attached to it. Server 2: running on an Intel laptop running Linux with an attached gamepad.

In that case, you need to run the indigo_server on the laptop with -r or --remote-server to connect to the Rpi. and control everything from the indigo_server running on the laptop as it will proxy the devices running in the remote server.

If this is the case please bear in mind that if you restart the remote indigo_server the connection will be lost and such behaviour can be expected.

Also please note, that we are a team of developers and astronomers, there are no clairvoyants on our team. Please consider this handicap of ours when explaining the issue you are experiencing.

— Reply to this email directly, view it on GitHubhttps://github.com/indigo-astronomy/indigo/issues/509#issuecomment-1817470792, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AE4EZIE5ARMWIK3CNLIWZF3YFCE4TAVCNFSM6AAAAAA7QXTSSCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQMJXGQ3TANZZGI. You are receiving this because you authored the thread.

wgoltz commented 7 months ago

I have changed my server setup on my laptop so that indigo-server runs with -r indigosky, and reset the setup on indigosky to its default (I modified the appropriate files in /usr/lib/systemd/system). The laptop enables the drivers HID-gamepad and Mount Agent. indigosky enables the drivers Mount Simulator, Mount SynScan and Mount Nexstar which I want to test.

When I run indigo_control_panel on indigosky, the left hand pane of the screen lists all the loaded drivers @ indigosky and the output is indigo_control_panel_startup_indigosky.txt

When I click on a driver name in indigo_control_panel when run on indigosky, it just shows option names in the right-hand pane with unfilled check-boxes and no text explaining the boxes.

The browser interface seems to work for both localhost:7624 and indigosky.local:7624. It can be very slow to update.

When I run indigo_control_panel on the laptop, the screen is empty. I get this output indigo_control_panel_startup.txt

The log-files viewed with the browser (e.g. localhost:7624/log) are unreliable. They seem to be cached by the browser and their actual location is a bit mystifying. I will try running the servers on both machines manually.

rumengb commented 7 months ago

Goto "File -> Manage services" and you should see both services. Then click the checkbox to connect to the service. you may want to check another setting: "Settings -> Auto connect new services".

rumengb commented 7 months ago

And one more thing. In this setup, you need to use the Indigo service on the laptop. Otherwise, you will not see devices exported by the laptop. In other words the service on the laptop is a proxy for the indigosky.

wgoltz commented 7 months ago

I did as you suggested and indigo_control_panel correctly showed the drivers at remote indigosky. But it did not show drivers on my laptop (namely, the gamepad). The browser interface worked correctly.

I tried to get the local (laptop services) to appear by removing "-r indigosky" when starting indigo_server on the laptop, but get the error:

avahi: Failed to resolve service 'indigosky': Timeout reached

I played with .config/indigo_control_panel.services (trying removing the reference to indigosky, and deleting the file altogether).

With indigosky server running and server on laptop run as: indigo_server -vv -r indigosky

indigo_control_panel console messages (run on laptop): indigo_control_panel_initial_startup.txt

With server on laptop running as: indigo_server -vv (no reference to indigosky, and with .config/indigo_control_panel.services deleted)

indigo_control_panel console messages (run on laptop): indigo_control_panel_startup.txt

indigo_control_panel now seems always to be looking for indigosky, regardless of whether it should be looking; and it should be able to find it (the browser interface can). It stops looking for anything else (the local drivers) when indigosky can't be found. I end up with nothing displayed in indigo_control_panel window when I click on manage services. All the devices, local and remote, are there in the browser interface, and get correctly set as the servers change state.

wgoltz commented 7 months ago

Reboot of indigosky machine got us back to where we were - indigosky drivers visible but not the laptop drivers.

wgoltz commented 7 months ago

Shutdown indigosky and removed reference to indigosky in .config/indigo_control_panel.services. Running indigo_control_panel no longer looks for indigosky.

Nor does it find the services on the local machine. Output from indigo_control_panel: indigo_control_panel_noindigosky.txt

I get blank screen when clicking manage_services in indigo_control_panel, though I have the joystick and a couple of other devices selected according to the browser interface.

rumengb commented 7 months ago

This looks like an avahi/mdns misconfiguration on your laptop, some sort of a firewall or something else. I have no idea what it may be. In any case it is something NOT related to indigo. However, there is one problem at first glance - hostname is not "indigosky" but "indigosky.local". And also .config/indigo_* files are not supposed to be edited manually. So I advise you to remove them and let the applications create them.

wgoltz commented 7 months ago

Starting "indigo_server -r indigosky.local" on the laptop seems to have worked wonderfully well!

Initial experiments show mount simulator and ccd camera simulator are working perfectly, but yet to try with the real telescope and camera.

Thankyou.