Closed kkretzschmar closed 3 years ago
hi Klaus, IMHO strict locking is necessary for several reasons, main one is that each update and enumeration require writing to the bus, and it may happen when half of one update is sent then the another update starts so this will break the messages. I think there is no deadlock here, just a blocked thread, but probably we can introduce some read and write timeouts for such situations. We will discuss this with Peter.
By the way, I have just released a tool to make easier debugging such situations. it is called indigo_deadlock_detector, it is available trough our ppa: http://www.indigo-astronomy.org/indigo-sky.html - see the last section
also git repo for it is here: https://github.com/indigo-astronomy/indigo_deadlock_detector
can you please run it in this situation just to see what it will report? It should report clearly who is waiting on what and where...
Thanks!
Rumen
On Mon, Oct 26, 2020 at 7:32 PM kkretzschmar notifications@github.com wrote:
Hi, while testing the scenario of multiple client connections (see #319 https://github.com/indigo-astronomy/indigo/issues/319), I ended up in an unresponsive server (see thread callstacks below).
This issue occurs if:
- at the time when a client wants to attach the bus (thread 11 - indigo_attach_client) there is a concurrent write operation to the bus (thread 10 - indigo_update_property)
- the write operation is blocked (indigo_write) since the network connection was interrupted
The reason is that the indigo_update_property holds the same mutex lock which the indigo_attach_client function is waiting for. So when the indigo_write does not unlock the mutex, no client can attach the bus.
My question: Is this strict locking here necessary? My naive assumption is that an indigo_update_property cannot change the definition of any Indigo property (only its value), and attaching a new client often comes with the enumeration of properties which is afaik an operation to get the definition of the properties supported by the devices.
Thanks, Klaus
(gdb) thread 11 [Switching to thread 11 (Thread 0x6fbff3d0 (LWP 14685))]
0 __libc_do_syscall () at
../sysdeps/unix/sysv/linux/arm/libc-do-syscall.S:46 46 ../sysdeps/unix/sysv/linux/arm/libc-do-syscall.S: No such file or directory. (gdb) bt
0 __libc_do_syscall () at
../sysdeps/unix/sysv/linux/arm/libc-do-syscall.S:46
1 https://github.com/indigo-astronomy/indigo/issues/1 0x75ee8aea in
__lll_lock_wait (futex=0x76d129b0
, private=0) at lowlevellock.c:46 2 https://github.com/indigo-astronomy/indigo/issues/2 0x75ee33ee in
__GI___pthread_mutex_lock (mutex=0x76d129b0
) at pthread_mutex_lock.c:113 3 https://github.com/indigo-astronomy/indigo/pull/3 0x75f2f740 in
indigo_attach_client (client=client@entry=0x6f2005b8) at indigo_bus.c:374
4 https://github.com/indigo-astronomy/indigo/pull/4 0x75f107ac in
start_worker_thread (client_socket=0x6fbdedb8) at indigo_server_tcp.c:95
5 https://github.com/indigo-astronomy/indigo/pull/5 0x75ee1614 in
start_thread (arg=0x29a2323b) at pthread_create.c:463
6 https://github.com/indigo-astronomy/indigo/pull/6 0x75e7c7fc in ??
() at ../sysdeps/unix/sysv/linux/arm/clone.S:73 from /lib/arm-linux-gnueabihf/libc.so.6 Backtrace stopped: previous frame identical to this frame (corrupt stack?) (gdb) thread 10 [Switching to thread 10 (Thread 0x705ff3d0 (LWP 14273))]
0 __libc_do_syscall () at
../sysdeps/unix/sysv/linux/arm/libc-do-syscall.S:47 47 in ../sysdeps/unix/sysv/linux/arm/libc-do-syscall.S (gdb) bt
0 __libc_do_syscall () at
../sysdeps/unix/sysv/linux/arm/libc-do-syscall.S:47
1 https://github.com/indigo-astronomy/indigo/issues/1 0x75ee8cdc in
__libc_write (fd=fd@entry=25, buf=buf@entry=0x705de610, nbytes=nbytes@entry=88) at ../sysdeps/unix/sysv/linux/write.c:27
2 https://github.com/indigo-astronomy/indigo/issues/2 0x75f1cec0 in
indigo_write (handle=handle@entry=25, buffer=buffer@entry=0x705de610 "\n", length=length@entry=88) at indigo_io.c:383
3 https://github.com/indigo-astronomy/indigo/pull/3 0x75f1cf34 in
indigo_printf (handle=handle@entry=25, format=0x75f8af5c "<setNumberVector device='%s' name='%s' state='%s'%s>\n") at indigo_io.c:401
4 https://github.com/indigo-astronomy/indigo/pull/4 0x75f21c80 in
xml_device_adapter_update_property (client=0x710005b8, property=0x17a5fd8, message=message@entry=0x0, device=0x177d680) at indigo_driver_xml.c:164
5 https://github.com/indigo-astronomy/indigo/pull/5 0x75f2232c in
xml_device_adapter_update_property (client=client@entry=0x710005b8, device=device@entry=0x177d680, property=property@entry=0x17a5fd8, message=message@entry=0x0) at indigo_driver_xml.c:250
6 https://github.com/indigo-astronomy/indigo/pull/6 0x75f300a4 in
indigo_update_property (device=device@entry=0x177d680, property=0x17a5fd8, format=0x0) at indigo_bus.c:567
7 https://github.com/indigo-astronomy/indigo/issues/7 0x75f2cb60 in
indigo_update_coordinates (device=device@entry=0x177d680, message=message@entry=0x0) at indigo_mount_driver.c:1146
8 https://github.com/indigo-astronomy/indigo/pull/8 0x74b7340c in
position_timer_callback (device=0x177d680) at indigo_mount_nexstar.c:495
9 https://github.com/indigo-astronomy/indigo/pull/9 0x75f17364 in
timer_func (timer=0x71039278) at indigo_timer.c:84
10 https://github.com/indigo-astronomy/indigo/pull/10 0x75ee1614 in
start_thread (arg=0x29a2323b) at pthread_create.c:463
11 https://github.com/indigo-astronomy/indigo/pull/11 0x75e7c7fc in ??
() at ../sysdeps/unix/sysv/linux/arm/clone.S:73 from /lib/arm-linux-gnueabihf/libc.so.6
— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/indigo-astronomy/indigo/issues/380, or unsubscribe https://github.com/notifications/unsubscribe-auth/AE5EZBIZQ36P5MFF3XMG4ZDSMWXDFANCNFSM4S7WCPOQ .
Hi Rumen,
I think there is no deadlock here
Yes I agree, there is no deadlock in this situation.
it is called indigo_deadlock_detecto
Yes I've already found that tool, I'll use it when I think I am running into a deadlock. Good idea!
We will discuss this with Peter.
OK, thanks!
Klaus
Can you please send me the output of it?
On Mon, Oct 26, 2020, 8:22 PM kkretzschmar notifications@github.com wrote:
Hi Rumen,
I think there is no deadlock here Yes I agree, there is no deadlock in this situation.
it is called indigo_deadlock_detecto Yes I've already found that tool, I'll use it when I think I am running into a deadlock. Good idea!
We will discuss this with Peter. OK, thanks!
Klaus
— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/indigo-astronomy/indigo/issues/380#issuecomment-716737855, or unsubscribe https://github.com/notifications/unsubscribe-auth/AE5EZBL5UK473P4NW6H3HELSMW443ANCNFSM4S7WCPOQ .
I didn't use the deadlock_detector yet, when the indigo server wasn't responsive I attached with an gdb and created the stacktraces as posted above. Next time when I run into the problem I'll use the deadlock detector.
Ok, it will not only report deadlocks it will report self locks, blocked threads and what are they waiting for. Maybe the name is not very good as it is not only deadlock detector...
On Mon, Oct 26, 2020, 11:02 PM kkretzschmar notifications@github.com wrote:
I didn't use the deadlock_detector yet, when the indigo server wasn't responsive I attached with an gdb and created the stacktraces as posted above. Next time when I run into the problem I'll use the deadlock detector.
— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/indigo-astronomy/indigo/issues/380#issuecomment-716821339, or unsubscribe https://github.com/notifications/unsubscribe-auth/AE5EZBPCV63L76T7NJTB6W3SMXPU5ANCNFSM4S7WCPOQ .
Sounds good, I'll try to reproduce the issue and use it !
I have prototype. It is a scary change, should be tested extensively. Added 5s read and write timeout.. This should not block the server for more than 5 seconds in such situations... Will add it in separate branch until well tested...
On Mon, Oct 26, 2020 at 11:17 PM kkretzschmar notifications@github.com wrote:
Sounds good, I'll try to reproduce the issue and use it !
— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/indigo-astronomy/indigo/issues/380#issuecomment-716828703, or unsubscribe https://github.com/notifications/unsubscribe-auth/AE5EZBOHH2RPULX3WP2DIILSMXRM5ANCNFSM4S7WCPOQ .
I have created a new branch "socket_timeout" with socket read an and write timeouts. Unfortunately timeouts were not handled so I had to make changes in many places and I most likely I have missed some. So this needs extensive testing and fixing before it gets to the master branch.
Klaus, can you please confirm this approach works in your scenario? if not the whole point of this is not worth the effort.
Peter please review my changes!
On Tue, Oct 27, 2020 at 10:54 AM Rumen Bogdanovski rumen@skyarchive.org wrote:
I have created a new branch "socket_timeout" with socket read an and write timeouts. Unfortunately timeouts were not handled so I had to make changes in many places and I most likely I have missed some. So this needs extensive testing and fixing before it gets to the master branch.
Klaus, can you please confirm this approach works in your scenario? if not the whole point of this is not worth the effort.
Great, I am going to test it immediately.
Thanks Rumen, Klaus
First tests look very promising. I can switch between different interfaces (LAN-WLAN-LAN) and reconnection works as expected. I'll do further testing.
Thanks a lot ! I'm going to release the PixInsight update with zeroconf support soon.
Thanks a lot ! I'm going to release the PixInsight update with zeroconf support soon.
Nice! looking forward to it! ;)
On 27/10/2020 21:07, kkretzschmar wrote:
First tests look very promising. I can switch between different interfaces (LAN-WLAN-LAN) and reconnection works as expected. I'll do further testing.
Thanks a lot ! I'm going to release the PixInsight update with zeroconf support soon.
— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/indigo-astronomy/indigo/issues/380#issuecomment-717508045, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACIWSBHSQQM37Z5GNLL3UJTSM4R6NANCNFSM4S7WCPOQ.
Great :) I am glad it is promising. We discussed this approach with with Peter. I will polish it a bit tomorrow do some more tests and I will merge it in the master.
On Tue, Oct 27, 2020, 10:24 PM Juan Conejero notifications@github.com wrote:
Thanks a lot ! I'm going to release the PixInsight update with zeroconf support soon.
Nice! looking forward to it! ;)
On 27/10/2020 21:07, kkretzschmar wrote:
First tests look very promising. I can switch between different interfaces (LAN-WLAN-LAN) and reconnection works as expected. I'll do further testing.
Thanks a lot ! I'm going to release the PixInsight update with zeroconf support soon.
— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub < https://github.com/indigo-astronomy/indigo/issues/380#issuecomment-717508045>,
or unsubscribe < https://github.com/notifications/unsubscribe-auth/ACIWSBHSQQM37Z5GNLL3UJTSM4R6NANCNFSM4S7WCPOQ .
— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/indigo-astronomy/indigo/issues/380#issuecomment-717517070, or unsubscribe https://github.com/notifications/unsubscribe-auth/AE5EZBJS2EQVLZBKNB2GHGDSM4T7VANCNFSM4S7WCPOQ .
Can you try now! I removed the read timeout as it is not the problem. It waits on reads anyway. The trouble is with send/write. And it is handled correctly if it fails. And write time out is definitely a failure. So it is handled correctly. On the other hand read timeout is not a failure. This may be just the client being silent. So I changed the code in many places in order to handle silent clients correctly. But I am not sure I did it everywhere. So I want you to try it with write timeout and no read timeout.
On Tue, Oct 27, 2020, 11:50 PM Rumen Bogdanovski rumen@skyarchive.org wrote:
Great :) I am glad it is promising. We discussed this approach with with Peter. I will polish it a bit tomorrow do some more tests and I will merge it in the master.
On Tue, Oct 27, 2020, 10:24 PM Juan Conejero notifications@github.com wrote:
Thanks a lot ! I'm going to release the PixInsight update with zeroconf support soon.
Nice! looking forward to it! ;)
On 27/10/2020 21:07, kkretzschmar wrote:
First tests look very promising. I can switch between different interfaces (LAN-WLAN-LAN) and reconnection works as expected. I'll do further testing.
Thanks a lot ! I'm going to release the PixInsight update with zeroconf support soon.
— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub < https://github.com/indigo-astronomy/indigo/issues/380#issuecomment-717508045>,
or unsubscribe < https://github.com/notifications/unsubscribe-auth/ACIWSBHSQQM37Z5GNLL3UJTSM4R6NANCNFSM4S7WCPOQ .
— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/indigo-astronomy/indigo/issues/380#issuecomment-717517070, or unsubscribe https://github.com/notifications/unsubscribe-auth/AE5EZBJS2EQVLZBKNB2GHGDSM4T7VANCNFSM4S7WCPOQ .
Hi Rumen, I tried and it looks still good. However I found a problem, which is not easily reproducible and I dont know if it has to do with your recent change ... but I used the deadlock detector to create a snapshot of the server status.
When the problem occurs, the server behaves as if it runs all the time in the write timeout. The client gets updates (e.g. mount lst time, coordinates) every 5 seconds, though the connection is alive. Below is the output of the deadlock detector. I added a corresponding snapshot for the good case at the end of this message.
Thanks!
================== Server with problems =======================================
Inspecting 'indigo_worker' (pid = 23940) for deadlocks
[New LWP 23941]
[New LWP 23942]
[New LWP 23943]
[New LWP 23944]
[New LWP 23946]
[New LWP 23948]
[New LWP 23949]
[New LWP 23953]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/aarch64-linux-gnu/libthread_db.so.1".
0x0000ffffb01d2f98 in __libc_accept (fd=
Thread 9 (Thread 0xffff9ffff140 (LWP 23953)):
Thread 8 (Thread 0xffffac9e6140 (LWP 23949)):
Backtrace stopped: previous frame inner to this frame (corrupt stack?)
Thread 7 (Thread 0xffffad1e7140 (LWP 23948)):
Thread 6 (Thread 0xffffad9e8140 (LWP 23946)):
Thread 5 (Thread 0xffffae1e9140 (LWP 23944)):
Thread 4 (Thread 0xffffae9ea140 (LWP 23943)):
Thread 3 (Thread 0xffffaf45c140 (LWP 23942)):
Thread 2 (Thread 0xffffafc5d140 (LWP 23941)):
Thread 1 (Thread 0xffffb1102af0 (LWP 23940)):
Blocked threads:
[Inferior 1 (process 23940) detached]
============================ Server Ok ========================================
Inspecting 'indigo_worker' (pid = 23940) for deadlocks
[New LWP 23941]
[New LWP 23942]
[New LWP 23943]
[New LWP 23944]
[New LWP 23946]
[New LWP 23948]
[New LWP 23949]
[New LWP 24363]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/aarch64-linux-gnu/libthread_db.so.1".
0x0000ffffb01d2f98 in __libc_accept (fd=
Thread 9 (Thread 0xffff9f7fe140 (LWP 24363)):
Thread 8 (Thread 0xffffac9e6140 (LWP 23949)):
Thread 7 (Thread 0xffffad1e7140 (LWP 23948)):
Thread 6 (Thread 0xffffad9e8140 (LWP 23946)):
Thread 5 (Thread 0xffffae1e9140 (LWP 23944)):
Thread 4 (Thread 0xffffae9ea140 (LWP 23943)):
Thread 3 (Thread 0xffffaf45c140 (LWP 23942)):
Thread 2 (Thread 0xffffafc5d140 (LWP 23941)):
Thread 1 (Thread 0xffffb1102af0 (LWP 23940)):
Blocked threads:
[Inferior 1 (process 23940) detached]
Can you do something for me? See indigo_server_tcp.c and change the timeout to 10s. And see if this happens with 10s interval. If so it is related. I am on the phone now but if you see the commit you will find the place easily :) Thanks!
On Wed, Oct 28, 2020, 8:34 PM kkretzschmar notifications@github.com wrote:
Hi Rumen, I tried and it looks still good. However I found a problem, which is not easily reproducible and I dont know if it has to do with your recent change ... but I used the deadlock detector to create a snapshot of the server status.
When the problem occurs, the server behaves as if it runs all the time in the write timeout. The client gets updates (e.g. mount lst time, coordinates) every 5 seconds, though the connection is alive. Below is the output of the deadlock detector. I added a corresponding snapshot for the good case at the end of this message.
Thanks!
================== Server with problems
Inspecting 'indigo_worker' (pid = 23940) for deadlocks [New LWP 23941] [New LWP 23942] [New LWP 23943] [New LWP 23944] [New LWP 23946] [New LWP 23948] [New LWP 23949] [New LWP 23953] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/aarch64-linux-gnu/libthread_db.so.1". 0x0000ffffb01d2f98 in __libc_accept (fd=, addr=addr@entry=..., len=len@entry=0xffffde3e2b04) at ../sysdeps/unix/sysv/linux/accept.c:26 None
Thread 9 (Thread 0xffff9ffff140 (LWP 23953)):
0 __libc_read (nbytes=524288, buf=0xffff94000ce0, fd=-1610614720) at
../sysdeps/unix/sysv/linux/read.c:26
1 https://github.com/indigo-astronomy/indigo/issues/1 __libc_read
(fd=fd@entry=21, buf=buf@entry=0xffff94000ce0, nbytes=nbytes@entry=524288) at ../sysdeps/unix/sysv/linux/read.c:24
2 https://github.com/indigo-astronomy/indigo/issues/2
0x0000ffffb0223eb8 in read (nbytes=524288, buf=0xffff94000ce0, __fd=21) at /usr/include/aarch64-linux-gnu/bits/unistd.h:44
3 https://github.com/indigo-astronomy/indigo/pull/3 indigo_xml_parse
(device=device@entry=0x0, client=client@entry=0xffff94000b60) at indigo_xml.c:1277
4 https://github.com/indigo-astronomy/indigo/pull/4 0x0000ffffb020ed10
in start_worker_thread (client_socket=0xaaaafbdf05a0) at indigo_server_tcp.c:96
5 https://github.com/indigo-astronomy/indigo/pull/5 0x0000ffffb01c84fc
in start_thread (arg=0xffffde3e2a2f) at pthread_create.c:477
6 https://github.com/indigo-astronomy/indigo/pull/6 0x0000ffffb0122f2c
in thread_start () at ../sysdeps/unix/sysv/linux/aarch64/clone.S:78
Thread 8 (Thread 0xffffac9e6140 (LWP 23949)):
0 __libc_write (nbytes=43, buf=0xffffac9c4f18, fd=-1399042280) at
../sysdeps/unix/sysv/linux/write.c:26
1 https://github.com/indigo-astronomy/indigo/issues/1 __libc_write
(fd=fd@entry=21, buf=buf@entry=0xffffac9c4f18, nbytes=nbytes@entry=43) at ../sysdeps/unix/sysv/linux/write.c:24
2 https://github.com/indigo-astronomy/indigo/issues/2
0x0000ffffb020cb38 in indigo_write (handle=handle@entry=21, buffer=buffer@entry=0xffffac9c4f18 "20.4429\n", length=length@entry=43) at indigo_io.c:437
3 https://github.com/indigo-astronomy/indigo/pull/3 0x0000ffffb020cc4c
in indigo_printf (handle=handle@entry=21, format=format@entry=0xffffb02a6700 "%s\n") at indigo_io.c:448
4 https://github.com/indigo-astronomy/indigo/pull/4 0x0000ffffb0215460
in xml_device_adapter_update_property (client=0xffff94000b60, property=0xaaaafbd76ba0, message=, device=) at indigo_driver_xml.c:170
5 https://github.com/indigo-astronomy/indigo/pull/5 0x0000ffffb0215b08
in xml_device_adapter_update_property (client=, device=, property=, message=) at indigo_driver_xml.c:147
6 https://github.com/indigo-astronomy/indigo/pull/6 0x0000ffffb022ef04
in indigo_update_property (device=device@entry=0xaaaafbd643e0, property=0xaaaafbd76ba0, format=format@entry=0x0) at indigo_bus.c:588
7 https://github.com/indigo-astronomy/indigo/issues/7
0x0000ffffb022bd44 in indigo_update_coordinates (device=0xaaaafbd643e0, message=0x0) at indigo_mount_driver.c:1209
8 https://github.com/indigo-astronomy/indigo/pull/8 0x0000ffffaec48624
in ?? ()
9 https://github.com/indigo-astronomy/indigo/pull/9 0x0000ffffa4000d68
in ?? () Backtrace stopped: previous frame inner to this frame (corrupt stack?)
Thread 7 (Thread 0xffffad1e7140 (LWP 23948)):
0 futex_wait_cancelable (private=0, expected=0,
futex_word=0xffffa0139b20) at ../sysdeps/nptl/futex-internal.h:183
1 https://github.com/indigo-astronomy/indigo/issues/1
__pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0xffffa0139b28, cond=0xffffa0139af8) at pthread_cond_wait.c:508
2 https://github.com/indigo-astronomy/indigo/issues/2
__pthread_cond_wait (cond=cond@entry=0xffffa0139af8, mutex=mutex@entry=0xffffa0139b28) at pthread_cond_wait.c:638
3 https://github.com/indigo-astronomy/indigo/pull/3 0x0000ffffb020fc7c
in timer_func (timer=0xffffa0139ad0) at indigo_timer.c:123
4 https://github.com/indigo-astronomy/indigo/pull/4 0x0000ffffb01c84fc
in start_thread (arg=0xffffad9c685f) at pthread_create.c:477
5 https://github.com/indigo-astronomy/indigo/pull/5 0x0000ffffb0122f2c
in thread_start () at ../sysdeps/unix/sysv/linux/aarch64/clone.S:78
Thread 6 (Thread 0xffffad9e8140 (LWP 23946)):
0 __libc_read (nbytes=524288, buf=0xffffa0000ce0, fd=-1382119360) at
../sysdeps/unix/sysv/linux/read.c:26
1 https://github.com/indigo-astronomy/indigo/issues/1 __libc_read
(fd=fd@entry=20, buf=buf@entry=0xffffa0000ce0, nbytes=nbytes@entry=524288) at ../sysdeps/unix/sysv/linux/read.c:24
2 https://github.com/indigo-astronomy/indigo/issues/2
0x0000ffffb0223eb8 in read (nbytes=524288, buf=0xffffa0000ce0, __fd=20) at /usr/include/aarch64-linux-gnu/bits/unistd.h:44
3 https://github.com/indigo-astronomy/indigo/pull/3 indigo_xml_parse
(device=device@entry=0x0, client=client@entry=0xffffa0000b60) at indigo_xml.c:1277
4 https://github.com/indigo-astronomy/indigo/pull/4 0x0000ffffb020ed10
in start_worker_thread (client_socket=0xaaaafbdf0560) at indigo_server_tcp.c:96
5 https://github.com/indigo-astronomy/indigo/pull/5 0x0000ffffb01c84fc
in start_thread (arg=0xffffde3e2a2f) at pthread_create.c:477
6 https://github.com/indigo-astronomy/indigo/pull/6 0x0000ffffb0122f2c
in thread_start () at ../sysdeps/unix/sysv/linux/aarch64/clone.S:78
Thread 5 (Thread 0xffffae1e9140 (LWP 23944)):
0 __libc_read (nbytes=1, buf=0xffffae1e886f, fd=) at
../sysdeps/unix/sysv/linux/read.c:26
1 https://github.com/indigo-astronomy/indigo/issues/1 __libc_read
(fd=, buf=0xffffae1e886f, nbytes=1) at ../sysdeps/unix/sysv/linux/read.c:24
2 https://github.com/indigo-astronomy/indigo/issues/2
0x0000ffffb1088d98 in ?? () from /lib/aarch64-linux-gnu/libdns_sd.so.1
3 https://github.com/indigo-astronomy/indigo/pull/3 0x0000ffffb1088f20
in ?? () from /lib/aarch64-linux-gnu/libdns_sd.so.1
4 https://github.com/indigo-astronomy/indigo/pull/4 0x0000ffffb01c84fc
in start_thread (arg=0xffffde3e286f) at pthread_create.c:477
5 https://github.com/indigo-astronomy/indigo/pull/5 0x0000ffffb0122f2c
in thread_start () at ../sysdeps/unix/sysv/linux/aarch64/clone.S:78
Thread 4 (Thread 0xffffae9ea140 (LWP 23943)):
0 __libc_read (nbytes=1, buf=0xffffae9e986f, fd=) at
../sysdeps/unix/sysv/linux/read.c:26
1 https://github.com/indigo-astronomy/indigo/issues/1 __libc_read
(fd=, buf=0xffffae9e986f, nbytes=1) at ../sysdeps/unix/sysv/linux/read.c:24
2 https://github.com/indigo-astronomy/indigo/issues/2
0x0000ffffb1088d98 in ?? () from /lib/aarch64-linux-gnu/libdns_sd.so.1
3 https://github.com/indigo-astronomy/indigo/pull/3 0x0000ffffb1088f20
in ?? () from /lib/aarch64-linux-gnu/libdns_sd.so.1
4 https://github.com/indigo-astronomy/indigo/pull/4 0x0000ffffb01c84fc
in start_thread (arg=0xffffde3e286f) at pthread_create.c:477
5 https://github.com/indigo-astronomy/indigo/pull/5 0x0000ffffb0122f2c
in thread_start () at ../sysdeps/unix/sysv/linux/aarch64/clone.S:78
Thread 3 (Thread 0xffffaf45c140 (LWP 23942)):
0 0x0000ffffb0119a9c in __GI___poll (fds=0xffffa8000b60, nfds=2,
timeout=) at ../sysdeps/unix/sysv/linux/poll.c:41
1 https://github.com/indigo-astronomy/indigo/issues/1
0x0000ffffaff45b54 in ?? () from /lib/aarch64-linux-gnu/libusb-1.0.so.0
2 https://github.com/indigo-astronomy/indigo/issues/2
0x0000ffffaff46d44 in libusb_handle_events_timeout_completed () from /lib/aarch64-linux-gnu/libusb-1.0.so.0
3 https://github.com/indigo-astronomy/indigo/pull/3 0x0000ffffaff46d9c
in libusb_handle_events () from /lib/aarch64-linux-gnu/libusb-1.0.so.0
4 https://github.com/indigo-astronomy/indigo/pull/4 0x0000ffffb0206e50
in hotplug_thread (arg=) at indigo_driver.c:613
5 https://github.com/indigo-astronomy/indigo/pull/5 0x0000ffffb01c84fc
in start_thread (arg=0xffffde3e2abf) at pthread_create.c:477
6 https://github.com/indigo-astronomy/indigo/pull/6 0x0000ffffb0122f2c
in thread_start () at ../sysdeps/unix/sysv/linux/aarch64/clone.S:78
Thread 2 (Thread 0xffffafc5d140 (LWP 23941)):
0 0x0000ffffb0119a9c in __GI___poll (fds=0xffffafc5c918, nfds=2,
timeout=) at ../sysdeps/unix/sysv/linux/poll.c:41
1 https://github.com/indigo-astronomy/indigo/issues/1
0x0000ffffaff4caa0 in ?? () from /lib/aarch64-linux-gnu/libusb-1.0.so.0
2 https://github.com/indigo-astronomy/indigo/issues/2
0x0000ffffb01c84fc in start_thread (arg=0xffffde3e283f) at pthread_create.c:477
3 https://github.com/indigo-astronomy/indigo/pull/3 0x0000ffffb0122f2c
in thread_start () at ../sysdeps/unix/sysv/linux/aarch64/clone.S:78
Thread 1 (Thread 0xffffb1102af0 (LWP 23940)):
0 0x0000ffffb01d2f98 in __libc_accept (fd=, addr=addr@entry=...,
len=len@entry=0xffffde3e2b04) at ../sysdeps/unix/sysv/linux/accept.c:26
1 https://github.com/indigo-astronomy/indigo/issues/1
0x0000ffffb020d7f4 in indigo_server_start (callback=) at indigo_server_tcp.c:378
2 https://github.com/indigo-astronomy/indigo/issues/2
0x0000aaaacc1bf078 in server_main () at indigo_server.c:1152
3 https://github.com/indigo-astronomy/indigo/pull/3 0x0000aaaacc1bd2cc
in main (argc=, argv=0xffffde3e3da8) at indigo_server.c:1294
Blocked threads:
[Inferior 1 (process 23940) detached]
============================ Server Ok
Inspecting 'indigo_worker' (pid = 23940) for deadlocks [New LWP 23941] [New LWP 23942] [New LWP 23943] [New LWP 23944] [New LWP 23946] [New LWP 23948] [New LWP 23949] [New LWP 24363] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/aarch64-linux-gnu/libthread_db.so.1". 0x0000ffffb01d2f98 in __libc_accept (fd=, addr=addr@entry=..., len=len@entry=0xffffde3e2b04) at ../sysdeps/unix/sysv/linux/accept.c:26 None
Thread 9 (Thread 0xffff9f7fe140 (LWP 24363)):
0 __libc_read (nbytes=524288, buf=0xffff8c000ce0, fd=-1619007424) at
../sysdeps/unix/sysv/linux/read.c:26
1 https://github.com/indigo-astronomy/indigo/issues/1 __libc_read
(fd=fd@entry=22, buf=buf@entry=0xffff8c000ce0, nbytes=nbytes@entry=524288) at ../sysdeps/unix/sysv/linux/read.c:24
2 https://github.com/indigo-astronomy/indigo/issues/2
0x0000ffffb0223eb8 in read (nbytes=524288, buf=0xffff8c000ce0, __fd=22) at /usr/include/aarch64-linux-gnu/bits/unistd.h:44
3 https://github.com/indigo-astronomy/indigo/pull/3 indigo_xml_parse
(device=device@entry=0x0, client=client@entry=0xffff8c000b60) at indigo_xml.c:1277
4 https://github.com/indigo-astronomy/indigo/pull/4 0x0000ffffb020ed10
in start_worker_thread (client_socket=0xaaaafbd5f460) at indigo_server_tcp.c:96
5 https://github.com/indigo-astronomy/indigo/pull/5 0x0000ffffb01c84fc
in start_thread (arg=0xffffde3e2a2f) at pthread_create.c:477
6 https://github.com/indigo-astronomy/indigo/pull/6 0x0000ffffb0122f2c
in thread_start () at ../sysdeps/unix/sysv/linux/aarch64/clone.S:78
Thread 8 (Thread 0xffffac9e6140 (LWP 23949)):
0 futex_abstimed_wait_cancelable (private=0, abstime=0xffffac9e5918,
clockid=, expected=0, futex_word=0xffffa4000d90) at ../sysdeps/nptl/futex-internal.h:320
1 https://github.com/indigo-astronomy/indigo/issues/1
__pthread_cond_wait_common (abstime=0xffffac9e5918, clockid=, mutex=0xffffa4000d98, cond=0xffffa4000d68) at pthread_cond_wait.c:520
2 https://github.com/indigo-astronomy/indigo/issues/2
__pthread_cond_timedwait (cond=cond@entry=0xffffa4000d68, mutex=mutex@entry=0xffffa4000d98, abstime=abstime@entry=0xffffac9e5918) at pthread_cond_wait.c:656
3 https://github.com/indigo-astronomy/indigo/pull/3 0x0000ffffb020fd34
in timer_func (timer=0xffffa4000d40) at indigo_timer.c:74
4 https://github.com/indigo-astronomy/indigo/pull/4 0x0000ffffb01c84fc
in start_thread (arg=0xffffad1e67ef) at pthread_create.c:477
5 https://github.com/indigo-astronomy/indigo/pull/5 0x0000ffffb0122f2c
in thread_start () at ../sysdeps/unix/sysv/linux/aarch64/clone.S:78
Thread 7 (Thread 0xffffad1e7140 (LWP 23948)):
0 futex_wait_cancelable (private=0, expected=0,
futex_word=0xffffa0139b20) at ../sysdeps/nptl/futex-internal.h:183
1 https://github.com/indigo-astronomy/indigo/issues/1
__pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0xffffa0139b28, cond=0xffffa0139af8) at pthread_cond_wait.c:508
2 https://github.com/indigo-astronomy/indigo/issues/2
__pthread_cond_wait (cond=cond@entry=0xffffa0139af8, mutex=mutex@entry=0xffffa0139b28) at pthread_cond_wait.c:638
3 https://github.com/indigo-astronomy/indigo/pull/3 0x0000ffffb020fc7c
in timer_func (timer=0xffffa0139ad0) at indigo_timer.c:123
4 https://github.com/indigo-astronomy/indigo/pull/4 0x0000ffffb01c84fc
in start_thread (arg=0xffffad9c685f) at pthread_create.c:477
5 https://github.com/indigo-astronomy/indigo/pull/5 0x0000ffffb0122f2c
in thread_start () at ../sysdeps/unix/sysv/linux/aarch64/clone.S:78
Thread 6 (Thread 0xffffad9e8140 (LWP 23946)):
0 __libc_read (nbytes=524288, buf=0xffffa0000ce0, fd=-1382119360) at
../sysdeps/unix/sysv/linux/read.c:26
1 https://github.com/indigo-astronomy/indigo/issues/1 __libc_read
(fd=fd@entry=20, buf=buf@entry=0xffffa0000ce0, nbytes=nbytes@entry=524288) at ../sysdeps/unix/sysv/linux/read.c:24
2 https://github.com/indigo-astronomy/indigo/issues/2
0x0000ffffb0223eb8 in read (nbytes=524288, buf=0xffffa0000ce0, __fd=20) at /usr/include/aarch64-linux-gnu/bits/unistd.h:44
3 https://github.com/indigo-astronomy/indigo/pull/3 indigo_xml_parse
(device=device@entry=0x0, client=client@entry=0xffffa0000b60) at indigo_xml.c:1277
4 https://github.com/indigo-astronomy/indigo/pull/4 0x0000ffffb020ed10
in start_worker_thread (client_socket=0xaaaafbdf0560) at indigo_server_tcp.c:96
5 https://github.com/indigo-astronomy/indigo/pull/5 0x0000ffffb01c84fc
in start_thread (arg=0xffffde3e2a2f) at pthread_create.c:477
6 https://github.com/indigo-astronomy/indigo/pull/6 0x0000ffffb0122f2c
in thread_start () at ../sysdeps/unix/sysv/linux/aarch64/clone.S:78
Thread 5 (Thread 0xffffae1e9140 (LWP 23944)):
0 __libc_read (nbytes=1, buf=0xffffae1e886f, fd=) at
../sysdeps/unix/sysv/linux/read.c:26
1 https://github.com/indigo-astronomy/indigo/issues/1 __libc_read
(fd=, buf=0xffffae1e886f, nbytes=1) at ../sysdeps/unix/sysv/linux/read.c:24
2 https://github.com/indigo-astronomy/indigo/issues/2
0x0000ffffb1088d98 in ?? () from /lib/aarch64-linux-gnu/libdns_sd.so.1
3 https://github.com/indigo-astronomy/indigo/pull/3 0x0000ffffb1088f20
in ?? () from /lib/aarch64-linux-gnu/libdns_sd.so.1
4 https://github.com/indigo-astronomy/indigo/pull/4 0x0000ffffb01c84fc
in start_thread (arg=0xffffde3e286f) at pthread_create.c:477
5 https://github.com/indigo-astronomy/indigo/pull/5 0x0000ffffb0122f2c
in thread_start () at ../sysdeps/unix/sysv/linux/aarch64/clone.S:78
Thread 4 (Thread 0xffffae9ea140 (LWP 23943)):
0 __libc_read (nbytes=1, buf=0xffffae9e986f, fd=) at
../sysdeps/unix/sysv/linux/read.c:26
1 https://github.com/indigo-astronomy/indigo/issues/1 __libc_read
(fd=, buf=0xffffae9e986f, nbytes=1) at ../sysdeps/unix/sysv/linux/read.c:24
2 https://github.com/indigo-astronomy/indigo/issues/2
0x0000ffffb1088d98 in ?? () from /lib/aarch64-linux-gnu/libdns_sd.so.1
3 https://github.com/indigo-astronomy/indigo/pull/3 0x0000ffffb1088f20
in ?? () from /lib/aarch64-linux-gnu/libdns_sd.so.1
4 https://github.com/indigo-astronomy/indigo/pull/4 0x0000ffffb01c84fc
in start_thread (arg=0xffffde3e286f) at pthread_create.c:477
5 https://github.com/indigo-astronomy/indigo/pull/5 0x0000ffffb0122f2c
in thread_start () at ../sysdeps/unix/sysv/linux/aarch64/clone.S:78
Thread 3 (Thread 0xffffaf45c140 (LWP 23942)):
0 0x0000ffffb0119a9c in __GI___poll (fds=0xffffa8000b60, nfds=2,
timeout=) at ../sysdeps/unix/sysv/linux/poll.c:41
1 https://github.com/indigo-astronomy/indigo/issues/1
0x0000ffffaff45b54 in ?? () from /lib/aarch64-linux-gnu/libusb-1.0.so.0
2 https://github.com/indigo-astronomy/indigo/issues/2
0x0000ffffaff46d44 in libusb_handle_events_timeout_completed () from /lib/aarch64-linux-gnu/libusb-1.0.so.0
3 https://github.com/indigo-astronomy/indigo/pull/3 0x0000ffffaff46d9c
in libusb_handle_events () from /lib/aarch64-linux-gnu/libusb-1.0.so.0
4 https://github.com/indigo-astronomy/indigo/pull/4 0x0000ffffb0206e50
in hotplug_thread (arg=) at indigo_driver.c:613
5 https://github.com/indigo-astronomy/indigo/pull/5 0x0000ffffb01c84fc
in start_thread (arg=0xffffde3e2abf) at pthread_create.c:477
6 https://github.com/indigo-astronomy/indigo/pull/6 0x0000ffffb0122f2c
in thread_start () at ../sysdeps/unix/sysv/linux/aarch64/clone.S:78
Thread 2 (Thread 0xffffafc5d140 (LWP 23941)):
0 0x0000ffffb0119a9c in __GI___poll (fds=0xffffafc5c918, nfds=2,
timeout=) at ../sysdeps/unix/sysv/linux/poll.c:41
1 https://github.com/indigo-astronomy/indigo/issues/1
0x0000ffffaff4caa0 in ?? () from /lib/aarch64-linux-gnu/libusb-1.0.so.0
2 https://github.com/indigo-astronomy/indigo/issues/2
0x0000ffffb01c84fc in start_thread (arg=0xffffde3e283f) at pthread_create.c:477
3 https://github.com/indigo-astronomy/indigo/pull/3 0x0000ffffb0122f2c
in thread_start () at ../sysdeps/unix/sysv/linux/aarch64/clone.S:78
Thread 1 (Thread 0xffffb1102af0 (LWP 23940)):
0 0x0000ffffb01d2f98 in __libc_accept (fd=, addr=addr@entry=...,
len=len@entry=0xffffde3e2b04) at ../sysdeps/unix/sysv/linux/accept.c:26
1 https://github.com/indigo-astronomy/indigo/issues/1
0x0000ffffb020d7f4 in indigo_server_start (callback=) at indigo_server_tcp.c:378
2 https://github.com/indigo-astronomy/indigo/issues/2
0x0000aaaacc1bf078 in server_main () at indigo_server.c:1152
3 https://github.com/indigo-astronomy/indigo/pull/3 0x0000aaaacc1bd2cc
in main (argc=, argv=0xffffde3e3da8) at indigo_server.c:1294
Blocked threads:
[Inferior 1 (process 23940) detached]
— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/indigo-astronomy/indigo/issues/380#issuecomment-718129394, or unsubscribe https://github.com/notifications/unsubscribe-auth/AE5EZBMPFUNGMCJXM2IAMV3SNBP4JANCNFSM4S7WCPOQ .
I tried several times, but couldn't reproduce the observed issue above. When I observed the issue I checked the update time over several update cycles. The update time was quite excatly 5 seconds, so I thing it has to do with the new timeout.
There are several 5s timeouts. So I do not know which one is it. Can you provide trace log of the Indigo server?
On Thu, Oct 29, 2020, 12:17 AM kkretzschmar notifications@github.com wrote:
I tried several times, but couldn't reproduce the observed issue above. When I observed the issue I checked the update time over several update cycles. The update time was quite excatly 5 seconds, so I thing it has to do with the new timeout.
— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/indigo-astronomy/indigo/issues/380#issuecomment-718240379, or unsubscribe https://github.com/notifications/unsubscribe-auth/AE5EZBIZYTHXCCOU6PAJGX3SNCJ6LANCNFSM4S7WCPOQ .
i looked at the backtrace Tread 8: Thread 8 (Thread 0xffffac9e6140 (LWP 23949)):
../sysdeps/unix/sysv/linux/write.c:26
(fd=fd@entry=21, buf=buf@entry=0xffffac9c4f18, nbytes=nbytes@entry=43) at ../sysdeps/unix/sysv/linux/write.c:24
in indigo_write (handle=handle@entry=21, buffer=buffer@entry=0xffffac9c4f18 "20.4429\n", length=length@entry=43) at indigo_io.c:437
in indigo_printf (handle=handle@entry=21, format=format@entry=0xffffb02a6700 "%s\n") at indigo_io.c:448
what bothers me that in my socket_timeout branch at line 437 there is no write() I have: remains -= bytes_written;
write is at: 430: long bytes_written = write(handle, buffer, remains);
by the way, it is the samr on "master", are you using some modified version of indigo?
On Thu, Oct 29, 2020 at 12:22 AM Rumen Bogdanovski rumen@skyarchive.org wrote:
There are several 5s timeouts. So I do not know which one is it. Can you provide trace log of the Indigo server?
On Thu, Oct 29, 2020, 12:17 AM kkretzschmar notifications@github.com wrote:
I tried several times, but couldn't reproduce the observed issue above. When I observed the issue I checked the update time over several update cycles. The update time was quite excatly 5 seconds, so I thing it has to do with the new timeout.
— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/indigo-astronomy/indigo/issues/380#issuecomment-718240379, or unsubscribe https://github.com/notifications/unsubscribe-auth/AE5EZBIZYTHXCCOU6PAJGX3SNCJ6LANCNFSM4S7WCPOQ .
This is indeed strange ... I didn't change the sources and see in the indigo_io.c "remains -= bytes_written" at line 437 as well. What I did:
Possible causes:
------- git log --------------
Author: Rumen G.Bogdanovski rumen@skyarchive.org Date: Wed Oct 28 12:31:30 2020 +0200
remove recv/read timout
commit 4a1fd99f8a1335b5f347189254294749937ac0bf Author: Rumen G.Bogdanovski rumen@skyarchive.org Date: Tue Oct 27 10:10:32 2020 +0200
add socket timeout
commit 26c5b0d8ad777bd53aa594edf16c1dfad99f1073 Merge: 38573c1f 7ee01c33 Author: Peter Polakovic peter.polakovic@gmail.com Date: Mon Oct 26 15:43:03 2020 +0100
Merge branch 'master' of https://github.com/indigo-astronomy/indigo
commit 38573c1f8522e543a411acefd581c0596f0d378f Author: Peter Polakovic peter.polakovic@gmail.com Date: Mon Oct 26 15:42:59 2020 +0100
dome_baader/dome_nexdome: missing includes added
commit 5f6a6fd3430fbb1a5f2f2a2f64fb9ddaa104ffbf Author: Peter Polakovic peter.polakovic@gmail.com Date: Mon Oct 26 15:42:32 2020 +0100
ccd_simulator: CCD_IMAGE_PROPERTY state fixed
commit 1f6352a216410f76769af576a403ddedfb6ac837 Author: Peter Polakovic peter.polakovic@gmail.com Date: Mon Oct 26 15:41:59 2020 +0100
Would you please remove the apt installed version. And try to reproduce with 10s timeout.
Also can you enable trace debug level this may help pinpoint the problem. This will produce a lot of output so redirect it to a file please.
To be honest I do not really understand the problem. Can you explain it a bit more? How does it affect the operation?
On Thu, Oct 29, 2020, 10:27 PM kkretzschmar notifications@github.com wrote:
This is indeed strange ... I didn't change the sources and see in the indigo_io.c "remains -= bytes_written" at line 437 as well. What I did:
- checked out socket_timeout branch and built incrementally with master built artifacts (see git log below)
- started build/bin/indigo_server build/drivers/indigo_mount_simulator
Possible causes:
- I have an indigo installation which I have installed with apt -get from last Saturday
- Is it possible that the server loads some shared libraries from that installation. I checked with strace but didnt find anything suspicious
- Maybe the deadlock_detector gets its backlrace from the wrong library? The deadlock_detector is definitely from the Saturday installation
------- git log --------------
Author: Rumen G.Bogdanovski rumen@skyarchive.org Date: Wed Oct 28 12:31:30 2020 +0200
remove recv/read timout
commit 4a1fd99 https://github.com/indigo-astronomy/indigo/commit/4a1fd99f8a1335b5f347189254294749937ac0bf Author: Rumen G.Bogdanovski rumen@skyarchive.org Date: Tue Oct 27 10:10:32 2020 +0200
add socket timeout
commit 26c5b0d https://github.com/indigo-astronomy/indigo/commit/26c5b0d8ad777bd53aa594edf16c1dfad99f1073 Merge: 38573c1 https://github.com/indigo-astronomy/indigo/commit/38573c1f8522e543a411acefd581c0596f0d378f 7ee01c3 https://github.com/indigo-astronomy/indigo/commit/7ee01c3311ff93cd36912522cac80dbbc93d003c Author: Peter Polakovic peter.polakovic@gmail.com Date: Mon Oct 26 15:43:03 2020 +0100
Merge branch 'master' of https://github.com/indigo-astronomy/indigo
commit 38573c1 https://github.com/indigo-astronomy/indigo/commit/38573c1f8522e543a411acefd581c0596f0d378f Author: Peter Polakovic peter.polakovic@gmail.com Date: Mon Oct 26 15:42:59 2020 +0100
dome_baader/dome_nexdome: missing includes added
commit 5f6a6fd https://github.com/indigo-astronomy/indigo/commit/5f6a6fd3430fbb1a5f2f2a2f64fb9ddaa104ffbf Author: Peter Polakovic peter.polakovic@gmail.com Date: Mon Oct 26 15:42:32 2020 +0100
ccd_simulator: CCD_IMAGE_PROPERTY state fixed
commit 1f6352a https://github.com/indigo-astronomy/indigo/commit/1f6352a216410f76769af576a403ddedfb6ac837 Author: Peter Polakovic peter.polakovic@gmail.com Date: Mon Oct 26 15:41:59 2020 +0100
— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/indigo-astronomy/indigo/issues/380#issuecomment-719001934, or unsubscribe https://github.com/notifications/unsubscribe-auth/AE5EZBJD4J7ANBHLRBGEZOLSNHF23ANCNFSM4S7WCPOQ .
Or even better try 3 seconds timeout. It may nappen more often.
On Thu, Oct 29, 2020, 11:08 PM Rumen Bogdanovski rumen@skyarchive.org wrote:
Would you please remove the apt installed version. And try to reproduce with 10s timeout.
Also can you enable trace debug level this may help pinpoint the problem. This will produce a lot of output so redirect it to a file please.
To be honest I do not really understand the problem. Can you explain it a bit more? How does it affect the operation?
On Thu, Oct 29, 2020, 10:27 PM kkretzschmar notifications@github.com wrote:
This is indeed strange ... I didn't change the sources and see in the indigo_io.c "remains -= bytes_written" at line 437 as well. What I did:
- checked out socket_timeout branch and built incrementally with master built artifacts (see git log below)
- started build/bin/indigo_server build/drivers/indigo_mount_simulator
Possible causes:
- I have an indigo installation which I have installed with apt -get from last Saturday
- Is it possible that the server loads some shared libraries from that installation. I checked with strace but didnt find anything suspicious
- Maybe the deadlock_detector gets its backlrace from the wrong library? The deadlock_detector is definitely from the Saturday installation
------- git log --------------
Author: Rumen G.Bogdanovski rumen@skyarchive.org Date: Wed Oct 28 12:31:30 2020 +0200
remove recv/read timout
commit 4a1fd99 https://github.com/indigo-astronomy/indigo/commit/4a1fd99f8a1335b5f347189254294749937ac0bf Author: Rumen G.Bogdanovski rumen@skyarchive.org Date: Tue Oct 27 10:10:32 2020 +0200
add socket timeout
commit 26c5b0d https://github.com/indigo-astronomy/indigo/commit/26c5b0d8ad777bd53aa594edf16c1dfad99f1073 Merge: 38573c1 https://github.com/indigo-astronomy/indigo/commit/38573c1f8522e543a411acefd581c0596f0d378f 7ee01c3 https://github.com/indigo-astronomy/indigo/commit/7ee01c3311ff93cd36912522cac80dbbc93d003c Author: Peter Polakovic peter.polakovic@gmail.com Date: Mon Oct 26 15:43:03 2020 +0100
Merge branch 'master' of https://github.com/indigo-astronomy/indigo
commit 38573c1 https://github.com/indigo-astronomy/indigo/commit/38573c1f8522e543a411acefd581c0596f0d378f Author: Peter Polakovic peter.polakovic@gmail.com Date: Mon Oct 26 15:42:59 2020 +0100
dome_baader/dome_nexdome: missing includes added
commit 5f6a6fd https://github.com/indigo-astronomy/indigo/commit/5f6a6fd3430fbb1a5f2f2a2f64fb9ddaa104ffbf Author: Peter Polakovic peter.polakovic@gmail.com Date: Mon Oct 26 15:42:32 2020 +0100
ccd_simulator: CCD_IMAGE_PROPERTY state fixed
commit 1f6352a https://github.com/indigo-astronomy/indigo/commit/1f6352a216410f76769af576a403ddedfb6ac837 Author: Peter Polakovic peter.polakovic@gmail.com Date: Mon Oct 26 15:41:59 2020 +0100
— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/indigo-astronomy/indigo/issues/380#issuecomment-719001934, or unsubscribe https://github.com/notifications/unsubscribe-auth/AE5EZBJD4J7ANBHLRBGEZOLSNHF23ANCNFSM4S7WCPOQ .
Klaus, I have been trying to reproduce the issue you described but i was unable. I added error logging on write and read next time when it happens we will know for sure if it was the write timeout. please use the latest "socket_timeout". I will not merge it to master until I get all green.
Rumen
On Thu, Oct 29, 2020 at 11:16 PM Rumen Bogdanovski rumen@skyarchive.org wrote:
Or even better try 3 seconds timeout. It may nappen more often.
On Thu, Oct 29, 2020, 11:08 PM Rumen Bogdanovski rumen@skyarchive.org wrote:
Would you please remove the apt installed version. And try to reproduce with 10s timeout.
Also can you enable trace debug level this may help pinpoint the problem. This will produce a lot of output so redirect it to a file please.
To be honest I do not really understand the problem. Can you explain it a bit more? How does it affect the operation?
On Thu, Oct 29, 2020, 10:27 PM kkretzschmar notifications@github.com wrote:
This is indeed strange ... I didn't change the sources and see in the indigo_io.c "remains -= bytes_written" at line 437 as well. What I did:
- checked out socket_timeout branch and built incrementally with master built artifacts (see git log below)
- started build/bin/indigo_server build/drivers/indigo_mount_simulator
Possible causes:
- I have an indigo installation which I have installed with apt -get from last Saturday
- Is it possible that the server loads some shared libraries from that installation. I checked with strace but didnt find anything suspicious
- Maybe the deadlock_detector gets its backlrace from the wrong library? The deadlock_detector is definitely from the Saturday installation
------- git log --------------
Author: Rumen G.Bogdanovski rumen@skyarchive.org Date: Wed Oct 28 12:31:30 2020 +0200
remove recv/read timout
commit 4a1fd99 https://github.com/indigo-astronomy/indigo/commit/4a1fd99f8a1335b5f347189254294749937ac0bf Author: Rumen G.Bogdanovski rumen@skyarchive.org Date: Tue Oct 27 10:10:32 2020 +0200
add socket timeout
commit 26c5b0d https://github.com/indigo-astronomy/indigo/commit/26c5b0d8ad777bd53aa594edf16c1dfad99f1073 Merge: 38573c1 https://github.com/indigo-astronomy/indigo/commit/38573c1f8522e543a411acefd581c0596f0d378f 7ee01c3 https://github.com/indigo-astronomy/indigo/commit/7ee01c3311ff93cd36912522cac80dbbc93d003c Author: Peter Polakovic peter.polakovic@gmail.com Date: Mon Oct 26 15:43:03 2020 +0100
Merge branch 'master' of https://github.com/indigo-astronomy/indigo
commit 38573c1 https://github.com/indigo-astronomy/indigo/commit/38573c1f8522e543a411acefd581c0596f0d378f Author: Peter Polakovic peter.polakovic@gmail.com Date: Mon Oct 26 15:42:59 2020 +0100
dome_baader/dome_nexdome: missing includes added
commit 5f6a6fd https://github.com/indigo-astronomy/indigo/commit/5f6a6fd3430fbb1a5f2f2a2f64fb9ddaa104ffbf Author: Peter Polakovic peter.polakovic@gmail.com Date: Mon Oct 26 15:42:32 2020 +0100
ccd_simulator: CCD_IMAGE_PROPERTY state fixed
commit 1f6352a https://github.com/indigo-astronomy/indigo/commit/1f6352a216410f76769af576a403ddedfb6ac837 Author: Peter Polakovic peter.polakovic@gmail.com Date: Mon Oct 26 15:41:59 2020 +0100
— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/indigo-astronomy/indigo/issues/380#issuecomment-719001934, or unsubscribe https://github.com/notifications/unsubscribe-auth/AE5EZBJD4J7ANBHLRBGEZOLSNHF23ANCNFSM4S7WCPOQ .
Hi Rumen, I could reproduce the problem. Here is the trace output:
socket 20 - active WLAN connection socket 21 - interrupted LAN connection
Hope that helps.
Thanks, Klaus
8:01:50.371094 indigo_server: 'Mount Simulator'.'MOUNT_LST_TIME' NUMBER ro Ok 2.0 0 {
18:01:50.371179 indigo_server: 'TIME' = 19.7262
18:01:50.371246 indigo_server: }
18:01:50.371321 indigo_server: 20 ←
Good, it is related, for some reason the connection is not terminated. And what about the normal case? Does it terminate the connection?
On Sat, Oct 31, 2020 at 7:10 PM kkretzschmar notifications@github.com wrote:
Hi Rumen, I could reproduce the problem. Here is the trace output:
socket 20 - active WLAN connection socket 21 - interrupted LAN connection
Hope that helps.
Thanks, Klaus
8:01:50.371094 indigo_server: 'Mount Simulator'.'MOUNT_LST_TIME' NUMBER ro Ok 2.0 0 { 18:01:50.371179 indigo_server: 'TIME' = 19.7262 18:01:50.371246 indigo_server: } 18:01:50.371321 indigo_server: 20 ← 18:01:50.371516 indigo_server: 20 ← 19.7262 18:01:50.371623 indigo_server: 20 ← 18:01:50.371715 indigo_server: 21 ← 18:01:55.490826 indigo_server: indigo_write(): Resource temporarily unavailable 18:01:55.490988 indigo_server: 21 ← 19.7262 18:02:00.610825 indigo_server: indigo_write(): Resource temporarily unavailable 18:02:00.610982 indigo_server: 21 ← 18:02:05.730832 indigo_server: indigo_write(): Resource temporarily unavailable 18:02:05.730990 indigo_server: INDIGO Bus: property update 18:02:05.731069 indigo_server: 'Mount Simulator'.'MOUNT_EQUATORIAL_COORDINATES' NUMBER rw Ok 2.0 0 { 18:02:05.731154 indigo_server: 'RA' = 13.7262 18:02:05.731229 indigo_server: 'DEC' = 90 18:02:05.731295 indigo_server: } 18:02:05.731370 indigo_server: 20 ← 18:02:05.731566 indigo_server: 20 ← 13.7262 18:02:05.731673 indigo_server: 20 ← 90 18:02:05.731797 indigo_server: 20 ← 18:02:05.731926 indigo_server: 21 ← 18:02:10.850852 indigo_server: indigo_write(): Resource temporarily unavailable 18:02:10.851029 indigo_server: 21 ← 13.7262 18:02:15.970856 indigo_server: indigo_write(): Resource temporarily unavailable 18:02:15.971032 indigo_server: 21 ← 90 NULL, 0, NULL) = ? ERESTARTSYS (To be restarted if SA_RESTART is set) --- SIGWINCH {si_signo=SIGWINCH, si_code=SI_KERNEL} --- wait4(24259, NULL, 0, NULL) = ? ERESTARTSYS (To be restarted if SA_RESTART is set) --- SIGWINCH {si_signo=SIGWINCH, si_code=SI_KERNEL} --- wait4(24259, NULL, 0, NULL) = ? ERESTARTSYS (To be restarted if SA_RESTART is set) --- SIGWINCH {si_signo=SIGWINCH, si_code=SI_KERNEL} --- wait4(24259, NULL, 0, NULL) = ? ERESTARTSYS (To be restarted if SA_RESTART is set) --- SIGWINCH {si_signo=SIGWINCH, si_code=SI_KERNEL} --- wait4(24259, NULL, 0, NULL) = ? ERESTARTSYS (To be restarted if SA_RESTART is set) --- SIGWINCH {si_signo=SIGWINCH, si_code=SI_KERNEL} --- wait4(24259, NULL, 0, NULL) = ? ERESTARTSYS (To be restarted if SA_RESTART is set) --- SIGWINCH {si_signo=SIGWINCH, si_code=SI_KERNEL} --- wait4(24259, NULL, 0, NULL) = ? ERESTARTSYS (To be restarted if SA_RESTART is set) --- SIGWINCH {si_signo=SIGWINCH, si_code=SI_KERNEL} --- wait4(24259, 18:02:21.090818 indigo_server: indigo_write(): Resource temporarily unavailable
— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/indigo-astronomy/indigo/issues/380#issuecomment-719961454, or unsubscribe https://github.com/notifications/unsubscribe-auth/AE5EZBKSBWDI3NXXLUHXBV3SNRAIFANCNFSM4S7WCPOQ .
When I have a single connection and I temporarily interrupt the connection (plug-out LAN cable), the the server keeps sending updates. What do you exactly mean with normal case?
I think I know what the problem is and I think it has to do with my client implementation. I'll check it next week.
What surprises me is that you observe this behaviour from time to time. It should not always. On the server we just call send and we do not care if it succeeds. I thought we handle such cases but we do not :( So what you see is pretty normal. Unfortunately the fix seems a kind of scary at this point. I know how to fix it but it will take time and more testing... I will try to find a nice and elegant solution for that massive overlooking from our side... :(
On Sun, Nov 1, 2020, 12:00 AM kkretzschmar notifications@github.com wrote:
I think I know what the problem is and I think it has to do with my client implementation. I'll check it next week.
— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/indigo-astronomy/indigo/issues/380#issuecomment-719993609, or unsubscribe https://github.com/notifications/unsubscribe-auth/AE5EZBMLRLZ6RHLSEUDIYVLSNSCGNANCNFSM4S7WCPOQ .
Klaus, can you try the master branch. After the release yesterday we merged it and we added handling for write timeout...
Rumen
On Sun, Nov 1, 2020, 12:28 AM Rumen Bogdanovski rumen@skyarchive.org wrote:
What surprises me is that you observe this behaviour from time to time. It should not always. On the server we just call send and we do not care if it succeeds. I thought we handle such cases but we do not :( So what you see is pretty normal. Unfortunately the fix seems a kind of scary at this point. I know how to fix it but it will take time and more testing... I will try to find a nice and elegant solution for that massive overlooking from our side... :(
On Sun, Nov 1, 2020, 12:00 AM kkretzschmar notifications@github.com wrote:
I think I know what the problem is and I think it has to do with my client implementation. I'll check it next week.
— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/indigo-astronomy/indigo/issues/380#issuecomment-719993609, or unsubscribe https://github.com/notifications/unsubscribe-auth/AE5EZBMLRLZ6RHLSEUDIYVLSNSCGNANCNFSM4S7WCPOQ .
Klaus, Can you please test and give us feedback?
On Mon, Nov 2, 2020, 8:48 PM Rumen Bogdanovski rumen@skyarchive.org wrote:
Klaus, can you try the master branch. After the release yesterday we merged it and we added handling for write timeout...
Rumen
On Sun, Nov 1, 2020, 12:28 AM Rumen Bogdanovski rumen@skyarchive.org wrote:
What surprises me is that you observe this behaviour from time to time. It should not always. On the server we just call send and we do not care if it succeeds. I thought we handle such cases but we do not :( So what you see is pretty normal. Unfortunately the fix seems a kind of scary at this point. I know how to fix it but it will take time and more testing... I will try to find a nice and elegant solution for that massive overlooking from our side... :(
On Sun, Nov 1, 2020, 12:00 AM kkretzschmar notifications@github.com wrote:
I think I know what the problem is and I think it has to do with my client implementation. I'll check it next week.
— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/indigo-astronomy/indigo/issues/380#issuecomment-719993609, or unsubscribe https://github.com/notifications/unsubscribe-auth/AE5EZBMLRLZ6RHLSEUDIYVLSNSCGNANCNFSM4S7WCPOQ .
Hi Rumen, sorry for the late response. I tested and it works perfectly. No interruptions after change of network devices anymore. Thank you for fixing this, this saved my investment to support multiple network interfaces. I'll close this issue.
Thank you! Klaus
Great I hope to see this feature in the Pixinsight INDIGO module soon :)
Hi, while testing the scenario of multiple client connections (see https://github.com/indigo-astronomy/indigo/issues/319), I ended up in an unresponsive server (see thread callstacks below).
This issue occurs if:
The reason is that the indigo_update_property holds the same mutex lock which the indigo_attach_client function is waiting for. So when the indigo_write does not unlock the mutex, no client can attach the bus.
My question: Is this strict locking here necessary? My naive assumption is that an indigo_update_property cannot change the definition of any Indigo property (only its value), and attaching a new client often comes with the enumeration of properties which is afaik an operation to get the definition of the properties supported by the devices.
Thanks, Klaus
(gdb) thread 11 [Switching to thread 11 (Thread 0x6fbff3d0 (LWP 14685))]
0 __libc_do_syscall () at ../sysdeps/unix/sysv/linux/arm/libc-do-syscall.S:46
46 ../sysdeps/unix/sysv/linux/arm/libc-do-syscall.S: No such file or directory. (gdb) bt
0 __libc_do_syscall () at ../sysdeps/unix/sysv/linux/arm/libc-do-syscall.S:46
1 0x75ee8aea in __lll_lock_wait (futex=0x76d129b0, private=0) at lowlevellock.c:46
2 0x75ee33ee in __GI___pthread_mutex_lock (mutex=0x76d129b0) at pthread_mutex_lock.c:113
3 0x75f2f740 in indigo_attach_client (client=client@entry=0x6f2005b8) at indigo_bus.c:374
4 0x75f107ac in start_worker_thread (client_socket=0x6fbdedb8) at indigo_server_tcp.c:95
5 0x75ee1614 in start_thread (arg=0x29a2323b) at pthread_create.c:463
6 0x75e7c7fc in ?? () at ../sysdeps/unix/sysv/linux/arm/clone.S:73 from /lib/arm-linux-gnueabihf/libc.so.6
Backtrace stopped: previous frame identical to this frame (corrupt stack?) (gdb) thread 10 [Switching to thread 10 (Thread 0x705ff3d0 (LWP 14273))]
0 __libc_do_syscall () at ../sysdeps/unix/sysv/linux/arm/libc-do-syscall.S:47
47 in ../sysdeps/unix/sysv/linux/arm/libc-do-syscall.S (gdb) bt
0 __libc_do_syscall () at ../sysdeps/unix/sysv/linux/arm/libc-do-syscall.S:47
1 0x75ee8cdc in __libc_write (fd=fd@entry=25, buf=buf@entry=0x705de610, nbytes=nbytes@entry=88) at ../sysdeps/unix/sysv/linux/write.c:27
2 0x75f1cec0 in indigo_write (handle=handle@entry=25,
3 0x75f1cf34 in indigo_printf (handle=handle@entry=25, format=0x75f8af5c "<setNumberVector device='%s' name='%s' state='%s'%s>\n") at indigo_io.c:401
4 0x75f21c80 in xml_device_adapter_update_property (client=0x710005b8, property=0x17a5fd8, message=message@entry=0x0, device=0x177d680) at indigo_driver_xml.c:164
5 0x75f2232c in xml_device_adapter_update_property (client=client@entry=0x710005b8, device=device@entry=0x177d680, property=property@entry=0x17a5fd8,
6 0x75f300a4 in indigo_update_property (device=device@entry=0x177d680, property=0x17a5fd8, format=0x0) at indigo_bus.c:567
7 0x75f2cb60 in indigo_update_coordinates (device=device@entry=0x177d680, message=message@entry=0x0) at indigo_mount_driver.c:1146
8 0x74b7340c in position_timer_callback (device=0x177d680) at indigo_mount_nexstar.c:495
9 0x75f17364 in timer_func (timer=0x71039278) at indigo_timer.c:84
10 0x75ee1614 in start_thread (arg=0x29a2323b) at pthread_create.c:463
11 0x75e7c7fc in ?? () at ../sysdeps/unix/sysv/linux/arm/clone.S:73 from /lib/arm-linux-gnueabihf/libc.so.6