[v3.0.3] CrossMgrImpinj not reconnecting if network adapter is unplugged or disabled (Windows 10)

esitarski / CrossMgr

Cyclo Cross Management Application

MIT License

41 stars 20 forks source link

[v3.0.3] CrossMgrImpinj not reconnecting if network adapter is unplugged or disabled (Windows 10) #31

Closed rkantos closed 4 years ago

rkantos commented 5 years ago

I am planning to use a compute stick with a USB-Ethernet adapter to run CrossMgrImpinj. However, while testing with the USB-adapter, I notice that if the USB-adapter is removed from the device, CrossMgrImpinj doesn't recognize it and does not try to reconnect to the reader anymore. A manual reset is needed. As the compute stick doesn't have a monitor, this is of course non-optimal. The same behaviour is apparrent when disabling the ethernet adapter manually from Control Panel\Network and Internet\Network Connections. Reconnecting to the reader works as expected when just disconnecting the Ethernet-cable from the USB adapter.

stuartlynne commented 5 years ago

Linux or Windows?

On Wed, Jul 24, 2019 at 5:43 AM rkantos notifications@github.com wrote:

I am planning to use a compute stick with a USB-Ethernet adapter to run CrossMgrImpinj. However, while testing with the USB-adapter, I notice that if the USB-adapter is removed from the device, CrossMgrImpinj doesn't recognize it and does not try to reconnect to the reader anymore. A manual reset is needed. As the compute stick doesn't have a monitor, this is of course non-optimal. The same behaviour is apparrent when disabling the ethernet adapter manually from Control Panel\Network and Internet\Network Connections. Reconnecting to the reader works as expected when just disconnecting the Ethernet-cable from the USB adapter.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/esitarski/CrossMgr/issues/31?email_source=notifications&email_token=AACIUWLVAKMUVGC2KGAKDXLQBBE6LA5CNFSM4IGPU6OKYY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4HBGESRQ, or mute the thread https://github.com/notifications/unsubscribe-auth/AACIUWLETPMWPT2XH3Z4NUDQBBE6LANCNFSM4IGPU6OA .

-- __O____ -\<,____ ____()/()___

Stuart_Lynnestuart.lynne@gmail.com604-518-1749(m)__604-461-7532(h)

rkantos commented 5 years ago

Windows 10.. Will test Ubuntu with 16.04 next.

stuartlynne commented 5 years ago

Is there a reason you don't want to run CrossMgrImpinj on the CrossMgr laptop?

On Wed, Jul 24, 2019 at 5:28 PM rkantos notifications@github.com wrote:

Linux or Windows? … <#m2272346196539659816> On Wed, Jul 24, 2019 at 5:43 AM rkantos @.***> wrote: I am planning to use a compute stick with a USB-Ethernet adapter to run CrossMgrImpinj. However, while testing with the USB-adapter, I notice that if the USB-adapter is removed from the device, CrossMgrImpinj doesn't recognize it and does not try to reconnect to the reader anymore. A manual reset is needed. As the compute stick doesn't have a monitor, this is of course non-optimal. The same behaviour is apparrent when disabling the ethernet adapter manually from Control Panel\Network and Internet\Network Connections. Reconnecting to the reader works as expected when just disconnecting the Ethernet-cable from the USB adapter. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#31 https://github.com/esitarski/CrossMgr/issues/31?email_source=notifications&email_token=AACIUWLVAKMUVGC2KGAKDXLQBBE6LA5CNFSM4IGPU6OKYY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4HBGESRQ>, or mute the thread https://github.com/notifications/unsubscribe-auth/AACIUWLETPMWPT2XH3Z4NUDQBBE6LANCNFSM4IGPU6OA .

O -<,____ ()/(_)__ _____ Stuart_Lynnestuart.lynne@gmail.com __604-518-1749(m)604-461-7532(h)

Windows 10.. Will test Ubuntu with 16.04 next.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/esitarski/CrossMgr/issues/31?email_source=notifications&email_token=AACIUWKXID62BWLG6MNYA7LQBDXSRA5CNFSM4IGPU6OKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD2X7H5Q#issuecomment-514847734, or mute the thread https://github.com/notifications/unsubscribe-auth/AACIUWJMATZFJ5K2P7G7I7TQBDXSRANCNFSM4IGPU6OA .

-- __O____ -\<,____ ____()/()___

Stuart_Lynnestuart.lynne@gmail.com604-518-1749(m)__604-461-7532(h)

rkantos commented 5 years ago

Well, first I would like to keep the tag recorder on a machine that would not be running anything else than CrossMgrImpinj to keep it as sort of a self contained unit with a battery. Then I'd like to be able to have the reader wirelessly connect to the CrossMgr computer. I have also used the reader wirelessly, but apparrently it isn't always reliable enough. With a computer directly attached to the reader and then transmitting the data wirelessly from CrossMgrImpinj would mean I don't have to worry about losing tag reads. Also, currently I am using one or even two computers to run cameras, and would like to have minimal amount of wires in one computer too.

rkantos commented 5 years ago

I tested with Ubuntu 16.04.6 and the behaviour is as expected. This means the problem only occurs on Windows.

stuartlynne commented 5 years ago

My standard configuration is two R1000 readers, with one attached directly via Ethernet, and one across the road attached via WiFi.

Have never seen an issue with that.

I use a 2.4GHz connection. The reader end has a TPLink mini wifi router (2"x2" powered by USB) to connect the reader to the router on the other side of the road.

On Thu, Jul 25, 2019 at 12:48 AM rkantos notifications@github.com wrote:

Well, first I would like to keep the tag recorder on a machine that would not be running anything else than CrossMgrImpinj to keep it as sort of a self contained unit with a battery. Then I'd like to be able to have the reader wirelessly connect to the CrossMgr computer. I have also used the reader wirelessly, but apparrently it isn't always reliable enough. With a computer directly attached to the reader and then transmitting the data wirelessly from CrossMgrImpinj would mean I don't have to worry about losing tag reads. Also, currently I am using one or even two computers to run cameras, and would like to have minimal amount of wires in one computer too.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/esitarski/CrossMgr/issues/31?email_source=notifications&email_token=AACIUWNZNBFPH64MGFIOWZLQBFLDNA5CNFSM4IGPU6OKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD2YVA7A#issuecomment-514936956, or mute the thread https://github.com/notifications/unsubscribe-auth/AACIUWON6SKATEDLDRDQQM3QBFLDNANCNFSM4IGPU6OA .

-- __O____ -\<,____ ____()/()___

Stuart_Lynnestuart.lynne@gmail.com604-518-1749(m)__604-461-7532(h)

rkantos commented 5 years ago

My standard configuration is two R1000 readers, with one attached directly via Ethernet, and one across the road attached via WiFi. Have never seen an issue with that. I use a 2.4GHz connection. The reader end has a TPLink mini wifi router (2"x2" powered by USB) to connect the reader to the router on the other side of the road.

Where is your other reader pointed at, or are you using LHCP and RHCP antennas with the readers on opposite sides?

Yes, altough I am currently using only one Motorola FX9500 as my main reader, I also used a TP-Link WR703n powered by the reader's USB port. However, when I did some timing/testing in a town square, the wifi was not stable. I suspect there were some interference issues that the router didn't like.

As many newer laptops can only work with ethernet using a dongle, the issue of unplugging the network card is more and more important in the future. Therefore this problem definitely needs a fix. Meanwhile I will install some version of Linux on my compute stick... Which also brings to mind a question; Does CrossMgrImpinj even need a GUI? Would it be possible to run it just from the terminal as a basic console application?

stuartlynne commented 5 years ago

On Thu, Jul 25, 2019 at 5:08 PM rkantos notifications@github.com wrote:

My standard configuration is two R1000 readers, with one attached directly via Ethernet, and one across the road attached via WiFi. Have never seen an issue with that. I use a 2.4GHz connection. The reader end has a TPLink mini wifi router (2"x2" powered by USB) to connect the reader to the router on the other side of the road.

Where is your other reader pointed at, or are you using LHCP and RHCP antennas with the readers on opposite sides?

Two readers, each with one antenna.

I stay with older laptops so I can get an Ethernet port and to reduce costs if it gets rained on. Much less worried about a $400 used off eBay laptop than something new and a lot more expensive. Currently HP Zbook with i7, about $400 US.

I suspect that CrossMgrImpinj could be run headless easily. With command line args for network addresses.

If the connection was from CrossMgr to CrossMgrImpinj (instead of the current direction), CrossMgrImpinj would effectively just need the Reader IP address and what antenna ports to use.

Yes, altough I am currently using only one Motorola FX9500 as my main reader, I also used a TP-Link WR703n powered by the reader's USB port. However, when I did some timing/testing in a town square, the wifi was not stable. I suspect there were some interference issues that the router didn't like.

As many newer laptops can only work with ethernet using a dongle, the issue of unplugging the network card is more and more important in the future. Therefore this problem definitely needs a fix. Meanwhile I will install some version of Linux on my compute stick... Which also brings to mind a question; Does CrossMgrImpinj even need a GUI? Would it be possible to run it just from the terminal as a basic console application?

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/esitarski/CrossMgr/issues/31?email_source=notifications&email_token=AACIUWO3LQPJL27CXBSZD7DQBI56DA5CNFSM4IGPU6OKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD23DRGY#issuecomment-515258523, or mute the thread https://github.com/notifications/unsubscribe-auth/AACIUWO6UOWHYMQIRXJMBOLQBI56DANCNFSM4IGPU6OA .

-- __O____ -\<,____ ____()/()___

Stuart_Lynnestuart.lynne@gmail.com604-518-1749(m)__604-461-7532(h)

rkantos commented 5 years ago

On Thu, Jul 25, 2019 at 5:08 PM rkantos @.***> wrote: My standard configuration is two R1000 readers, with one attached directly via Ethernet, and one across the road attached via WiFi. Have never seen an issue with that. I use a 2.4GHz connection. The reader end has a TPLink mini wifi router (2"x2" powered by USB) to connect the reader to the router on the other side of the road. Where is your other reader pointed at, or are you using LHCP and RHCP antennas with the readers on opposite sides? Two readers, each with one antenna. I stay with older laptops so I can get an Ethernet port and to reduce costs if it gets rained on. Much less worried about a $400 used off eBay laptop than something new and a lot more expensive. Currently HP Zbook with i7, about $400 US. I suspect that CrossMgrImpinj could be run headless easily. With command line args for network addresses. If the connection was from CrossMgr to CrossMgrImpinj (instead of the current direction), CrossMgrImpinj would effectively just need the Reader IP address and what antenna ports to use. … Yes, altough I am currently using only one Motorola FX9500 as my main reader, I also used a TP-Link WR703n powered by the reader's USB port. However, when I did some timing/testing in a town square, the wifi was not stable. I suspect there were some interference issues that the router didn't like. As many newer laptops can only work with ethernet using a dongle, the issue of unplugging the network card is more and more important in the future. Therefore this problem definitely needs a fix. Meanwhile I will install some version of Linux on my compute stick... Which also brings to mind a question; Does CrossMgrImpinj even need a GUI? Would it be possible to run it just from the terminal as a basic console application? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#31?email_source=notifications&email_token=AACIUWO3LQPJL27CXBSZD7DQBI56DA5CNFSM4IGPU6OKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD23DRGY#issuecomment-515258523>, or mute the thread https://github.com/notifications/unsubscribe-auth/AACIUWO6UOWHYMQIRXJMBOLQBI56DANCNFSM4IGPU6OA .

__O____ -\<,____ ()/()_ ___ Stuart_Lynnestuart.lynne@gmail.com604-518-1749(m)__604-461-7532(h)

The same goes for me, while of course the Compute Stick can be powered from a large USB-power bank for at least a day. I also use older laptops, especially the Thinkpad X220 and X230, which can be had for less than 100$. While they weigh a bit, at least they can have 16GB ram... The thinkpads' keyboard can be poured with litres of liquid, which is why I imagine they will also survive a moderate thunderstorm without issues.

Meanwhile this problem is fixed, I'll install some linux on the compute stick, even though it is a bit of a pain. At least my reader now fits in a backpack with it's antenna, and unplugging the USB to ethernet adapter is not much of a risk.

rkantos commented 4 years ago

This is still an issue with CrossMgrImpinj.. It sometimes occurs (at least on Windows, probably not on Linux) even if the connection is just a bit unstable (via wifi) and the network connection is cut out. In this case CrossMgrImpinj does recognize the connection is cut, but still gets stuck on "Waiting for "get time" command from CrossMgr...".

esitarski commented 4 years ago

The status you reported shows the problem happens in the handshake part of the connect, not tag transmit. Your network must be very unstable.

That makes it tough (impossible?) to reproduce as the handshake takes a very short time. I reviewed the handshake code and made it simpler, and this made it more straightforward to handle errors. The new reconnect logic should now handle all failures during the handshake, but I wasn't able to test all the failure cases.

Updates are available in CrossMgrImpinj 3.0.4.

On Tue, Dec 3, 2019 at 10:08 AM rkantos notifications@github.com wrote:

This is still an issue with CrossMgrImpinj.. It sometimes occurs (at least on Windows, probably not on Linux) even if the connection is just a bit unstable (via wifi) and the network connection is cut out. In this case CrossMgrImpinj does recognize the connection is cut, but still gets stuck on "Waiting for "get time" command from CrossMgr...".

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/esitarski/CrossMgr/issues/31?email_source=notifications&email_token=AABGXKP4R7P73GVVRVCHPGDQWZZATA5CNFSM4IGPU6OKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEFZWCKY#issuecomment-561209643, or unsubscribe https://github.com/notifications/unsubscribe-auth/AABGXKNY3S4YE4VYBV3QCU3QWZZATANCNFSM4IGPU6OA .

Edward Sitarski

rkantos commented 4 years ago

The status you reported shows the problem happens in the handshake part of the connect, not tag transmit. Your network must be very unstable. That makes it tough (impossible?) to reproduce as the handshake takes a very short time. I reviewed the handshake code and made it simpler, and this made it more straightforward to handle errors. The new reconnect logic should now handle all failures during the handshake, but I wasn't able to test all the failure cases. Updates are available in CrossMgrImpinj 3.0.4. … On Tue, Dec 3, 2019 at 10:08 AM rkantos @.***> wrote: This is still an issue with CrossMgrImpinj.. It sometimes occurs (at least on Windows, probably not on Linux) even if the connection is just a bit unstable (via wifi) and the network connection is cut out. In this case CrossMgrImpinj does recognize the connection is cut, but still gets stuck on "Waiting for "get time" command from CrossMgr...". — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#31?email_source=notifications&email_token=AABGXKP4R7P73GVVRVCHPGDQWZZATA5CNFSM4IGPU6OKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEFZWCKY#issuecomment-561209643>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AABGXKNY3S4YE4VYBV3QCU3QWZZATANCNFSM4IGPU6OA . -- Edward Sitarski

That is true, the CrossMgr part of the reconnection bug is hard to reproduce. I think I will be able to reproduce it at some point once I have thought about how to plan a test for it.

I apologize for the confusion, as I actually first opened this bug, because of what was in the first post; e.g. the USB-networking card disconnection and the subsequent non-existant connection attempts of CrossMgrImpinj.

I now retested both versions of CrossMgrImpinj, and the bug on reconnection from CrossMgrImpinj to the reader is still present. Contrary to the comment about the second bug affecting the CrossMgrImpinj-CrossMgr connection part, the original, e.g. the reader disconnection bug is simple to reproduce. I see this as being the much more important bug to fix, as that is what actually leads to tags being missed completely.

I know made a demonstration video about this part:

The same behaviour is apparrent when disabling the ethernet adapter manually from Control Panel\Network and Internet\Network Connections. Reconnecting to the reader works as expected when just disconnecting the Ethernet-cable from the USB adapter.

While I try to induce both bugs, the reader disconnection bug becomes apparrent within the first minute. The latest version of 3.0.4 is installed after testing with 3.0.3.

esitarski commented 4 years ago

So... the problem is between CrossMgrImpinj and the RFID reader, not CrossMgrImpinj to CrossMgr. I wonder if the network connector is assigning different IPs to the reader when then network goes up and down. Have you tried using static IP addresses?

On Mon, Dec 9, 2019 at 2:34 PM rkantos notifications@github.com wrote:

The status you reported shows the problem happens in the handshake part of the connect, not tag transmit. Your network must be very unstable. That makes it tough (impossible?) to reproduce as the handshake takes a very short time. I reviewed the handshake code and made it simpler, and this made it more straightforward to handle errors. The new reconnect logic should now handle all failures during the handshake, but I wasn't able to test all the failure cases. Updates are available in CrossMgrImpinj 3.0.4. … <#m8531066972873055629> On Tue, Dec 3, 2019 at 10:08 AM rkantos @.***> wrote: This is still an issue with CrossMgrImpinj.. It sometimes occurs (at least on Windows, probably not on Linux) even if the connection is just a bit unstable (via wifi) and the network connection is cut out. In this case CrossMgrImpinj does recognize the connection is cut, but still gets stuck on "Waiting for "get time" command from CrossMgr...". — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#31 https://github.com/esitarski/CrossMgr/issues/31?email_source=notifications&email_token=AABGXKP4R7P73GVVRVCHPGDQWZZATA5CNFSM4IGPU6OKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEFZWCKY#issuecomment-561209643>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AABGXKNY3S4YE4VYBV3QCU3QWZZATANCNFSM4IGPU6OA . -- Edward Sitarski

That is true, the CrossMgr part of the reconnection bug is hard to reproduce. I think I will be able to reproduce it at some point once I have thought about how to plan a test for it.

I apologize for the confusion, as I actually first opened this bug, because of what was in the first post; e.g. the USB-networking card disconnection and the subsequent non-existant connection attempts of CrossMgrImpinj.

I now retested both versions of CrossMgrImpinj, and the bug on reconnection from CrossMgrImpinj to the reader is still present. Contrary to the comment about the second bug affecting the CrossMgrImpinj-CrossMgr connection part, the original, e.g. the reader disconnection bug is simple to reproduce. I see this as being the much more important bug to fix, as that is what actually leads to tags being missed completely.

I know made a demonstration video about this part:

The same behaviour is apparrent when disabling the ethernet adapter manually from Control Panel\Network and Internet\Network Connections. Reconnecting to the reader works as expected when just disconnecting the Ethernet-cable from the USB adapter.

While I try to induce both bugs, the reader disconnection bug becomes apparrent within the first minute.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/esitarski/CrossMgr/issues/31?email_source=notifications&email_token=AABGXKMI3U7GB4KDMRKROA3QX2MVBA5CNFSM4IGPU6OKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEGKMMSI#issuecomment-563398217, or unsubscribe https://github.com/notifications/unsubscribe-auth/AABGXKPA5SMMU2FHBN36FUTQX2MVBANCNFSM4IGPU6OA .

Edward Sitarski

rkantos commented 4 years ago

So... the problem is between CrossMgrImpinj and the RFID reader, not CrossMgrImpinj to CrossMgr. I wonder if the network connector is assigning different IPs to the reader when then network goes up and down. Have you tried using static IP addresses? … On Mon, Dec 9, 2019 at 2:34 PM rkantos @.> wrote: The status you reported shows the problem happens in the handshake part of the connect, not tag transmit. Your network must be very unstable. That makes it tough (impossible?) to reproduce as the handshake takes a very short time. I reviewed the handshake code and made it simpler, and this made it more straightforward to handle errors. The new reconnect logic should now handle all failures during the handshake, but I wasn't able to test all the failure cases. Updates are available in CrossMgrImpinj 3.0.4. … <#m8531066972873055629> On Tue, Dec 3, 2019 at 10:08 AM rkantos @.> wrote: This is still an issue with CrossMgrImpinj.. It sometimes occurs (at least on Windows, probably not on Linux) even if the connection is just a bit unstable (via wifi) and the network connection is cut out. In this case CrossMgrImpinj does recognize the connection is cut, but still gets stuck on "Waiting for "get time" command from CrossMgr...". — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#31 <#31>?email_source=notifications&email_token=AABGXKP4R7P73GVVRVCHPGDQWZZATA5CNFSM4IGPU6OKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEFZWCKY#issuecomment-561209643>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AABGXKNY3S4YE4VYBV3QCU3QWZZATANCNFSM4IGPU6OA . -- Edward Sitarski That is true, the CrossMgr part of the reconnection bug is hard to reproduce. I think I will be able to reproduce it at some point once I have thought about how to plan a test for it. I apologize for the confusion, as I actually first opened this bug, because of what was in the first post; e.g. the USB-networking card disconnection and the subsequent non-existant connection attempts of CrossMgrImpinj. I now retested both versions of CrossMgrImpinj, and the bug on reconnection from CrossMgrImpinj to the reader is still present. Contrary to the comment about the second bug affecting the CrossMgrImpinj-CrossMgr connection part, the original, e.g. the reader disconnection bug is simple to reproduce. I see this as being the much more important bug to fix, as that is what actually leads to tags being missed completely. I know made a demonstration video about this part: The same behaviour is apparrent when disabling the ethernet adapter manually from Control Panel\Network and Internet\Network Connections. Reconnecting to the reader works as expected when just disconnecting the Ethernet-cable from the USB adapter. While I try to induce both bugs, the reader disconnection bug becomes apparrent within the first minute. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#31?email_source=notifications&email_token=AABGXKMI3U7GB4KDMRKROA3QX2MVBA5CNFSM4IGPU6OKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEGKMMSI#issuecomment-563398217>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AABGXKPA5SMMU2FHBN36FUTQX2MVBANCNFSM4IGPU6OA . -- Edward Sitarski I am using static IP on the reader.

Were you able to replicate the problem by disabling the network adapter where the reader is connected on Windows?

It just seems that Python/CrossMgrImpinj on Windows cannot recognize that the network interface is completely gone. The network disconnection to the reader is not recognized at all by CrossMgrImpinj when the network adapter is disconnected.

As shown in the demo, CrossMgrImpinj "hangs" when the network adapter is disabled, and doesn't recognize the connection to the reader is lost. In other words, the CrossMgrImpinj window stays green and is just showing the last displayed tag. No reconnection is attempted before reset / restart of CrossMgrImpinj. (as in the Youtube demo)

Since this issue doesn't exist on Linux, where the unplugged network adapter properly makes CrossMgrImpinj start reconnecting: I think this is related to how Python works with network connections on Windows.

I will also try to analyze the logic where CrossMgrImpinj checks for reader status / whether it is up.

mbuckaway commented 4 years ago

I've followed this issue somewhat, and it smell a lot like a problem I had on MacOSX some two years ago where CrossMgrImpinj would not find CrossMgr when a race was started without hitting the reset button. I had to refactor Ed's code to properly null the socket (not just refine it) to get it to drop the connection and properly reconnect. When I did that, it worked.

I looked at the other side of the code for connecting to the reader, and I saw the same problem code. The socket is not set to None when a reconnect is required.

I will be visiting the Windows build shortly (I want to automate the build on github) and will be in a position to actually attempt to duplicate the problem.

esitarski commented 4 years ago

OK. I added voodoo code self.readerSocket = None to Impinj.py. I can't think of a reason why this should make any difference because the existing socket gets garbage collected when it gets a new value, and all the failure code closes it.

mbuckaway commented 4 years ago

Existing sockets do not get dropped until the socket is set to null (None). There was a note in the documentation in this regard and this can be platform specific since sockets are handled by the C/C++ code the connects to the system libraries.

Testing seems to confirm the issue has been resolved.

I setup the following situation:

Installed CrossMgr 3.0.42 and CrossMgrImpinj from release v3.0.42-20191229222004 on Windows 10
I started CrossMgrImpinj to talk to a an Impinj reader. This may not be a 100% value test because the reader in on the home network rather than stand alone network.
I started CrossMgr, and loaded in an old simulation race. Under properties, I loaded up the RFID test. Crossmgr connected to CrossMgrImpinj and passed tags along.
I pulled the ethernet cable from the back of the windows computer. It took about 2 seconds for CrossMgrImpinj to notice the lost connection. CrossMgrImping kept retrying to find the reader. After waiting about 10 seconds, I plugged the cable back in. It took a couple retries, but after about 5 seconds CrossMgrImpinj found the reader and was all green again.

@rkantos - If this problem persist in your testing/in the field, please reopen the issue and give us a details steps to reproduce so we can setup the situation in our "lab".

rkantos commented 4 years ago

Existing sockets do not get dropped until the socket is set to null (None). There was a note in the documentation in this regard and this can be platform specific since sockets are handled by the C/C++ code the connects to the system libraries.

Testing seems to confirm the issue has been resolved.

I setup the following situation:

Installed CrossMgr 3.0.42 and CrossMgrImpinj from release v3.0.42-20191229222004 on Windows 10

I started CrossMgrImpinj to talk to a an Impinj reader. This may not be a 100% value test because the reader in on the home network rather than stand alone network.

I started CrossMgr, and loaded in an old simulation race. Under properties, I loaded up the RFID test. Crossmgr connected to CrossMgrImpinj and passed tags along.

I pulled the ethernet cable from the back of the windows computer. It took about 2 seconds for CrossMgrImpinj to notice the lost connection. CrossMgrImping kept retrying to find the reader. After waiting about 10 seconds, I plugged the cable back in. It took a couple retries, but after about 5 seconds CrossMgrImpinj found the reader and was all green again.

@rkantos - If this problem persist in your testing/in the field, please reopen the issue and give us a details steps to reproduce so we can setup the situation in our "lab".

I tested the newest version from Github releases, and the problem outlined in the title still holds true (on Linux there is no issue, only on Windows) Disconnecting the cable is recognized as a disconnection, disconnecting (e.g. USB to ethernet adapter) or disabling the network adapter is not. CrossMgrImpinj still does not recognize the disconnection and thus never attemps to reconnect, even if the reader is on the network. A reset after the network adapter has been reconnected or re-enabled is required, thus making the use of CrossMgrImpinj rather manual on Windows if such a situation occurs. (I imagine this becoming more and more of a issue in the future, with laptops losing their ethernet jacks.)

This Youtube video outlines the steps to reproduce the issue: https://youtu.be/HTQEpmM2WUU

I suggest the following logical fix for this problem on Windows:

Actually keep pinging the reader, maybe via arping. Or instead check the traceroute to the reader IP. On Windows if the network adapter is disabled or disconnected
Run the disconnection routine if the traceroute fails within one keep-alive interval period.

I cannot reopen the issue, since I didn't close it.

mbuckaway commented 4 years ago

Reopened.

mbuckaway commented 4 years ago

I'll look at this issue again, but it sounds more like an edge case. Running a ping or a traceroute is a hack not a fix.

rkantos commented 4 years ago

I'll look at this issue again, but it sounds more like an edge case. Running a ping or a traceroute is a hack not a fix.

Have you been able to replicate the issue?

I disagree it being an edge case, especially going further in time with more and more portable computers losing their native ethernet ports. When using an ethernet-adapter, a situation where the usb-cable gets unplugged can easily be imagined.

I'm not a professional programmer, so I was trying to give out and example of a fix, based on my understanding; Since the problem doesn't appear with the same Python (v3) code on Linux, I fathom Python on Windows (both v2 & v3) has some limitations to the current code being used when it comes to network adapters. Maybe there is another way to check for a lost network adapter too. Based on this thinking, I thought that using a keepalive on the Network layer (PING/ARPing) might be a more reliable method when checking a device, that is on the network.

Maybe something like wmic (could be run on Windows to check if the Network adapter is actually running, available and not disabled or unavailable?

https://stackoverflow.com/a/25554796/5776626
# Get NIC list and index number:
wmic nic get name, index

I still think using something on the Network layer, would be the most reliable method. WMIC might not play nice in all situations, the network will either respond (especially arping) or not, irrelevant to the status of the network adapter.

stuartlynne commented 4 years ago

Unless Microsoft has changed the network stack lately, simply unplugging the Ethernet cable should help replicate the problem.

The Windows network stack drops connections used by the network interface when the cable is unplugged.

Linux (and probably Mac?) do not. You can unplug and replug and active network connections will remain connected until a timeout occurs.

On Thu, Jan 23, 2020 at 11:26 AM rkantos notifications@github.com wrote:

I'll look at this issue again, but it sounds more like an edge case. Running a ping or a traceroute is a hack not a fix.

Have you been able to replicate the issue?

I disagree it being an edge case, especially going further in time with more and more portable computers losing their native ethernet ports. When using an ethernet-adapter, a situation where the usb-cable gets unplugged can easily be imagined.

I'm not a professional programmer, so I was trying to give out and example of a fix, based on my understanding; Since the problem doesn't appear with the same Python (v3) code on Linux, I fathom Python on Windows (both v2 & v3) has some limitations to the current code being used when it comes to network adapters. Maybe there is another way to check for a lost network adapter too. Based on this thinking, I thought that using a keepalive on the Network layer (PING/ARPing) might be a more reliable method when checking a device, that is on the network.

Maybe something like wmic (could be run on Windows to check if the Network adapter is actually running, available and not disabled or unavailable?

https://stackoverflow.com/a/25554796/5776626

Get NIC list and index number:

wmic nic get name, index

I still think using something on the Network layer, would be the most reliable method. WMIC might not play nice in all situations, the network will either respond (especially arping) or not, irrelevant to the status of the network adapter.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/esitarski/CrossMgr/issues/31?email_source=notifications&email_token=AACIUWIEJ5MDAKP3EOYG3RTQ7HVMZA5CNFSM4IGPU6OKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEJYRHEQ#issuecomment-577835922, or unsubscribe https://github.com/notifications/unsubscribe-auth/AACIUWID4JK5ERRAZZI62NLQ7HVMZANCNFSM4IGPU6OA .

-- __O____ -\<,____ ____()/()___

Stuart_Lynnestuart.lynne@gmail.com604-518-1749(m)__604-461-7532(h)

rkantos commented 4 years ago

Unless Microsoft has changed the network stack lately, simply unplugging the Ethernet cable should help replicate the problem. The Windows network stack drops connections used by the network interface when the cable is unplugged. Linux (and probably Mac?) do not. You can unplug and replug and active network connections will remain connected until a timeout occurs.

Again: This is incorrect, and does not describe the bug!

CrossMgrImpinj still works as it should when disconnecting the cable from the network adapter! CrossMgrImpinj doesn't recognize the disconnection on Windows, when the network adapter (not the cable) is unplugged or the network adapter disabled in the control panel. There is a difference!

Disconnecting the network cable is not the same thing as disconnecting a USB-ethernet adapter! The other leads to CrossMgrImpinj properly detecting network disconnection, the other does not.

In one sentence: CrossMgrImpinj does not recognize a disappeared network adapter on Windows.

Please, I ask you to replicate the bug yourself, it is trivial. Go in to the Control panel and disable the network adapter that is currently connected to the network the LLRP-reader is on.

Shortest way I can think of: (With CrossMgrImpinj connected to a reader and showing green) Win+R -> type in "ncpa.cpl" -> Run -> find your network adapter -> right-click -> disable

-> Observe CrossMgrImpinj staying green and never reconnecting to the reader, even if the network adapter is re-enabled.

mbuckaway commented 4 years ago

So, I went back to the original report and you are attempting to run windows on a compute stick where the problem is present. The problem reported is that when the USB Ethernet port disappears, CrossMgrImpinj fails to notice the dropped connection. Somehow the issue has morphed into a concern that everyone running windows on a laptop with USB Ethernet port will have this same issue, which I think is overstating the severity of the issue.

First, please remember this is an open source project. I believe Ed asks for a small one time fee, but it doesn't even cover any of the costs of the project including thing like Windows Signing Certs, etc.. So, while we might look into issues, I'm inclined to spend my free time working on issues that affect me directly (like the Linux and Mac builds) or a large number of users.

So, while is might seem interesting to use a compute stick, it is not something I would do. Whereas a Raspberry PI or similar UNIX computer would be a better choice (I'd use a Orange PI Zero myself with a high gain external antenna). You don't need the UI of CrossMgr for what it does. It is something I might be looking as this summer.

As for USB Ethernet is something all laptops will have. Yes, my Mac laptop which I used for race timing requires a dongle. Never had a issue with the cord coming out during a race. I currently use a cheap X86 laptop running Linux for race timing. After Windows Update forced a reboot on the Forest City Velodrome's computer in the middle of a event, I will never use Windows for a race timing system ever again. That said, there are those that prefer it. As the USB Ethernet getting disconnected on a laptop being an issue, I disagree. Kicking the cord from the computer is something you probably would notice during a race and a CrossMgr Reset should correct the issue once its plugged back in. As for an embedded solution, Windows is just a horrible choice.

That said, the problem with Crossmgrimpinj not detecting the disconnected USB ethernet port is more likely to be a issue with the operating system not passing the disconnected socket up the chain rather than the python code itself or there is something special needed for Windows. I found I had to modify the code to get it to work correctly on MacOSX.

If anyone wants to be helpful, please research the issue. It should be a simple matter of setting up a sample python server on one computer with a python receiver on another, and see what happens when the USB ethernet adapter is pulled. There are tons of examples on Stack Overflow. Then, ask on StackOverflow what windows python doesn't notice the dropped connection. Personally, I don't have the time to look into. Ed might, but currently is working at the MIlton Velodrome for the Track World Cup.

If someone comes up with a solution to the problem, and not a hack, I'll implement it.

Mark

mbuckaway commented 4 years ago

Thanks for the update. In the future, please work in the dev branch and merge changes to the dev branch. Dev is for testing, and master is for release.

mbuckaway commented 4 years ago

Please download and test the Development Build when complete. Confirm it is corrected, and then close the issue.

rkantos commented 4 years ago

With pull request #45 , the ConnectionResetError: [WinError 10054] An existing connection was forcibly closed by the remote host -error iss now catched in the right place, and CrossMgrImpinj properly runs the disconnection routine, and doesn't just freeze when a network adapter disappears on Windows.