search

MakerLabsVan / makerlabs-acm

MakerLabs Access Control Management (ACM) System
https://makerlabsvan.github.io/makerlabs-acm/index.html
MIT License
1 stars 0 forks source link

Laser Reader Freezing #15

Open MakerLabs-Van opened 5 years ago

MakerLabs-Van commented 5 years ago

The reader on the laser cutter would freeze on the "searching" function.

It would respond when the tag was removed or placed on the reader, but would never complete the search.

Tried to restart 3 times unsuccessful, then the reader rebooted on it's own. After 1 unsuccessful search it was able find the user and started the safety systems.

paulreimer commented 5 years ago

Do you know the time range when this occurred? And ideally the Maker Name or Tag ID used?

As well, can you describe what happened during the "unsuccessful search" behaviour?

MakerLabs-Van commented 5 years ago

-- Please reply above this line --

MakerLabs replied, on Feb 15 @ 1:33pm (PST):

The tag was presented to the reader. The tag ID and search bar appeared, and the loading bar slowly filled (approx 15 seconds), but then stopped.

This was from 1:50 - 2:03 today with the tags 4480 & 26368

MakerLabs Vancouver hello@makerlabs.com

How would you rate my reply? Great [1]    Okay [2]    Not Good [3]

Links:

[1] https://secure.helpscout.net/satisfaction/237449671/record/2169294501/1/ [2] https://secure.helpscout.net/satisfaction/237449671/record/2169294501/2/ [3] https://secure.helpscout.net/satisfaction/237449671/record/2169294501/3/

Makerlabsvan/makerlabs-Acm sent a message, on Feb 15 @ 1:23pm (PST):

Do you know the time range when this occurred? And ideally the Maker

Name or Tag ID used? 

As well, can you describe what happened during the "unsuccessful

search" behaviour? 

-

You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub [1], or mute the thread [2].

Links:

[1] https://github.com/MakerLabsVan/makerlabs-acm/issues/15#issuecomment-464205211 [2] https://github.com/notifications/unsubscribe-auth/ArM3dQaIqMTVGP9HvUmLcnbxKTQ7t26pks5vNyVegaJpZM4a-caR

paulreimer commented 5 years ago

Also, during "Tried to restart 3 times unsuccessful, then the reader rebooted on it's own", what were the messages displayed on the screen at that time?

MakerLabs-Van commented 5 years ago

The reader was restarted:

ACM Logged In appeared on the screen, then when presented with the tag would search.

when it restarted on it's own:

paulreimer commented 5 years ago

OK. Just to confirm the time range, do you mean 12:50 ~ 1:03?

And, was the reader manually restarted the 3 times (using the reset button?)

So, the behaviour was:

  1. Could not successfully get info for tag (Tag ___ Search... -> 15s timeout)
    • Including repeated scans (and to confirm, each time it would beep after reading the tag and start a new 15s progress bar? Did it also beep each time the tag was removed?)
  2. Pressed reset button (or cycled USB power) 3 times, same behaviour as (1)
  3. Rebooted on its own (potentially unrelated to the manual resets)
  4. Failed initial tag search
  5. System normal on second tag search

I can say that (4) may be expected after a reboot (only the first tag read after reset may not trigger operation). And (1) + (2) sound like a network issue (which would eventually lead to (3), automatic reboot). Do you know if other wifi devices were working at that time?

paulreimer commented 5 years ago

It looks like this made it to the spreadsheet -- does "Activity, De-duped", starting at row 7308, describe the time range? (in that case: not a networking issue but something else.)

MakerLabs-Van commented 5 years ago

Yes, that is the correct time range. I don't know if all the operations made it to the time sheet, because we tried scanning the tags many times,

MakerLabs-Van commented 5 years ago

This is consistently happening with multiple members and FOBS, both guest and assigned. It's also not sending the signal for the safety systems to turn off when the FOB is removed.

paulreimer commented 5 years ago

OK, I have modified the logging from the attached Raspberry Pi's to be more durable. I don't have the logs for the time of the original issue, but I may have the logs for the more recent occurrences.

If you know the time ranges when this occurs (as well as times when it is reset manually), that would be most helpful. I'll keep an watching the (new) logs to see if it happens again.

paulreimer commented 5 years ago

Just curious if this issue has re-occurred at all? I have all the logs since the previous message, and I believe I can leave it enable for the near future (I think the amount of logs is within the free quota).

MakerLabs-Van commented 5 years ago

-- Please reply above this line --

MakerLabs replied, on Feb 21 @ 7:27pm (PST):

Yes, it occurred again at about 9:40am yesterday,

Alyssa

-- MakerLabs Vancouver hello@makerlabs.com

How would you rate my reply? Great [1]    Okay [2]    Not Good [3]

Links:

[1] https://secure.helpscout.net/satisfaction/238817720/record/2183538830/1/ [2] https://secure.helpscout.net/satisfaction/238817720/record/2183538830/2/ [3] https://secure.helpscout.net/satisfaction/238817720/record/2183538830/3/

Makerlabsvan/makerlabs-Acm sent a message, on Feb 21 @ 6:54pm (PST):

Just curious if this issue has re-occurred at all? I have all the

logs since the previous message, and I believe I can leave it enable for the near future (I think the amount of logs is within the free quota). 

-

You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub [1], or mute the thread [2].

Links:

[1] https://github.com/MakerLabsVan/makerlabs-acm/issues/15#issuecomment-466254789 [2] https://github.com/notifications/unsubscribe-auth/ArM3dZO0TtN-sFEuKX8ToMBnHhBT5mzfks5vP1vrgaJpZM4a-caR

paulreimer commented 5 years ago

Hey this is really interesting. Glad I have some logs to look at, but at the time you mention, the Raspberry Pi that I have connected to the device had just finished rebooting (this is not expected to happen except e.g. in a power outage).

Meanwhile, throughout the logs there is a recurring message about the Raspberry Pi running into a low-voltage condition.

Both these things make me suspicious of the USB brick that is powering the Pi (and by extension the TARS reader unit, which is being powered from the Pi, for remote logging purposes).

If there is a good quality USB power brick for the Pi, that may fix some of these issues. (The "Low Voltage" condition warnings should stop, as a confirmation of that test). Could you send a picture of what is powering that Pi? I believe at least 5V@2A is required, and 3A would be even better.

MakerLabs-Van commented 5 years ago

-- Please reply above this line --

MakerLabs replied, on Feb 21 @ 8:27pm (PST):

Hello,

I believe that at 9:40 I was rebooting the reader to see if that would help (it was probably occurring from 9:30-9:42 or so)

I will try and find a better brick for the reader, I've attached a photo of the one that we had plugged in,

Thanks,

Alyssa

-- MakerLabs Vancouver hello@makerlabs.com

How would you rate my reply? Great [1]    Okay [2]    Not Good [3]

Links:

[1] https://secure.helpscout.net/satisfaction/238825119/record/2183603539/1/ [2] https://secure.helpscout.net/satisfaction/238825119/record/2183603539/2/ [3] https://secure.helpscout.net/satisfaction/238825119/record/2183603539/3/

Makerlabsvan/makerlabs-Acm sent a message, on Feb 21 @ 8:01pm (PST):

Hey this is really interesting. Glad I have some logs to look at, but

at the time you mention, the Raspberry Pi that I have connected to the device had just finished rebooting. 

Meanwhile, throughout the logs there is a recurring message about the

Raspberry Pi running into a low-voltage condition. 

Both these things make me suspicious of the USB brick that is

powering the Pi (and the TARS reader unit, which is being powered from the Pi, for remote logging purposes). 

If there is a good quality USB power brick for the Pi, that may fix

some of these issues. (The "Low Voltage" condition warnings should stop, as a confirmation of that test). Could you send a picture of what is powering that Pi? I believe at least 5V@2A is required, and 3A would be even better. 

-

You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub [1], or mute the thread [2].

Links:

[1] https://github.com/MakerLabsVan/makerlabs-acm/issues/15#issuecomment-466265542 [2] https://github.com/notifications/unsubscribe-auth/ArM3daTLpH1zlbPJtNJB4hbOeQwEpqjIks5vP2uvgaJpZM4a-caR

MakerLabs-Van commented 5 years ago

This is still happening on a regular basis, it happened today with tag ID #13952. I don't know the sequence, but it normally happens and we restart the reader 1 to 3 times, and then it has a 15-30 seconds to reboot and then search before registering the tag and allowing the laser cutting.

MakerLabs-Van commented 5 years ago

After it having trouble finding my tag and freezing today at approximately 11:55am, we have taken the laser fob system offline and are running the bypass to use the laser cutter.

paulreimer commented 5 years ago

Noted.

I have some ideas on how to approach these reliability issues.

The simplest approach would involve modifying the firmware to switch from HTTP to MQTT (still re-using AWS lambda functions -> Google services as before). It is a more efficient use of the network connection, and the device logic becomes simpler. I can look into estimating the time it would take to perform these changes.

An alternate approach would be to modify the reader device to integrate more closely with the Windows PC that controls the laser cutter, and for the reader device to forward scanned tag IDs to the PC to use its much faster wifi connection (and perhaps display a popup on the PC screen with the User info when someone logs in, and/or completes a job). This is more complicated than the above suggestion (HTTP -> MQTT), but does offer more features by leveraging the PC.

Both of these approaches have the potential to resolve this issue (#15), as well as #13 and #16 .

MakerLabs-Van commented 5 years ago

-- Please reply above this line --

MakerLabs replied, on Apr 9 @ 1:54pm (PDT):

Hi Paul,

Thanks we will await your estimated for those changes. When should we expect follow-up?

Alyssa

-- MakerLabs Vancouver hello@makerlabs.com

How would you rate my reply? Great [1]    Okay [2]    Not Good [3]

Links:

[1] https://secure.helpscout.net/satisfaction/248185599/record/2296158998/1/ [2] https://secure.helpscout.net/satisfaction/248185599/record/2296158998/2/ [3] https://secure.helpscout.net/satisfaction/248185599/record/2296158998/3/

Makerlabsvan/makerlabs-Acm sent a message, on Apr 7 @ 8:30pm (PDT):

Noted. 

I have some ideas on how to approach these reliability issues. 

The simplest approach would involve modifying the firmware to switch

from HTTP to MQTT (still re-using AWS lambda functions -> Google services as before). It is a more efficient use of the network connection, and the device logic becomes simpler. I can look into estimating the time it would take to perform these changes. 

An alternate approach would be to modify the reader device to

integrate more closely with the Windows PC that controls the laser cutter, and for the reader device to forward scanned tag IDs to the PC to use its much faster wifi connection (and perhaps display a popup on the PC screen with the User info someone logs in, and/or completes a job). This is more complicated than the above suggestion (HTTP -> MQTT), but does offer more features by leveraging the PC. 

Both of these approaches have the potential to resolve this issue

(#15 [1]), as well as #13 [2] and #16 [3] . 

-

You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub [4], or mute the thread [5].

Links:

[1] https://github.com/MakerLabsVan/makerlabs-acm/issues/15 [2] https://github.com/MakerLabsVan/makerlabs-acm/issues/13 [3] https://github.com/MakerLabsVan/makerlabs-acm/issues/16 [4] https://github.com/MakerLabsVan/makerlabs-acm/issues/15#issuecomment-480671165 [5] https://github.com/notifications/unsubscribe-auth/ArM3dTlcMLV4vM6zjxOEKAOoM1qznGKtks5verfGgaJpZM4a-caR

MakerLabs-Van commented 5 years ago

-- Please reply above this line --

MakerLabs replied, on Apr 17 @ 1:20pm (PDT):

Hi Paul,

Just following up on those estimates you were going to make. Is it possible to get that sometime within the week?

I will check in this Sunday just to be safe.

Thanks,

Marc

MakerLabs Vancouver hello@makerlabs.com

How would you rate my reply? Great [1]    Okay [2]    Not Good [3]

Links:

[1] https://secure.helpscout.net/satisfaction/248185599/record/2315947582/1/ [2] https://secure.helpscout.net/satisfaction/248185599/record/2315947582/2/ [3] https://secure.helpscout.net/satisfaction/248185599/record/2315947582/3/

MakerLabs replied, on Apr 9 @ 1:54pm (PDT):

Hi Paul,

Thanks we will await your estimated for those changes. When should we expect follow-up?

Alyssa

-- MakerLabs Vancouver hello@makerlabs.com

Makerlabsvan/makerlabs-Acm sent a message, on Apr 7 @ 8:30pm (PDT):

Noted. 

I have some ideas on how to approach these reliability issues. 

The simplest approach would involve modifying the firmware to switch

from HTTP to MQTT (still re-using AWS lambda functions -> Google services as before). It is a more efficient use of the network connection, and the device logic becomes simpler. I can look into estimating the time it would take to perform these changes. 

An alternate approach would be to modify the reader device to

integrate more closely with the Windows PC that controls the laser cutter, and for the reader device to forward scanned tag IDs to the PC to use its much faster wifi connection (and perhaps display a popup on the PC screen with the User info someone logs in, and/or completes a job). This is more complicated than the above suggestion (HTTP -> MQTT), but does offer more features by leveraging the PC. 

Both of these approaches have the potential to resolve this issue

(#15 [1]), as well as #13 [2] and #16 [3] . 

-

You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub [4], or mute the thread [5].

Links:

[1] https://github.com/MakerLabsVan/makerlabs-acm/issues/15 [2] https://github.com/MakerLabsVan/makerlabs-acm/issues/13 [3] https://github.com/MakerLabsVan/makerlabs-acm/issues/16 [4] https://github.com/MakerLabsVan/makerlabs-acm/issues/15#issuecomment-480671165 [5] https://github.com/notifications/unsubscribe-auth/ArM3dTlcMLV4vM6zjxOEKAOoM1qznGKtks5verfGgaJpZM4a-caR

paulreimer commented 5 years ago

Hi Marc, all,

Given the amount of risk/uncertainty in estimating research work for this solution, it is not possible to make an estimate until approximately the 80% completion mark. To that end, I can report that the switch to MQTT should take a total time of approximately 20~25 hours including install time (but not including new issues discovered as a result of installation/testing), of which 16 hours have already been completed.

At this time the preliminary results with MQTT show the reliability should improve (overall there are fewer network connections used and it does not need to wait for a response between each scan). Performance is also improved approx. 2x for the time-tracking use-case (e.g. laser cutter units), and is compatible with the previous HTTP-based system (e.g. the Front Desk can continue to use the existing HTTP approach if desired)

Please let me know if you would like me to continue with the implementation, or investigate other possible issues, and/or if you would like to explore the other option (i.e. using the Windows laptops for internet/WiFi traffic)

MakerLabs-Van commented 5 years ago

-- Please reply above this line --

MakerLabs replied, on Apr 22 @ 8:40pm (PDT):

Hi Paul,

Had a look at the github conversation you were having with Alyssa, are we still getting the power issues on the pi? We can order some 3A power supply's to help address the problem. Do you think it might help?

Also, is it a pi 3 that the reader system is running on? Is the pi communicating with the database over the network or is the reader doing it?

Thanks,

Marc

MakerLabs Vancouver hello@makerlabs.com

How would you rate my reply? Great [1]    Okay [2]    Not Good [3]

Links:

[1] https://secure.helpscout.net/satisfaction/251128537/record/2328048663/1/ [2] https://secure.helpscout.net/satisfaction/251128537/record/2328048663/2/ [3] https://secure.helpscout.net/satisfaction/251128537/record/2328048663/3/

Makerlabsvan/makerlabs-Acm sent a message, on Apr 22 @ 9:15am (PDT):

Hi Marc, all, 

Given the amount of risk/uncertainty in estimating research work for

this solution, it is not possible to make an estimate until approximately the 80% completion mark. To that end, I can report that the switch to MQTT should take a total time of approximately 20 hours including install time (but not including new issues discovered as a result of installation/testing), of which 16 hours have already been completed. 

At this time the preliminary results with MQTT show the reliability

should improve (overall there are fewer network connections used and it does not need to wait for a response between each scan). Performance is also improved approx. 2x for the time-tracking use-case (e.g. laser cutter units), and is compatible with the previous HTTP-based system (e.g. the Front Desk can continue to use the existing HTTP approach if desired) 

Please let me know if you would like me to continue with the

implementation, or investigate other possible issues, and/or if you would like to explore the other option (i.e. using the Windows laptops for internet/WiFi traffic) 

-

You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub [1], or mute the thread [2].

Links:

[1] https://github.com/MakerLabsVan/makerlabs-acm/issues/15#issuecomment-485463328 [2] https://github.com/notifications/unsubscribe-auth/AKZTO5PEAXZEI5IF7F4UECTPRXQBVANCNFSM4GXZY2IQ

paulreimer commented 5 years ago

Hi Marc,

Yes, there is still under-voltage condition regularly on the Pi. You could try a beefier power supply to the Pi, and I can let you know if the power quality is improved (e.g. the error goes away).

The Pi is only for out-of-band monitoring, and could be removed (so the reader units could be powered by the DC barrel jack -- no Pi or USB cable should be used in that case). You could also try that (now, before, or after the 3A supply tests) to see if this is issue goes away permanently.

FYI I have not yet found the Pi to be useful in having useful logs near the time when something was reported, but without the Pi I would be unable to check into things. But you could take the Pi out -- try it, and if the problem ever comes up again (then it wasn't necessarily the under-voltage issue as the root cause), then put the Pi back in.

MakerLabs-Van commented 5 years ago

-- Please reply above this line --

MakerLabs replied, on Apr 23 @ 11:24am (PDT):

Thanks Paul,

I'm going to give the new power supply idea a try, then I will try it without the pi and see if anything changes. Will get back to you about possibly going forward with other solutions once I've done this.

Marc

MakerLabs Vancouver hello@makerlabs.com

How would you rate my reply? Great [1]    Okay [2]    Not Good [3]

Links:

[1] https://secure.helpscout.net/satisfaction/251281929/record/2330188288/1/ [2] https://secure.helpscout.net/satisfaction/251281929/record/2330188288/2/ [3] https://secure.helpscout.net/satisfaction/251281929/record/2330188288/3/

Makerlabsvan/makerlabs-Acm sent a message, on Apr 22 @ 9:40pm (PDT):

Hi Marc, 

Yes, there is still under-voltage condition regularly on the Pi. You

could try a beefier power supply to the Pi, and I can let you know if the power quality is improved (e.g. the error goes away). 

The Pi is only for out-of-band monitoring, and could be removed (so

the reader units could be powered by the DC barrel jack -- no Pi or USB cable should be used in that case). You could also try that (now, before, or after the 3A supply tests) to see if this is issue goes away permanently. 

FYI I have not yet found the Pi to be useful in having useful logs

near the time when something was reported, but without the Pi I would be unable to check into things. But you could take the Pi out -- try it, and if the problem ever comes up again (then it wasn't necessarily the under-voltage issue as the root cause), then put the Pi back in. 

-

You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub [1], or mute the thread [2].

Links:

[1] https://github.com/MakerLabsVan/makerlabs-acm/issues/15#issuecomment-485640936 [2] https://github.com/notifications/unsubscribe-auth/AKZTO5JX2XSKK6TVUI6OBQLPR2HM7ANCNFSM4GXZY2IQ

paulreimer commented 5 years ago

OK, if you let me know when the power supply has been swapped out, I can let you know if the under-voltage warning is still being logged.

MakerLabs-Van commented 5 years ago

-- Please reply above this line --

MakerLabs replied, on Apr 24 @ 12:44pm (PDT):

OK!

MakerLabs Vancouver hello@makerlabs.com

How would you rate my reply? Great [1]    Okay [2]    Not Good [3]

Links:

[1] https://secure.helpscout.net/satisfaction/251692082/record/2333336756/1/ [2] https://secure.helpscout.net/satisfaction/251692082/record/2333336756/2/ [3] https://secure.helpscout.net/satisfaction/251692082/record/2333336756/3/

Makerlabsvan/makerlabs-Acm sent a message, on Apr 24 @ 8:22am (PDT):

OK, if you let me know when the power supply has been swapped out, I

can let you know if the under-voltage warning is still being logged. 

-

You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub [1], or mute the thread [2].

Links:

[1] https://github.com/MakerLabsVan/makerlabs-acm/issues/15#issuecomment-486288710 [2] https://github.com/notifications/unsubscribe-auth/AKZTO5IJKMHFP2V6744IOKDPSB3LTANCNFSM4GXZY2IQ