Closed thematrixdev closed 2 years ago
The phenomenon is similar to that of a reboot due to a heavy load.
The last restart time can be found below. grep Reboot /media/mmc/atomhack.log
In the case of AtomSwing, the iCamera_app log is small, so the cause may not be clear.
I am in GMT+8. The time below is GMT+9.
2022/04/03 23:02:41 : Reboot & Start watchdog
2022/04/06 00:12:51 : Reboot & Start watchdog
So it freezed just after I have got my recording finished. My Python script keeps on reading the RTSP stream and detect for human (OpenCV and MediaPipe). So I think there won't be any extra loading at any time?
May I know where iCamera_app log is located?
The log for iCamera_app is /tmp/log/atom.log. However, it is a file on tmpfs, so it will disappear after restart. Near the end of /scripts/atom_init.sh, there is a place where the output of iCamera_app is redirected. You can change this to /media/mmc/atom.log so that it remains even after restart. However, because of page caching, it is likely that the last portion will not be recorded.
By the way, I have found that after reboot, no PING is made.
$ ps | grep watchdog
10 root 0 SW [watchdog/0]
735 root 2744 S {watchdog.sh} /bin/sh /scripts/watchdog.sh
And RTSP is working.
The version number on bottom-right hand corner is not showing. Maybe it is due to network check fails?
Is there a webcmd? ps | grep webcmd.sh
519 root 2744 S {webcmd.sh} /bin/sh /scripts/webcmd.sh
539 root 2744 S {webcmd.sh} /bin/sh /scripts/webcmd.sh
I am not sure. Can you add the code to output the log around the 1min interval in watchdog.sh and take a look?
At 09:51 the PING is resumed.
[root@atom-3603:~]# ps | grep webcmd.sh
519 root 2744 S {webcmd.sh} /bin/sh /scripts/webcmd.sh
539 root 2744 S {webcmd.sh} /bin/sh /scripts/webcmd.sh
5340 root 2808 S grep webcmd.sh
[root@atom-3603:~]# ps | grep watchdog
10 root 0 SW [watchdog/0]
735 root 2744 S {watchdog.sh} /bin/sh /scripts/watchdog.sh
5361 root 2808 S grep watchdog
The latest version number is also being displayed on web UI.
As I understand, when the web UI loads, an AJAX request is made to cmd.cgi. And cmd.cgi gets the latest version from GitHub, right? If so, I doubt Internet connection was lost overnight (for this specific camera only).
P.S. For the case of this camera, it was always powered on last night. I have rebooted the router the camera was connected to, and this case happened.
I have one camera outside my flat. I have another camera setup for my neighborhood. His camera is connected to my mini router setup outside my flat beside my camera. The mini router is an access point, connecting to my main router with LAN cable. My camera is connected to my main router.
Today at 16:22 my neighborhood's camera was reported no PING. At 16:35 my camera was reported no PING. I have checked both the main router and mini router had not rebooted at that time. They are grey-ed out in Atom-App. I was in office at that time. So I could do nothing. When I go home, I see both of them are flashing blue and yellow LED, like the video in the first post.
The healthcheck.log
of my neighborhood's camera looks like this, requesting per minute:
# tail -f healthcheck.log
OK2022/04/06 21:52:13
OK2022/04/06 21:53:14
OK2022/04/06 21:54:15
OK2022/04/06 21:55:17
OK2022/04/06 21:56:18
OK2022/04/06 21:57:19
OK2022/04/06 21:58:21
OK2022/04/06 21:59:22
OK2022/04/06 22:16:51
OK
The one on my camera... around every 20 seconds?
# tail -f healthcheck.log
OK2022/04/07 20:37:29
OK2022/04/07 20:37:50
OK2022/04/07 20:38:12
OK2022/04/07 20:38:33
OK2022/04/07 20:38:55
OK2022/04/07 20:39:16
OK2022/04/07 20:39:38
OK2022/04/07 20:40:01
OK2022/04/07 20:40:24
OK
I have not changed about /tmp/log/atom.log
.
Maybe we could save the log file by date, and store time, cpu and memory load at that time?
For all of my cameras, all motion detection on Atom-App are disabled. All functions on AtomCam-Tools are disabled except RTSP-via-HTTP. I believe it should not due to loading.
Just noticed "wifi restart" in /media/mmc/atomhack.log
.
My camera
Fri Apr 8 14:22:03 JST 2022
total used free shared buffers cached
Mem: 74192 68404 5788 0 636 20836
-/+ buffers/cache: 46932 27260
Swap: 131068 564 130504
Fri Apr 8 15:23:27 JST 2022
total used free shared buffers cached
Mem: 74192 68452 5740 0 636 20888
-/+ buffers/cache: 46928 27264
Swap: 131068 564 130504
Fri Apr 8 16:24:52 JST 2022
total used free shared buffers cached
Mem: 74192 69100 5092 0 636 20948
-/+ buffers/cache: 47516 26676
Swap: 131068 564 130504
Fri Apr 8 17:26:15 JST 2022
total used free shared buffers cached
Mem: 74192 68572 5620 0 636 21000
-/+ buffers/cache: 46936 27256
Swap: 131068 564 130504
WiFi restart
WiFi restart
WiFi restart
WiFi restart
WiFi restart
WiFi restart
WiFi restart
WiFi restart
WiFi restart
WiFi restart
WiFi restart
WiFi restart
WiFi restart
WiFi restart
WiFi restart
WiFi restart
WiFi restart
WiFi restart
WiFi restart
WiFi restart
WiFi restart
WiFi restart
WiFi restart
WiFi restart
WiFi restart
WiFi restart
WiFi restart
WiFi restart
WiFi restart
WiFi restart
WiFi restart
WiFi restart
WiFi restart
WiFi restart
WiFi restart
WiFi restart
WiFi restart
WiFi restart
WiFi restart
WiFi restart
WiFi restart
WiFi restart
WiFi restart
WiFi restart
WiFi restart
WiFi restart
WiFi restart
WiFi restart
WiFi restart
WiFi restart
WiFi restart
WiFi restart
Fri Apr 8 20:12:05 JST 2022
total used free shared buffers cached
Mem: 74192 71004 3188 0 476 23172
-/+ buffers/cache: 47356 26836
Swap: 131068 324 130744
WiFi restart
WiFi restart
WiFi restart
WiFi restart
WiFi restart
WiFi restart
WiFi restart
WiFi restart
WiFi restart
WiFi restart
WiFi restart
WiFi restart
WiFi restart
WiFi restart
WiFi restart
WiFi restart
WiFi restart
WiFi restart
WiFi restart
WiFi restart
WiFi restart
WiFi restart
WiFi restart
WiFi restart
WiFi restart
WiFi restart
WiFi restart
WiFi restart
WiFi restart
WiFi restart
WiFi restart
WiFi restart
WiFi restart
WiFi restart
WiFi restart
WiFi restart
WiFi restart
WiFi restart
WiFi restart
WiFi restart
WiFi restart
WiFi restart
2022/04/08 22:16:11 : Reboot & Start watchdog
My neighbor's
Fri Apr 8 14:10:59 JST 2022
total used free shared buffers cached
Mem: 74192 70048 4144 0 752 21724
-/+ buffers/cache: 47572 26620
Swap: 131068 2108 128960
Fri Apr 8 15:12:23 JST 2022
total used free shared buffers cached
Mem: 74192 70096 4096 0 752 21776
-/+ buffers/cache: 47568 26624
Swap: 131068 2108 128960
Fri Apr 8 16:13:42 JST 2022
total used free shared buffers cached
Mem: 74192 70124 4068 0 752 21824
-/+ buffers/cache: 47548 26644
Swap: 131068 2108 128960
Fri Apr 8 17:15:01 JST 2022
total used free shared buffers cached
Mem: 74192 70196 3996 0 752 21688
-/+ buffers/cache: 47756 26436
Swap: 131068 2088 128980
WiFi restart
WiFi restart
WiFi restart
WiFi restart
WiFi restart
WiFi restart
WiFi restart
WiFi restart
WiFi restart
WiFi restart
WiFi restart
WiFi restart
WiFi restart
WiFi restart
WiFi restart
WiFi restart
WiFi restart
WiFi restart
WiFi restart
WiFi restart
WiFi restart
WiFi restart
WiFi restart
WiFi restart
WiFi restart
WiFi restart
WiFi restart
WiFi restart
WiFi restart
WiFi restart
WiFi restart
WiFi restart
WiFi restart
2022/04/08 22:24:07 : Reboot & Start watchdog
New discovery.
I have power-cycle my mini router. Below is from my neighbor's camera.
# ip route
192.168.50.0/24 dev wlan0 src 192.168.50.60
[root@atom-3603:~]# ping 192.168.50.1
PING 192.168.50.1 (192.168.50.1): 56 data bytes
64 bytes from 192.168.50.1: seq=0 ttl=64 time=2.241 ms
64 bytes from 192.168.50.1: seq=1 ttl=64 time=1.302 ms
^C
--- 192.168.50.1 ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max = 1.302/1.771/2.241 ms
[root@atom-3603:~]# ping 1.1.1.1
PING 1.1.1.1 (1.1.1.1): 56 data bytes
ping: sendto: Network is unreachable
So after wifi is lost, the default route is disappeared.
I understand. Maybe the following command will set the default gw.
killall -USR1 udhcpc
I will add it to watchdog.sh.
I fixed it in ver.1.2.18.
There is a camera "no PING" for 6 hours. The camera is inside my home connecting to home-router. It is not the one disconnecting often. Web, App, and SSH are all okay.
[root@atom-front:~]# ip route
default via 192.168.50.1 dev wlan0
192.168.50.0/24 dev wlan0 src 192.168.50.62
[root@atom-front:~]# ping 192.168.50.1
PING 192.168.50.1 (192.168.50.1): 56 data bytes
64 bytes from 192.168.50.1: seq=0 ttl=64 time=2.516 ms
64 bytes from 192.168.50.1: seq=1 ttl=64 time=9.331 ms
[root@atom-front:mmc]# tail -f healthcheck.log
2022/04/10 11:23:01 :
2022/04/10 11:24:11 :
2022/04/10 11:25:32 :
2022/04/10 11:26:42 :
2022/04/10 11:27:52 :
2022/04/10 11:29:02 :
2022/04/10 11:30:12 :
2022/04/10 11:31:23 :
2022/04/10 11:32:33 :
2022/04/10 11:33:43 :
[root@atom-front:mmc]# nslookup hc-ping.com
Server: 192.168.50.2
Address 1: 192.168.50.2 telephone-booth
Name: hc-ping.com
Address 1: 176.9.71.146 static.146.71.9.176.clients.your-server.de
Address 2: 178.63.26.145 static.145.26.63.178.clients.your-server.de
Address 3: 188.40.122.95 static.95.122.40.188.clients.your-server.de
Address 4: 159.69.66.229 static.229.66.69.159.clients.your-server.de
Address 5: 2a01:4f8:151:18c::2
Address 6: 2a01:4f8:231:1214::2
Address 7: 2a01:4f8:221:2d1b::2
Address 8: 2a01:4f8:141:4258::2
[root@atom-front:mmc]# traceroute hc-ping.com
traceroute to hc-ping.com (176.9.71.146), 30 hops max, 38 byte packets
1 RT-AX3000 (192.168.50.1) 0.045 ms * *
2 * * *
3 * * *
4 * * *
5 * * *
6 * * *
7 * * *
8 * * *
9 * * *
10 * * *
11 * * *
12 * * *
13 * * *
14 * * *
15 * * *
16 * * *
17 * * *
18 * * *
19 * * *
20 * * *
21 * * *
22 * * *
23 * * *
24 * * *
25 * * *
26 * * *
27 * * *
28 * * *
29 * * *
30 * * *
After reboot:
[root@atom-front:~]# nslookup hc-ping.com
Server: 8.8.8.8
Address 1: 8.8.8.8 dns.google
Name: hc-ping.com
Address 1: 159.69.66.229 static.229.66.69.159.clients.your-server.de
Address 2: 178.63.26.145 static.145.26.63.178.clients.your-server.de
Address 3: 176.9.71.146 static.146.71.9.176.clients.your-server.de
Address 4: 188.40.122.95 static.95.122.40.188.clients.your-server.de
Address 5: 2a01:4f8:221:2d1b::2
Address 6: 2a01:4f8:141:4258::2
Address 7: 2a01:4f8:231:1214::2
Address 8: 2a01:4f8:151:18c::2
[root@atom-front:~]# traceroute hc-ping.com
traceroute to hc-ping.com (159.69.66.229), 30 hops max, 38 byte packets
1 192.168.50.1 (192.168.50.1) 0.063 ms 2.386 ms 0.437 ms
2 * * *
3 10.30.54.121 (10.30.54.121) 38.599 ms 29.330 ms 4.806 ms
4 10.30.29.81 (10.30.29.81) 2.429 ms 2.783 ms 10.366 ms
5 10.28.21.29 (10.28.21.29) 2.858 ms 6.580 ms 5.833 ms
6 218.188.28.66 (218.188.28.66) 10.462 ms 21.734 ms 5.863 ms
7 * * *
8 ix-ae-15-0.tcore1.hk2-hongkong.as6453.net (116.0.67.65) 31.310 ms 10.148 ms 20.432 ms
9 if-ae-37-2.tcore2.hk2-hongkong.as6453.net (116.0.93.144) 213.272 ms 195.087 ms if-ae-7-2.thar1.hk2-hongkong.as6453.net (180.87.112.142) 204.772 ms
10 * if-ae-2-2.tcore2.hk2-hongkong.as6453.net (116.0.82.63) 196.043 ms *
11 * * *
12 if-ae-31-5.tcore1.wyn-marseille.as6453.net (80.231.217.196) 201.295 ms 226.129 ms if-ae-2-2.tcore1.svw-singapore.as6453.net (180.87.12.1) 196.406 ms
13 if-ae-2-3.tcore2.wyn-marseille.as6453.net (80.231.217.51) 235.012 ms * if-be-41-2.ecore1.emrs2-marseille.as6453.net (80.231.165.101) 195.406 ms
14 * if-be-42-2.ecore2.emrs2-marseille.as6453.net (80.231.200.17) 200.011 ms 199.832 ms
15 if-ae-50-2.tcore2.fnm-frankfurt.as6453.net (195.219.87.214) 243.121 ms 193.805 ms 209.232 ms
16 * * *
17 static.229.66.69.159.clients.your-server.de (159.69.66.229) 200.811 ms 225.562 ms 202.841 ms
[root@atom-front:mmc]# traceroute 176.9.71.146
traceroute to 176.9.71.146 (176.9.71.146), 30 hops max, 38 byte packets
1 192.168.50.1 (192.168.50.1) 1.178 ms 1.691 ms 2.247 ms
2 * * *
3 10.30.54.121 (10.30.54.121) 4.745 ms 41.478 ms 21.234 ms
4 10.30.29.81 (10.30.29.81) 14.418 ms 17.389 ms 4.517 ms
5 10.28.21.29 (10.28.21.29) 3.883 ms 46.651 ms 57.885 ms
6 218.188.28.66 (218.188.28.66) 27.847 ms 45.601 ms 52.210 ms
7 * * *
8 ix-ae-15-0.tcore1.hk2-hongkong.as6453.net (116.0.67.65) 14.809 ms 11.451 ms 9.960 ms
9 if-ae-37-2.tcore2.hk2-hongkong.as6453.net (116.0.93.144) 203.153 ms if-ae-37-8.tcore2.hk2-hongkong.as6453.net (116.0.93.138) 227.644 ms if-ae-7-2.thar1.hk2-hongkong.as6453.net (180.87.112.142) 224.822 ms
10 if-ae-2-2.tcore2.hk2-hongkong.as6453.net (116.0.82.63) 201.299 ms * *
11 * * *
12 if-ae-31-5.tcore1.wyn-marseille.as6453.net (80.231.217.196) 212.524 ms * *
13 * * *
14 * * if-be-2-2.ecore2.emrs2-marseille.as6453.net (195.219.174.1) 215.776 ms
15 * * *
16 * if-ae-59-2.tcore1.fr0-frankfurt.as6453.net (195.219.87.195) 189.935 ms *
17 static.146.71.9.176.clients.your-server.de (176.9.71.146) 212.362 ms * *
Content of atomhack.log
:
2022/04/10 00:42:09 : Reboot & Start watchdog
Sun Apr 10 01:43:38 JST 2022
total used free shared buffers cached
Mem: 74192 69940 4252 0 1116 20900
-/+ buffers/cache: 47924 26268
Swap: 131068 716 130352
Sun Apr 10 02:45:03 JST 2022
total used free shared buffers cached
Mem: 74192 70652 3540 0 1424 20408
-/+ buffers/cache: 48820 25372
Swap: 131068 740 130328
Sun Apr 10 03:46:26 JST 2022
total used free shared buffers cached
Mem: 74192 70560 3632 0 1172 20664
-/+ buffers/cache: 48724 25468
Swap: 131068 776 130292
Sun Apr 10 04:47:47 JST 2022
total used free shared buffers cached
Mem: 74192 69300 4892 0 984 20896
-/+ buffers/cache: 47420 26772
Swap: 131068 888 130180
Sun Apr 10 05:57:05 JST 2022
total used free shared buffers cached
Mem: 74192 70140 4052 0 984 21380
-/+ buffers/cache: 47776 26416
Swap: 131068 888 130180
Sun Apr 10 07:07:04 JST 2022
total used free shared buffers cached
Mem: 74192 70272 3920 0 992 21408
-/+ buffers/cache: 47872 26320
Swap: 131068 888 130180
Sun Apr 10 08:17:15 JST 2022
total used free shared buffers cached
Mem: 74192 69552 4640 0 952 21208
-/+ buffers/cache: 47392 26800
Swap: 131068 952 130116
Sun Apr 10 09:27:16 JST 2022
total used free shared buffers cached
Mem: 74192 69572 4620 0 952 21228
-/+ buffers/cache: 47392 26800
Swap: 131068 952 130116
Sun Apr 10 10:37:30 JST 2022
total used free shared buffers cached
Mem: 74192 68408 5784 0 716 20532
-/+ buffers/cache: 47160 27032
Swap: 131068 1092 129976
2022/04/10 11:43:35 : Reboot & Start watchdog
Running on the camera connecting to home-router outside my home:
[root@atom-entrance:~]# nslookup hc-ping.com
Server: 192.168.50.2
Address 1: 192.168.50.2 telephone-booth
Name: hc-ping.com
Address 1: 176.9.71.146 static.146.71.9.176.clients.your-server.de
Address 2: 159.69.66.229 static.229.66.69.159.clients.your-server.de
Address 3: 188.40.122.95 static.95.122.40.188.clients.your-server.de
Address 4: 178.63.26.145 static.145.26.63.178.clients.your-server.de
Address 5: 2a01:4f8:221:2d1b::2
Address 6: 2a01:4f8:141:4258::2
Address 7: 2a01:4f8:231:1214::2
Address 8: 2a01:4f8:151:18c::2
[root@atom-entrance:~]# traceroute hc-ping.com
traceroute to hc-ping.com (176.9.71.146), 30 hops max, 38 byte packets
1 RT-AX3000 (192.168.50.1) 0.606 ms 4.145 ms 2.292 ms
2 * * *
3 10.30.54.121 (10.30.54.121) 25.877 ms 6.854 ms 8.858 ms
4 10.30.28.81 (10.30.28.81) 3.326 ms 4.811 ms 4.192 ms
5 10.28.21.25 (10.28.21.25) 2.745 ms 3.225 ms 3.020 ms
6 218.188.28.73 (218.188.28.73) 5.123 ms 3.408 ms 5.313 ms
7 * * *
8 ix-ae-15-0.tcore1.hk2-hongkong.as6453.net (116.0.67.65) 9.710 ms 6.591 ms 14.447 ms
9 if-ae-37-2.tcore2.hk2-hongkong.as6453.net (116.0.93.144) 207.286 ms if-ae-37-6.tcore2.hk2-hongkong.as6453.net (116.0.93.136) 214.508 ms 223.943 ms
10 if-ae-32-2.tcore2.svw-singapore.as6453.net (180.87.15.80) 205.687 ms * if-ae-2-2.tcore2.hk2-hongkong.as6453.net (116.0.82.63) 199.765 ms
11 if-ae-2-2.tcore1.svw-singapore.as6453.net (180.87.12.1) 210.355 ms * *
12 * * *
13 * if-ae-49-2.tcore2.wyn-marseille.as6453.net (80.231.217.49) 200.656 ms if-ae-2-4.tcore2.wyn-marseille.as6453.net (80.231.217.53) 193.570 ms
14 * * if-ae-49-2.tcore2.wyn-marseille.as6453.net (80.231.217.49) 198.655 ms
15 if-ae-50-2.tcore2.fnm-frankfurt.as6453.net (195.219.87.214) 196.145 ms 201.011 ms 193.413 ms
16 * * *
17 static.146.71.9.176.clients.your-server.de (176.9.71.146) 198.375 ms 208.137 ms if-ae-4-2.tcore1.fr0-frankfurt.as6453.net (195.219.87.18) 200.850 ms
On watchdog.sh
, may I know what is the use of these two lines?
It seems not writing anything to the log file.
/scripts/lighttpd.sh watchdog >> /media/mmc/atomhack.log
/scripts/rtspserver.sh watchdog >> /media/mmc/atomhack.log
And why watchdog.sh
counts like a clock and run the commands, but not using crontab
?
I see the commands are running with delay.
It looks like reach the router fine, but not going beyond it.
/scripts/lighttpd.sh watchdog >> /media/mmc/atomhack.log
/scripts/rtspserver.sh watchdog >> /media/mmc/atomhack.log
These output nothing when normal. These scripts check the existence of each process.
And why watchdog.sh counts like a clock and run the commands, but not using crontab? I see the commands are running with delay.
Initially, the reasons were as follows
May I separate the PING check from watchdog, and to be done by crontab? Since I am checking by minute to make sure the camera is online. As I see from the PING log, sometimes the PING could be several minutes or more.
OK. I understand. I will separate them so that they use crontab.
Thank you very much.
I have pulled the latest main
branch and merged with my modification on CURL
command
It seems 4 of my cameras are quite stable overnight.
1 of the 4 periodically does not report PING:
[root@atom-desk:~]# tail -f /media/mmc/healthcheck.log
2022/04/11 11:08:00 : 200
2022/04/11 11:09:00 : 200
2022/04/11 11:10:00 : 200
2022/04/11 11:11:00 : 200
2022/04/11 11:12:00 : 200
2022/04/11 11:13:00 : 200
2022/04/11 11:14:00 : 200
2022/04/11 11:15:00 : 200
2022/04/11 11:16:00 : 408
2022/04/11 11:18:00 : 200
I can make the grace period into a larger value. It seems the connection problem is settled, I hope.
Thanks.
This may not be related to be a problem of this hack.
May I know what could be possible reason of this?
At 20:52:11, my Python script has just detected motion from RTSP from this camera. The saved video ended at 20:52:35. I have received the video in Telegram at 20:53. At 20:54, I have got a notification that there is no PING from this camera for 2 minutes. Checking the Atom App the camera was grey-out. No connection to the web interface or SSH. Checking the camera, the LED was flashing blue and then yellow. https://user-images.githubusercontent.com/19472561/161760258-790c0e3f-1259-45dc-9804-774b9a3fa998.mp4
I have not restarted the camera yet. Is there any log file I can find to debug?
Thanks.