sa project

[chenmingzhang]

For the SA3 system on 15/03/2021, which is equipped with two pi zeros, one 3G dongle, full on. The system lost connection to the internet at night, once around 12AM the second time around 5AM. once I went close to the system in the morning the next day, I am able to connect to the system from wifi router. however, either enx0c5b8f279a64 (3g dongle ) is missing in ifconfig or it does not assign with a ip address. Visual inspection on the system shows that the lights on the dongle is shown blink in green, or blue, not a constant blue indicating the system is connected to internet.

reboot the PI DOES NOT solve the problem.

a hard pull out of power cable and push back in solve this issue.

SA1 had exactly the same issue on the next day, which is very different from the saltmarsh system and kalgoolie system, where it is been very stable for long time.

the reason behind this 3g disconnection is very likely induced by low battery power and high power consumption (2 pis + dongle). As try is heavily used in python command, which may disrupt the triggering of RESET of RPI, induced by arduino. I decided to change the soft shutdown (shutdown -h now) to a hard_reset_system.

echo "this script first kill all python proces, reset rpi and shutdown rpi" pkill -f python sleep 10 echo -n "abc" > /dev/ttyS0 sleep 1 echo -n "RESET" > /dev/ttyS0 sleep 1 /sbin/shutdown -h now also camera pi is switched off at night.

this is now implemented in SA1, and will later added to the rest of systems.

[chenmingzhang]

sa3 piezo programmed and runing

[yuanchixiao]

sa2 piezo programmed and running, enclosure closed, desiccant added.

[yuanchixiao]

water proof test was done in enclosure sa3 and sa2.

[chenmingzhang]

for sa1 57 23 /home/pi/pyduino/bash/hard_reset_system.sh 57 12 /home/pi/pyduino/bash/reboot_pi.sh

for sa2 11 00 /home/pi/pyduino/bash/hard_reset_system.sh 11 12 /home/pi/pyduino/bash/reboot_pi.sh

for sa3 27 00 /home/pi/pyduino/bash/hard_reset_system.sh 27 04 /home/pi/pyduino/bash/reboot_pi.sh 27 12 /home/pi/pyduino/bash/reboot_pi.sh 27 15 /home/pi/pyduino/bash/reboot_pi.sh

[chenmingzhang]

on 17/03/2021, sa3 system failed to connected to 3g. once staying close to the system, I connected to sa3 via wifi. ifconfig shows the information below

`enx0c5b8f279a64: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 ether 0c:5b:8f:27:9a:64 txqueuelen 1000 (Ethernet) RX packets 3 bytes 84 (84.0 B) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 0 bytes 0 (0.0 B) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536 inet 127.0.0.1 netmask 255.0.0.0 inet6 ::1 prefixlen 128 scopeid 0x10 loop txqueuelen 1000 (Local Loopback) RX packets 11985 bytes 1765077 (1.6 MiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 11985 bytes 1765077 (1.6 MiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

usb0: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500 ether 9e:70:fd:d2:9d:40 txqueuelen 1000 (Ethernet) RX packets 0 bytes 0 (0.0 B) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 0 bytes 0 (0.0 B) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

wlan0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet 192.168.4.1 netmask 255.255.255.0 broadcast 192.168.4.255 inet6 fe80::ba27:ebff:fe78:80b0 prefixlen 64 scopeid 0x20 ether b8:27:eb:78:80:b0 txqueuelen 1000 (Ethernet) RX packets 8295 bytes 631043 (616.2 KiB) RX errors 0 dropped 4 overruns 0 frame 0 TX packets 7627 bytes 835429 (815.8 KiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
`

reboot the pc and it seems the problem does resolve the problem.

I put a hard reset to the system at 12AM and make the reboot twice per day.

[chenmingzhang]

SA1 57 23 /home/pi/pyduino/bash/hard_reset_system.sh 57 12 /2 /home/pi/pyduino/bash/reboot_pi.sh

SA2 11 00 /home/pi/pyduino/bash/hard_reset_system.sh 11 12 /2 /home/pi/pyduino/bash/reboot_pi.sh

[chenmingzhang]

need to develop a code that can check the system internet connection, if not working well will need to reboot, the reboot should not be too frequent so that local logging and checking can not be achieved.

[chenmingzhang]

Sa1 Pi failed to boot properly, resulting in arduino not receiving serial communication for the following hour. A reboot triggered by arduino solved this.

[chenmingzhang]

sa3 status:

@reboot /home/pi/pyduino/bash/send_ip_tb.sh

@reboot /home/pi/pyduino/bash/ewatering_sa3_sensor.sh 45 /12 /home/pi/script/send_ip_name_volume.sh

@reboot /home/pi/pyduino/bash/autossh_to_sftp_uqgec.sh

@reboot /home/pi/pyduino/bash/autossh_first_channel.sh @reboot /home/pi/pyduino/bash/autossh_second_channel.sh 33 /2 /home/pi/pyduino/bash/check_internet_connection_and_reboot_if_needed.sh

sudo 27 22 /home/pi/pyduino/bash/hard_reset_system.sh 27 7 /home/pi/pyduino/bash/hard_reset_system.sh 27 12 * /home/pi/pyduino/bash/hard_reset_system.sh

[chenmingzhang]

Python Instant writing on SD card is now disabled, as the system move into stable condition

[chenmingzhang]

given that the disconnection only occure at night, i got rid of the reboot at midnight in SA3

27 22 * /home/pi/pyduino/bash/hard_reset_system.sh

27 7 /home/pi/pyduino/bash/hard_reset_system.sh 27 12 /home/pi/pyduino/bash/hard_reset_system.sh

SA2 11 00 /home/pi/pyduino/bash/hard_reset_system.sh 11 12 /7 /home/pi/pyduino/bash/reboot_pi.sh

SA3 57 23 /home/pi/pyduino/bash/hard_reset_system.sh 57 12 /7 /home/pi/pyduino/bash/reboot_pi.sh

[chenmingzhang]

for sa 1

# m h  dom mon dow   command
#39 15 * * * /home/pi/script/email_content.sh
#@reboot /home/pi/script/email_content.sh
#32 00 * * * /home/pi/pyduino/bash/reboot_pi.sh
57 23 */7 * * /home/pi/pyduino/bash/hard_reset_system.sh
57 12 */14 * * /home/pi/pyduino/bash/reboot_pi.sh
#57 23 * * * /home/pi/pyduino/bash/hard_reset_system.sh
#@reboot /home/pi/script/vnc_start.sh start

# Output of the crontab jobs (including errors) is sent through
# email to the user the crontab file belongs to (unless redirected).
#
# For example, you can run a backup of all your user accounts
# at 5 a.m every week with:
# 0 5 * * 1 tar -zcf /var/backups/home.tgz /home/
#
# For more information see the manual pages of crontab(5) and cron(8)
#
# m h  dom mon dow   command
@reboot  /home/pi/pyduino/bash/send_ip_tb.sh
@reboot  /home/pi/pyduino/bash/ewatering_sa1_sensor.sh
@reboot  /home/pi/pyduino/bash/weather_sa1.sh
45 */12  * * *  /home/pi/script/send_ip_name_volume.sh
@reboot /home/pi/pyduino/bash/autossh_first_channel.sh
@reboot /home/pi/pyduino/bash/autossh_second_channel.sh
30 14  */2 * * /home/pi/pyduino/bash/check_internet_connection_and_reboot_if_needed.sh

[chenmingzhang]

deleted all the photos in sa1 site, so that good amount of memory is left

pi@sa1:~ $ df -Hh Filesystem Size Used Avail Use% Mounted on /dev/root 13G 5.5G 6.7G 46% /

pi@sa1_camera:~ $ df -Hh Filesystem Size Used Avail Use% Mounted on /dev/root 13G 5.6G 6.7G 46% /

[chenmingzhang]

it is found that the pressure transducer cable tends to be exposed near the bore. which occured on both bore 1 and bore 4. now deep trench is done to put the cable into a deeper depth

[chenmingzhang]

for sa 3 27 12 /7 /home/pi/pyduino/bash/hard_reset_system.sh 27 07 15 * /home/pi/pyduino/bash/reboot_pi.sh

normal

@reboot /home/pi/pyduino/bash/send_ip_tb.sh

@reboot /home/pi/pyduino/bash/ewatering_sa3_sensor.sh 45 /12 /home/pi/script/send_ip_name_volume.sh

@reboot /home/pi/pyduino/bash/autossh_to_sftp_uqgec.sh

@reboot /home/pi/pyduino/bash/autossh_first_channel.sh @reboot /home/pi/pyduino/bash/autossh_second_channel.sh 33 /8 /home/pi/pyduino/bash/check_internet_connection_and_reboot_if_needed.sh

[chenmingzhang]

for sa2 superuser 11 00 /14 /home/pi/pyduino/bash/hard_reset_system.sh 11 12 7,21 * /home/pi/pyduino/bash/reboot_pi.sh

@reboot /home/pi/script/vnc_start.sh start

normal user @reboot /home/pi/pyduino/bash/ewatering_sa2_sensor.sh @reboot /home/pi/pyduino/bash/autossh_first_channel.sh @reboot /home/pi/pyduino/bash/autossh_second_channel.sh 21 /12 /home/pi/pyduino/bash/check_internet_connection_and_reboot_if_needed.sh

[chenmingzhang]

sa camera connection seems to be weak.

pi@sa1:~ $ ssh pi@192.168.4.19 ssh: connect to host 192.168.4.19 port 22: No route to host pi@sa1:~ $ ssh pi@192.168.4.19 ssh: connect to host 192.168.4.19 port 22: No route to host pi@sa1:~ $ arp -a -n ? (192.168.8.1) at ba:ab:be:34:00:00 [ether] on enx0c5b8f279a64 ? (192.168.4.19) at on wlan0

[chenmingzhang]

comparing photos on June 24 and July 10 in station sa2 show no difference, indicating the station is very stable

[chenmingzhang]

a slight rotation was identified on station sa2 between 2021-08-06_0803_sa2.jpg and 2021-08-05_1603_sa2.jpg

[chenmingzhang]

photo image commands was changed to

raspistill  -o $DATE --quality 13 

and from eyes, the quality is the same as compared with the image without '--quality 13 '

we need to keep using this all the time.

also we deleted the photos in sa3_camera

pi@sa3_camera:~/camera_photo $ df -Hh
Filesystem      Size  Used Avail Use% Mounted on
/dev/root        28G  5.7G   21G  22% /
devtmpfs        181M     0  181M   0% /dev
tmpfs           186M  4.0K  186M   1% /dev/shm
tmpfs           186M  2.7M  183M   2% /run
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs           186M     0  186M   0% /sys/fs/cgroup
/dev/mmcblk0p6   68M   22M   47M  32% /boot
tmpfs            38M     0   38M   0% /run/user/1000

[chenmingzhang]

all the three telstra card are recharged with 1 year data package. it is noted that the expirty date for telstra does not roll over......

also rebooted SA1 to fix the issue that wind speed does not register.

[chenmingzhang]

SA2 lost signal since 2AM and return to normal. check the internet connection file and see that there are three events

Sat  5 Feb 12:21:02 AEST 2022 0  0  0  Online 2
Sun  6 Feb 00:21:01 AEST 2022 0  0  0  Online 2
Sun  6 Feb 12:21:01 AEST 2022 0  0  0  Online 2
Mon  7 Feb 00:21:01 AEST 2022 0  0  0  Online 2
Mon  7 Feb 12:21:01 AEST 2022 6  4  1  completely offline, start reboot
Tue  8 Feb 00:21:02 AEST 2022 0  0  0  Online 2
Tue  8 Feb 12:21:02 AEST 2022 0  0  0  Online 2
Wed  9 Feb 00:21:01 AEST 2022 0  0  0  Online 2
Wed  9 Feb 12:21:02 AEST 2022 0  0  0  Online 2
Thu 10 Feb 00:21:01 AEST 2022 0  0  0  Online 2
Thu 10 Feb 12:21:01 AEST 2022 6  4  1  completely offline, start reboot

[chenmingzhang]

increase the number of checking for internet in sa2, as the signal tends to lose at night. normal user

@reboot  /home/pi/pyduino/bash/ewatering_sa2_sensor.sh
@reboot /home/pi/pyduino/bash/autossh_first_channel.sh
@reboot /home/pi/pyduino/bash/autossh_second_channel.sh
21 */6 * * * /home/pi/pyduino/bash/check_internet_connection_and_reboot_if_needed.sh

su

11 00 */14 * * /home/pi/pyduino/bash/hard_reset_system.sh
11 12 7,21 * * /home/pi/pyduino/bash/reboot_pi.sh

[chenmingzhang]

sa2 did not upload file during apr 1 to apr 4. neither the photos, but the system says it is online.....

Fri 1 Apr 00:21:01 AEST 2022 0 0 0 Online 2 Fri 1 Apr 06:21:01 AEST 2022 0 0 0 Online 2 Fri 1 Apr 12:21:01 AEST 2022 0 0 0 Online 2 Fri 1 Apr 18:21:01 AEST 2022 0 0 0 Online 2 Sat 2 Apr 00:21:01 AEST 2022 0 0 0 Online 2 Sat 2 Apr 06:21:01 AEST 2022 0 0 0 Online 2 Sat 2 Apr 12:21:01 AEST 2022 0 0 0 Online 2 Sat 2 Apr 18:21:01 AEST 2022 0 0 0 Online 2 Sun 3 Apr 00:21:01 AEST 2022 0 0 0 Online 2 Sun 3 Apr 06:21:01 AEST 2022 0 0 0 Online 2 Sun 3 Apr 12:21:01 AEST 2022 0 0 0 Online 2 Sun 3 Apr 18:21:01 AEST 2022 0 0 0 Online 2 Mon 4 Apr 00:21:01 AEST 2022 6 4 1 completely offline, start reboot Mon 4 Apr 06:21:01 AEST 2022 0 0 0 Online 2 Mon 4 Apr 12:21:01 AEST 2022 0 0 0 Online 2 Mon 4 Apr 18:21:02 AEST 2022 0 0 0 Online 2 Tue 5 Apr 00:21:01 AEST 2022 6 4 1 completely offline, start reboot Tue 5 Apr 06:21:02 AEST 2022 0 0 0 Online 2 Tue 5 Apr 12:21:01 AEST 2022 0 0 0 Online 2 Tue 5 Apr 18:21:01 AEST 2022 0 0 0 Online 2 Wed 6 Apr 00:21:02 AEST 2022 0 0 0 Online 2 Wed 6 Apr 06:21:01 AEST 2022 6 4 1 completely offline, start reboot

also the data seems to be succesfully uploaded as well.

camera keeps on
dht22,54,power,2,points,2,dummies,1,interval_mm,1000,19.45,40.80,
raw System Voltage: 744.0
SDI-12,62,default_cmd,read,power,22,power_off,1,no_sensors,1,Addr,0_MET,points,3,3026.20,19.1,924,
SDI-12,63,default_cmd,read,power,23,power_off,1,no_sensors,1,Addr,0_MET,points,3,2993.39,20.8,1491,
SDI-12,64,default_cmd,read,power,24,power_off,1,no_sensors,1,Addr,0_MET,points,3,2975.13,21.2,1732,
SDI-12,65,default_cmd,read,power,25,power_off,1,no_sensors,1,Addr,0_MET,points,3,2892.53,21.4,1755,
SDI-12,66,default_cmd,read,power,26,power_off,1,no_sensors,1,Addr,0_MET,points,3,2862.06,21.5,1893,
SDI-12,51,default_cmd,read,power,-1,power_off,1,no_sensors,1,Addr,0_Cam,points,2,.6987184,18.99301,
SDI-12,52,default_cmd,read,power,-1,power_off,1,no_sensors,1,Addr,0_Cam,points,2,.4182207,17.51250,
SDI-12,53,custom_cmd,aM!,power,-1,power_off,1,response,a0013,
SDI-12,53,custom_cmd,aD0!,power,-1,power_off,1,response,a+128.6639+19.48766+18946.07,
9548,0,power,-1,points,3,dummies,3,interval_mm,1000,type,5803,1014.90,1014.80,1014.70,18.99,1014.80,
9548,1,power,-1,points,3,dummies,3,interval_mm,1000,type,si1145,Vis,594.67,IR,3523.33,UV,181.33,
data successfully uploaded
sleep for 1200 seconds
2022-04-04 08:58:09
camera keeps on
dht22,54,power,2,points,2,dummies,1,interval_mm,1000,21.05,39.20,
raw System Voltage: 740.8
SDI-12,62,default_cmd,read,power,22,power_off,1,no_sensors,1,Addr,0_MET,points,3,3026.62,19.0,925,
SDI-12,63,default_cmd,read,power,23,power_off,1,no_sensors,1,Addr,0_MET,points,3,2993.43,20.8,1494,
SDI-12,64,default_cmd,read,power,24,power_off,1,no_sensors,1,Addr,0_MET,points,3,2977.21,21.2,1735,
SDI-12,65,default_cmd,read,power,25,power_off,1,no_sensors,1,Addr,0_MET,points,3,2893.90,21.4,1762,
SDI-12,66,default_cmd,read,power,26,power_off,1,no_sensors,1,Addr,0_MET,points,3,2862.41,21.5,1894,
SDI-12,51,default_cmd,read,power,-1,power_off,1,no_sensors,1,Addr,0_Cam,points,2,.6987041,19.01166,
SDI-12,52,default_cmd,read,power,-1,power_off,1,no_sensors,1,Addr,0_Cam,points,2,.4178442,17.63324,
SDI-12,53,custom_cmd,aM!,power,-1,power_off,1,response,a0013,
SDI-12,53,custom_cmd,aD0!,power,-1,power_off,1,response,a+128.6785+19.48688+18948.78,
9548,0,power,-1,points,3,dummies,3,interval_mm,1000,type,5803,1015.20,1015.50,1015.50,19.03,1015.40,
9548,1,power,-1,points,3,dummies,3,interval_mm,1000,type,si1145,Vis,690.33,IR,4366.33,UV,232.00,
data successfully uploaded
sleep for 1200 seconds

[chenmingzhang]

SA2 lost connection for 1 week,

• the internet connection was weak before lost of connection
• the humidity in the enclosure seems to be normal
• use # iw dev wlan0 scan | grep SSID from the camera on sa2, only found sa1 and sa3.
• likely the rpi is completely out of function right now.
• water level used to be higher, before the failure of the system.

this means the pi2 is not working at all, at the moment.

[chenmingzhang]

a field site inspection suggests that:

1. when opening the enclosure, it is very dry internally.
2. pi is switched off, arduino is on, but failed to reboot pi on the regular basis (8 hours), very likely due to the failure of 9543 measurements, which causes the freezing of the arduino.
3. now need to try if reset 9543 could make the system back to work.

solution:

1. add reset 9543 in the code.
2. installed an extra antenna on the system.
3. add a extra ubolt on the tripod.
4. tighten all the screws
5. camera cleaning.
6. shutdown pi at 11:12 and see if it can be revived by itsself tomorrow.

[chenmingzhang]

update to the fix:

1. Sa3 does not have the right sockets to insert an antenna, so we did not add it in. the existing socket is only for the spiral sockets, which the antenna can not get through. next time, bring antenna where it is already passing through the socket.
2. Sa1 also does not have the space to insert antenna. antes gets into the enclosure by biting the corner of the foam enclosure. the best way is also to bring an extra antenna that have already pass through the socket.
3. Sa2 the solar intensity sensor is down. the socket has four holes, may need to install another one.
4. The right size of the u bolt is about 25. Need to check and install another one for SA3.
5. need to bring screen to see what happens for the camera for sa1.

At sa1, ants gets into the enclosure, by biting some of the foam attached on the side.

Buried all the exposed cable

Sa2 added one u bolts. Tightened all the screws.