Problem With The Backup: collecting data to Node

[mz27ethio]

Greetings all, As you can see in the attached images, I have encountered problem related with issue number #105 #127 and #149 . The problem started after I updated the node. Description of the problem:

• on the main page for the node web UI, "time since backup" is shown as N/A (Picture shown)
• Also, after the start of tracking, it shows that the file will be saved as /None (Picture shown)
• Automatic backup is not working.

The version of the node:

As it was not allowing automatic backup, I tried forcing it as shown here and this was the result:

`cd /opt/ethoscope-node/node_src/scripts

sudo python backup_tool.py -e 1

Forcing backup for ethoscope 001 INFO:root:Initiating backup for device 001d54d8d1c94e43979d5805c9d995a7 INFO:root:Running backup for device 001d54d8d1c94e43979d5805c9d995a7 INFO:mysql.connector.authentication:package: mysql.connector.plugins INFO:mysql.connector.authentication:plugin_name: mysql_native_password INFO:mysql.connector.authentication:AUTHENTICATION_PLUGIN_CLASS: MySQLNativePasswordAuthPlugin INFO:root:Initializing local database static tables at /ethoscope_data/results/None INFO:root:Making parent directories WARNING:root:[Errno 17] File exists: '/ethoscope_data/results' INFO:root:Ensuring DAM file exists at /ethoscope_data/results/None.txt INFO:root:Database mirroring initialised ERROR:root:Cannot mirror the 'SENSORS' table ERROR:root:1146 (42S02): Table 'ETHOSCOPE_001_db.SENSORS' doesn't exist INFO:root:Backup status for 001d54d8d1c94e43979d5805c9d995a7 is 100.00% INFO:root:Backup done for for device 001d54d8d1c94e43979d5805c9d995a7 `

Then when I checked the /ethoscope_data/results directory, it has saved the data as NONE.db and NONE.txt

Also, I tried checking the status of the ethoscope_backup.service in systemctl and this was the result:

`● ethoscope_backup.service - Ethoscope node server Loaded: loaded (/usr/lib/systemd/system/ethoscope_backup.service; enabled; preset: disabled) Active: active (running) since Wed 2023-02-08 18:10:57 CST; 5min ago Main PID: 429 (python) Tasks: 1 (limit: 18983) Memory: 35.3M CPU: 231ms CGroup: /system.slice/ethoscope_backup.service └─429 /usr/bin/python /opt/ethoscope-node/node_src/scripts/backup_tool.py

Feb 08 18:11:03 node python[429]: urllib.error.URLError: <urlopen error [Errno 111] Connection refused> Feb 08 18:11:03 node python[429]: During handling of the above exception, another exception occurred: Feb 08 18:11:03 node python[429]: Traceback (most recent call last): Feb 08 18:11:03 node python[429]: File "/usr/lib/python3.10/threading.py", line 1016, in _bootstrap_inner Feb 08 18:11:03 node python[429]: self.run() Feb 08 18:11:03 node python[429]: File "/opt/ethoscope-node/node_src/ethoscope_node/utils/backups_helpers.py", line 148, in run Feb 08 18:11:03 node python[429]: active_devices = self.find_devices() Feb 08 18:11:03 node python[429]: File "/opt/ethoscope-node/node_src/ethoscope_node/utils/backups_helpers.py", line 89, in find_devices Feb 08 18:11:03 node python[429]: logging.error("The node ethoscope server %s is not running or cannot be reached. A list of available ethoscopes could not be found." % server) Feb 08 18:11:03 node python[429]: NameError: name 'server' is not defined`

As I am a beginner, I wanted to ask if you guys can help me on how to resolve this problem. I have several experiments lined up, so urgent help is much appreciated.

Thanks

[ggilestro]

Hi Michael - this should all be solved with early commits. Are you running the latest version of the software on the ethoscope too?

[mz27ethio]

Hello Giorgio,

Thanks for your response. I tried updating the Ethoscope through the web interface (http://node:8888)

However, it is not working. The Ethoscope version is the same before and after the update.

So, I opted for downloading latest version of the Ethoscope (the January 30, 2023 release) from the Notion Page and tried using Balena Etcher to burn it.

However, I encountered two problems:

1. When I loaded the downloaded file (img.tar.gz) to the Balena Etcher software, it throws the following warning message:

2. After the burning process is completed, I inserted the SD into the Ethoscope, however, the Ethoscope is not coming online.

Is the latest img viable or is it a problem on my side? If so, please let me know how I can go ahead and update the Ethoscope.

Thanks

[ggilestro]

I see. The previous image had a little bug that interfered with update. The current one has had an upload issue perhaps. I'll try to fix this with a new image in the next few days or else you can try burn the previous one again an I'll explain how to do a manual update. Sorry for the inconvenience

On Fri, 10 Feb 2023, 00:38 Michael Ewnetu, @.***> wrote:

Hello Giorgio,

Thanks for your response. I tried updating the Ethoscope through the web interface (http://node:8888)

[image: photo_5992234085734530854_y] https://user-images.githubusercontent.com/91686165/217938664-406f3eb9-5170-4f47-8aab-7de9a385176c.jpg

However, it is not working. The Ethoscope version is the same before and after the update.

[image: photo_5992234085734530855_y] https://user-images.githubusercontent.com/91686165/217938749-500c7a39-9308-49e3-b459-c9bda3322423.jpg

So, I opted for downloading latest version of the Ethoscope (the January 30, 2023 release) from the Notion Page https://www.notion.so/The-ethoscope-60952be38787404095aa99be37c42a27 and tried using Balena Etcher to burn it.

However, I encountered two problems:

1. When I loaded the downloaded file (img.tar.gz) to the Balena Etcher software, it throws the following warning message:

[image: photo_5994485885548215082_y] https://user-images.githubusercontent.com/91686165/217970248-2ec2909d-1eb2-44fc-abb5-12832d951932.jpg

1. After the burning process is completed, I inserted the SD into the Ethoscope, however, the Ethoscope is not coming online.

Is the latest img viable or is it a problem on my side? If so, please let me know how I can go ahead and update the Ethoscope.

Thanks

— Reply to this email directly, view it on GitHub https://github.com/gilestrolab/ethoscope/issues/178#issuecomment-1425020614, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAFKERPXI4GSXRUBR6HC4QLWWWEYLANCNFSM6AAAAAAUV5GDLU . You are receiving this because you commented.Message ID: @.***>

[rjs360]

Dear Giorgio and team, after a long caveat I have taken up working with the ethoscope platform again, and decided to update both my node and my ethoscopes to the latest versions (via the Notion site as @mz27ethio indicated).

I did this today, so these are what I assume are the most up-to-date images.

I now have what looks like an almost identical issue. For me, automatic video backup is inactive, and when I attempt to move the slider in the GUI's management settings, it reverts to inactive again after a while. At any rate, no video backups are generated during a tracking bout, nor during a video recording. My .txt and .db recording files are also labeled as "none.txt" and "none.db", but the "none.txt" file does seem to contain count data. I can provide more detailed information tomorrow when I am back in the lab, if needed.

Did this issue have a solution in the end?

Thanks.

[ggilestro]

Sorry, just to make sure we are on the same page: what is it that you are trying to do? Backing up videos or backing up tracking data?

ethoscope_backup is the service that is copying tracking data (db format) ethoscope_video_backup is the service copying video data, generated only when you use the "Video -> Record Video" button.

No video are generated by tracking.

[rjs360]

Dear Giorgio, I tried doing both. Both seem to have a problem. I have attached below some images that may clarify the situation. When I connect to my ethoscopes this is the default "settings" state in the Management section:

I can move the slider for the video backup to activate it, but after a while or a refresh of the page, it will revert to inactive. I am also getting related error messages in the log (note that this is not the complete log):

I just tried again to record a video (no tracking), and while the interface indicated that a video was recording (I even got a live video window this time), and would be saved to the node, it did not record any video to the node (i.e. the video folder is empty).

I also tried a tracking session, for which the interface also indicated that a file would be saved to the node (with the typical, long filename), but it did not actually save any tracking data to the node once I stopped the tracking.

It does not seem to be a browser issue, as this happens the same way in Firefox and Midori (the default browser in my manjaro linus distro).

Does this help diagnose the problem? -Ruud

[rjs360]

A small update on this issue:

I attempted to add the ethoscope_backup_service by running the set of commands indicated by @antortjim in issue #146:

"cp /opt/ethoscope-node/scripts/ethoscope_video_backup.service /etc/systemd/system systemctl daemon-reload # refresh database of services systemctl start ethoscope_video_backup # start the service systemctl status ethoscope_video_backup # see if the service works"

but it seems this is not working:

I then tried to at least confirm that my devices (node and ethoscopes) are updated properly; here's an image indicating the status:

I am not sure what "software broken" means, but this can't be good? Any suggestions would be most appreciated. -Ruud

[ggilestro]

Software broken indicates some problem with the update procedure that is based on git. The update server should show you something like the screenshot below. Not sure what is wrong there.

For the backup, I will need to see more logs to understand what is going on. Running the following should give you some logs: journalctl -u ethoscope_backup -rb

[antortjim]

As a workaround for now, you can download your videos from the ethoscope with scp or rsync. With scp the command line is quite simple, this should work:

# from a terminal in the node
scp -ru alarm@ethoscope_ip:/ethoscope_data/results/ /folder/in/node/where/you/want/your/videos

You will get a bunch of .h264 files under /folder/in/node/where/you/want/your/videos/long_id/ETHOSCOPE_XXX/date_time which you can compile together into a single file with the script @ggilestro provides here

https://github.com/gilestrolab/ethoscope/blob/master/scripts/tools/ethoscope_videos2mp4.sh

which you run I believe like so

# requires ffmpeg installed
./ethoscope_videos2mp4.sh /folder/in/node/where/you/want/your/videos/long_id/ETHOSCOPE_XXX/date_time

Good luck! A

[rjs360]

Hi Giorgio & Antonio, Thanks for the support! I had another look at the update server today and I am not getting the Software Broken message anymore; now it is showing this (while I am connected to my local ethoscope_wifi network):

I tried running another update while on my regular wifi (with internet), but that still doesn't work. I get a large error log that I have pasted below. I also ran the command that Giorgio suggested, and that output is listed at the end of this message:

Output by the GUI when attempting to update the node and ethoscopes via http://node:8888/:

"Traceback (most recent call last): File "/usr/lib/python3.10/urllib/request.py", line 1348, in do_open h.request(req.get_method(), req.selector, req.data, headers, File "/usr/lib/python3.10/http/client.py", line 1282, in request self._send_request(method, url, body, headers, encode_chunked) File "/usr/lib/python3.10/http/client.py", line 1328, in _send_request self.endheaders(body, encode_chunked=encode_chunked) File "/usr/lib/python3.10/http/client.py", line 1277, in endheaders self._send_output(message_body, encode_chunked=encode_chunked) File "/usr/lib/python3.10/http/client.py", line 1037, in _send_output self.send(msg) File "/usr/lib/python3.10/http/client.py", line 975, in send self.connect() File "/usr/lib/python3.10/http/client.py", line 941, in connect self.sock = self._create_connection( File "/usr/lib/python3.10/socket.py", line 845, in create_connection raise err File "/usr/lib/python3.10/socket.py", line 833, in create_connection sock.connect(sa) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/opt/ethoscope_updater/update_server.py", line 178, in group response = updates_api_wrapper(device['ip'], device['id'], what='device/update') File "/opt/ethoscope-node/scripts/ethoscope_updater/helpers.py", line 300, in updates_api_wrapper raise e File "/opt/ethoscope-node/scripts/ethoscope_updater/helpers.py", line 286, in updates_api_wrapper f = urllib.request.urlopen(req) File "/usr/lib/python3.10/urllib/request.py", line 216, in urlopen return opener.open(url, data, timeout) File "/usr/lib/python3.10/urllib/request.py", line 519, in open response = self._open(req, data) File "/usr/lib/python3.10/urllib/request.py", line 536, in _open result = self._call_chain(self.handle_open, protocol, protocol + File "/usr/lib/python3.10/urllib/request.py", line 496, in _call_chain result = func(*args) File "/usr/lib/python3.10/urllib/request.py", line 1377, in http_open return self.do_open(http.client.HTTPConnection, req) File "/usr/lib/python3.10/urllib/request.py", line 1351, in do_open raise URLError(err) urllib.error.URLError: <urlopen error [Errno 113] No route to host>"

And here is the output of: [rs@node ~]$ journalctl -u ethoscope_backup -rb

May 11 13:42:37 node python[506]: NameError: name 'server' is not defined May 11 13:42:37 node python[506]: logging.error("The node ethoscope server > May 11 13:42:37 node python[506]: File "/opt/ethoscope-node/node_src/ethoscop> May 11 13:42:37 node python[506]: active_devices = self.find_devices() May 11 13:42:37 node python[506]: File "/opt/ethoscope-node/node_src/ethoscop> May 11 13:42:37 node python[506]: self.run() May 11 13:42:37 node python[506]: File "/usr/lib/python3.10/threading.py", li> May 11 13:42:37 node python[506]: Traceback (most recent call last): May 11 13:42:37 node python[506]: During handling of the above exception, anoth> May 11 13:42:37 node python[506]: urllib.error.URLError: <urlopen error [Errno > May 11 13:42:37 node python[506]: raise URLError(err) May 11 13:42:37 node python[506]: File "/usr/lib/python3.10/urllib/request.py> May 11 13:42:37 node python[506]: return self.do_open(http.client.HTTPConne> May 11 13:42:37 node python[506]: File "/usr/lib/python3.10/urllib/request.py> May 11 13:42:37 node python[506]: result = func(*args) May 11 13:42:37 node python[506]: File "/usr/lib/python3.10/urllib/request.py> May 11 13:42:37 node python[506]: result = self._call_chain(self.handle_ope> May 11 13:42:37 node python[506]: File "/usr/lib/python3.10/urllib/request.py> May 11 13:42:37 node python[506]: response = self._open(req, data) May 11 13:42:37 node python[506]: File "/usr/lib/python3.10/urllib/request.py> May 11 13:42:37 node python[506]: return opener.open(url, data, timeout) May 11 13:42:37 node python[506]: File "/usr/lib/python3.10/urllib/request.py> May 11 13:42:37 node python[506]: f = urllib.request.urlopen(req, timeout=t> May 11 13:42:37 node python[506]: File "/opt/ethoscope-node/node_src/ethoscop> May 11 13:42:37 node python[506]: Traceback (most recent call last): May 11 13:42:37 node python[506]: During handling of the above exception, anoth> May 11 13:42:37 node python[506]: ConnectionRefusedError: [Errno 111] Connectio> May 11 13:42:37 node python[506]: sock.connect(sa) May 11 13:42:37 node python[506]: File "/usr/lib/python3.10/socket.py", line > May 11 13:42:37 node python[506]: raise err May 11 13:42:37 node python[506]: File "/usr/lib/python3.10/socket.py", line > May 11 13:42:37 node python[506]: self.sock = self._create_connection( May 11 13:42:37 node python[506]: File "/usr/lib/python3.10/http/client.py", > May 11 13:42:37 node python[506]: self.connect() May 11 13:42:37 node python[506]: File "/usr/lib/python3.10/http/client.py", > May 11 13:42:37 node python[506]: self.send(msg) May 11 13:42:37 node python[506]: File "/usr/lib/python3.10/http/client.py", > May 11 13:42:37 node python[506]: self._send_output(message_body, encode_ch> May 11 13:42:37 node python[506]: File "/usr/lib/python3.10/http/client.py", > May 11 13:42:37 node python[506]: self.endheaders(body, encode_chunked=enco> May 11 13:42:37 node python[506]: File "/usr/lib/python3.10/http/client.py", > May 11 13:42:37 node python[506]: self._send_request(method, url, body, hea> May 11 13:42:37 node python[506]: File "/usr/lib/python3.10/http/client.py", > May 11 13:42:37 node python[506]: h.request(req.get_method(), req.selector,> May 11 13:42:37 node python[506]: File "/usr/lib/python3.10/urllib/request.py> May 11 13:42:37 node python[506]: Traceback (most recent call last): May 11 13:42:37 node python[506]: Exception in thread Thread-1: May 11 13:42:37 node python[506]: Hit Ctrl-C to quit. May 11 13:42:37 node python[506]: Listening on http://0.0.0.0:82/ May 11 13:42:37 node python[506]: Bottle v0.12.23 server starting up (using WSG> May 11 13:42:37 node python[506]: INFO:root:Starting backup round May 11 13:42:36 node systemd[1]: Started Ethoscope node server.

Hope that helps narrow it down a bit. -Ruud

[antortjim]

1. Make sure the Python code you are running in the node is up to date (run git pull in the folder in the pc that contains the node code)
2. Your problem is with ethoscope_backup or ethoscope_video_backup? I.e. are you having trouble getting dbfiles or the raw videos? You mentioned first ethoscope_video_backup but now show the output for ethoscope_backup. Does either of them work? If none of them work, you may have a problem with the WiFi network configurations.

[rjs360]

Dear Antonio,

Re 1. I will attempt this soon when I have access to my node, and report back.

Re 2. At this time, I am not receiving any .db files nor videos on the node after either tracking attempts, or video recordings. I tried both processes several times, and thought there was something wrong with either the node version or ethoscope version (or both), which is why I am attempting to update them (unsuccessfully). Giorgio asked me to run the "journalctl -u ethoscope_backup -rb" command for troubleshooting purposes, so I submitted the output of that.

I'll let you know what happens when I perform the git pull. -Ruud

[rjs360]

ok, apologies if this is the wrong folder to run the "git pull" command in but this is the output for running "sudo git pull" in: /opt/ethoscope-node/

"error: unable to normalize alternate object path: /tmp/ethoscope-node/ethoscope-node/objects"

It lists the above line 3 times and finishes with: "Already up to date"

[rjs360]

Does anyone have any other suggestions/solutions? Or do I need to provide more data?

Could I perhaps try some older combination of node/ethoscope images, to see if the latest upgrade caused the issue for me? I can only download the latest images at the moment.

Thanks for any help -Ruud

[antortjim]

Dear @rjs360 Sorry for the late reply.

I am not sure what this means

"error: unable to normalize alternate object path: /tmp/ethoscope-node/ethoscope-node/objects"

but I think it's not important, since you are indeed getting finally the right message (already up to date). The best wayt to make sure you have the updated code is running git rev-parse HEAD and checking the output matches the hash code of the last commit in https://github.com/gilestrolab/ethoscope (currently 666c6dae12d88d534fe3367c9ada965f2082b79c).

Please upload the logfiles created with these commands:

In the node

journalctl -u ethoscope_backup  --since "2 week ago" > db_backup.log
journalctl -u ethoscope_video_backup  --since "2 week ago" > video_backup.log

In the ethoscope, put this in a text file and run it. Let me know if you don't know how to do it

#! /bin/bash
journalctl -u ethoscope_device  --since "2 week ago" > db_backup.log
mysql -uethoscope -pethoscope -e "USE ETHOSCOPE_XXX_db; SHOW TABLES;" > mysql.log

for ROI in {1..20}; do
    echo ROI_${ROI} >> mysql.log
    mysql -uethoscope -pethoscope -e "USE ETHOSCOPE_XXX_db; SELECT COUNT(*) FROM ROI_${ROI};" >> mysql.log
done

where XXX is the 3 digit code of yoru ethoscope number :)

This info will help us debug your problem, with the information we have I think it's very hard.

Cheers, A

[antortjim]

I can be more helpful if you allow me to access your system remotely, feel free to share the credentials if you want in a PM to antonio.ortega@kuleuven.be. Easiest way is via a remote desktop application like TeamViewer or AnyDesk

[pepelisu]

Hi all,

Some things that I see in the already sent logs:

1. File backup_helpers.py has en error on line 89 in branch "dev", the error is due to an undefined "server" variable, (inherited from old code). Reading that file, one can see that the original error is coming from a URLlib URL error, the error comes when the node tries to find the connected devices, to do that in the dev branch, it uses a JSON file hosted in the node, publicly available at this url: http://localhost/devices or http://NODEIP/devices. You could try to access this URL manually from the node to see if you can see the registered devices. According to your logs you would get an error. Probably a 404 or server error.

2. All this seems to be a connection problem, either way the web server is not starting in the node or the network is somehow not accesible. Could you tell us if the node has two separate ethernet connections or wifi + ethernet?

3. In another of your previous comments you mention that ethoscope_video_backup is not working (since in the message it says that is restarting all the time) could you perform this two commands: sudo systemctl start ethoscope_video_backup and then: sudo systemctl status ethoscope_video_backup

[rjs360]

Thank you Antonio! I will try to provide the information you are looking for, but I am not sure exactly how to get into the ethoscope to run a bash script. Do I need to attach the ethoscope to a monitor/keyboard etc? Is that the easiest way?

-Ruud

On Wed, May 24, 2023 at 3:13 PM Antonio @.***> wrote:

Dear @rjs360 https://github.com/rjs360 Sorry for the late reply.

I am not sure what this means

"error: unable to normalize alternate object path: /tmp/ethoscope-node/ethoscope-node/objects"

but I think it's not important, since you are indeed getting finally the right message (already up to date). The best wayt to make sure you have the updated code is running git rev-parse HEAD and checking the output matches the hash code of the last commit in https://github.com/gilestrolab/ethoscope (currently 666c6dae12d88d534fe3367c9ada965f2082b79c).

Please upload the logfiles created with these commands:

In the node

journalctl -u ethoscope_backup --since "2 week ago" > db_backup.log journalctl -u ethoscope_video_backup --since "2 week ago" > video_backup.log

In the ethoscope, put this in a text file and run it. Let me know if you don't know how to do it

! /bin/bash

journalctl -u ethoscope_device --since "2 week ago" > db_backup.log mysql -uethoscope -pethoscope -e "USE ETHOSCOPE_XXX_db; SHOW TABLES;" > mysql.log

for ROI in {1..20}; do echo ROI_${ROI} >> mysql.log mysql -uethoscope -pethoscope -e "USE ETHOSCOPE_XXXdb; SELECT COUNT(*) FROM ROI${ROI};" >> mysql.log done

where XXX is the 3 digit code of yoru ethoscope number :)

This info will help us debug your problem, with the information we have I think it's very hard.

Cheers, A

— Reply to this email directly, view it on GitHub https://github.com/gilestrolab/ethoscope/issues/178#issuecomment-1561796603, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABSQSRR7OJAFXUIS5PH6QD3XHZMVXANCNFSM6AAAAAAUV5GDLU . You are receiving this because you were mentioned.Message ID: @.***>

[ggilestro]

This error

"error: unable to normalize alternate object path: /tmp/ethoscope-node/ethoscope-node/objects"

Seems to be linked to an unsuccessful git installation. One thing you could try is to reinstall the node package. This should be enough, on a node terminal:

sudo pacman -R ethoscope-node
yay -S ethoscope-node

[antortjim]

@rjs360 yes, that's gonna be the easiest. You can use a terminal text editor like nano

nano script.sh
# this opens a terminal 'window' where you can type stuff
# then press Control+o, confirm the name script.sh by pressing enter and then press Control+x to exit

then you can run it like so

bash script.sh

Good luck

[rjs360]

Dear Antonio, I have produced the .log files you requested (and learned something about moving files back and forth between devices while doing so :)). See attached files. I should note that I re-imaged my node today, after I was unable to reinstall the node package (see Giorgio's instructions above) with yay -S, post removal of the package with pacman -R. So, perhaps the logs are a bit thin, since I attempted only one tracking instance for a few seconds. But I am hoping you might still be able to use them to diagnose.

Thanks and please let me know how I can help you determine what is going on exactly. -Ruud

db_backup.log video_backup.log Ethoscope_db_backup.log Ethoscope_mysql.log

[pepelisu]

Dear Ruud, The error seems to be that the update of the node failed somehow and some files are not where they should be, otherwise I am not sure why the logs are showing import errors. Could you try to reinstall the ethoscope-node package as suggested by ggilestro? That is the easiest way to recreate the git repository in the node.

[rjs360]

Is it normal for the node to NOT have yay installed? I cannot find it with whereis or which yay.

-Ruud

On Thu, May 25, 2023 at 16:46 Pepelisu @.***> wrote:

Dear Ruud, The error seems to be that the update of the node failed somehow and some files are not where they should be, otherwise I am not sure why the logs are showing import errors. Could you try to reinstall the ethoscope-node package as suggested by ggilestro?

— Reply to this email directly, view it on GitHub https://github.com/gilestrolab/ethoscope/issues/178#issuecomment-1563487009, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABSQSRX4WVEGOCRMNKFUEILXH7AKDANCNFSM6AAAAAAUV5GDLU . You are receiving this because you were mentioned.Message ID: @.***>

[rjs360]

Small update: Instead of "yay" (which does not seem to be installed by default) I used "pacman -S ethoscope-node" to reinstall the package, after first performing a "pacman -R ethoscope-node" removal.

After this some progress I think: I can now initiate a tracking event, and manually (i.e. via the browser interface) force a backup to the node ethoscope-data folder successfully.

The automated backup feature in the browser interface still is inactive, and once activated, it will become inactive again when the browser is refreshed. I am going to test if it may just take a very long time before the automated backup of tracking data is performed, but not holding my breath, as in older versions of the system the auto-backup was very fast after the tracking events were ended.

Do you have any more ideas of what may be going on? Are the any additional logs I should produce at this point? Thanks for the continued support! -Ruud

[pepelisu]

To debug what is happening now, after the reinstall. We would need at least the output of: (in the node) sudo systemctl status ethoscope_backup sudo systemctl status ethoscope_video_backup

and maybe, for more info: journalctl -u ethoscope_backup -rb

[rjs360]

Dear pepelisu, I just ran these three lines in my home folder and got the following responses, respectively:

sudo systemctl status ethoscope_backup: Unit ethoscope-backup.service could not be found

sudo systemctl status ethoscope_video_backup: Unit ethoscope-video-backup.service could not be found

journalctl -u ethoscope_backup -rb: -- No entries --

I looked in the /opt/ethoscope-node/scripts folder and did find the .service files, so perhaps there is something in the way these files are referenced that is wrong?

[ggilestro]

Make sure you use underscores. it's ethoscope_backup not ethoscope-backup

[rjs360]

Apologies, should not have copy/pasted; see attached for the requested outputs of the three commands. -Ruud

On Fri, May 26, 2023 at 10:21 AM Giorgio Gilestro @.***> wrote:

Make sure you use underscores. it's ethoscope_backup not ethoscope-backup

— Reply to this email directly, view it on GitHub https://github.com/gilestrolab/ethoscope/issues/178#issuecomment-1564471419, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABSQSRWYMP4LK3L73Q5IWGTXIC35FANCNFSM6AAAAAAUV5GDLU . You are receiving this because you were mentioned.Message ID: @.***>

[rjs360]

seems the attachments didn't come through. another attempt then: status_ethoscope_backup.txt status_ethoscope_video_backup.txt journalctl_ethoscope_backup.txt

[pepelisu]

Some things to try:

1. access http://localhost/devices, is there a json list of available devices? if no go to 2.
2. Update node. if successful try again to start the ethoscope_backup and check again http://localhost/devices.

The error that your logs are showing is solved in the last version of ethoscope node (branch dev)

[rjs360]

Hi pepelisu, this is what's listed at http://localhost/devices:

{"001c7cf3397741449c529aff01a294fb": {"status": "stopped", "ping": 2246, "ip": "192.168.0.100", "time": 1685062269.7118032, "error": null, "log_file": "/ethoscope_data/results/ethoscope.log", "dbg_img": "/ethoscope_data/results/dbg_img.png", "last_drawn_img": "/tmp/ethoscope_ba10yz9r/last_img.jpg", "db_name": "ETHOSCOPE_001_db", "monitor_info": {"last_positions": null, "last_time_stamp": 0, "fps": 0}, "experimental_info": {"run_id": "8b4184203893417b"}, "id": "001c7cf3397741449c529aff01a294fb", "name": "ETHOSCOPE_001", "version": {"id": "3312673383686b246be8885c5913c3a69ecc6049", "date": "2023-02-06 11:46:58"}, "backup_filename": "2023-05-26_00-49-14_001c7cf3397741449c529aff01a294fb.db", "CPU_temp": 41.3, "underpowered": false, "current_timestamp": 1685144781.726389, "backup_path": "/ethoscope_data/results/001c7cf3397741449c529aff01a294fb/ETHOSCOPE_001/2023-05-26_00-13-49/2023-05-26_00-13-49_001c7cf3397741449c529aff01a294fb.db", "backup_status": 100.0, "time_since_backup": 76557.93261098862, "backup_size": 147456, "previous_date_time": 1685060029.0185347, "previous_backup_filename": "2023-05-26_00-13-49_001c7cf3397741449c529aff01a294fb.db", "previous_user": "rschilder", "previous_location": "incubator"}}

[pepelisu]

If the http://localhost/devices is working and giving you that output, then I don't have a clear explanation of what is happening. It seems that the backup_helpers.py is not getting the right node_address, by default is "localhost" and it works when you do it manually. So either you have an extra parameter in the ethoscope_backup.service, for example -i or --server, then it can change the default. Other option is that your urllib version is throwing an error when calling lines 84, 85:

 req = urllib.request.Request(url, headers={'Content-Type': 'application/json'})            
 f = urllib.request.urlopen(req, timeout=timeout)

[ggilestro]

I was able to reproduce the issue on a new installation. The commit above should fix it. Please make sure you run this latest version of the node and the backup should work again.

[rjs360]

Dear Giorgio,

I am running the latest 06.07.23 version of the node now, but I am still not seeing any tracking data backed up to the node. I didn't try the manual backup procedure yet. What can I log to help diagnose? -Ruud

On Wed, Jun 7, 2023 at 4:19 PM Giorgio Gilestro @.***> wrote:

I was able to reproduce the issue on a new installation. The commit above should fix it. Please make sure you run this latest version of the node and the backup should work again.

— Reply to this email directly, view it on GitHub https://github.com/gilestrolab/ethoscope/issues/178#issuecomment-1581452721, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABSQSRXEEQWGFQNTNWHGNNDXKDO4JANCNFSM6AAAAAAUV5GDLU . You are receiving this because you were mentioned.Message ID: @.***>

[ggilestro]

The bug was in the backup service. Just make sure you restart the backup service after yesterday's upgrade.

[rjs360]

Thanks Giorgio,

"systemctl restart ethoscope_backup.service"

did the trick. Thanks all for the support! -Ruud

On Thu, Jun 8, 2023 at 2:40 AM Giorgio Gilestro @.***> wrote:

The bug was in the backup service. Just make sure you restart the backup service after yesterday's upgrade.

— Reply to this email directly, view it on GitHub https://github.com/gilestrolab/ethoscope/issues/178#issuecomment-1581975953, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABSQSRQQU4SEISJEX3CLOKLXKFXWZANCNFSM6AAAAAAUV5GDLU . You are receiving this because you were mentioned.Message ID: @.***>