search

gilestrolab / ethoscope

a platform from monitoring animal behaviour in real time from a raspberry pi
http://lab.gilest.ro/ethoscope/
GNU General Public License v3.0
17 stars 25 forks source link

Ethoscope stops working randomly #188

Closed BrainFriedLab closed 1 week ago

BrainFriedLab commented 7 months ago

Hi there,

I'm certainly a novice at all of this, but we've had a great ton of success troubleshooting. But now, I'm hitting a wall. I can't figure out why this is happening. The ethoscope just stops tracking at random times throughout the run. We were using an ethoscope that had the 2020 image on the micro sd card. It worked well, but then it crashed this week.

I burned the most recent image onto a sandisk 64 GB sd card (only brand I could get quickly to get the experiment back up and running). This worked as expected, but I started a run on Tuesday and came back wednesday to find out it stopped (only collecting about 18 hours of data). I then started another run last night and came back this morning to find that it only collected data for about one hour.

When I click the logs, this is what I see. The error logs here are not from the time it crashed, but instead seem to be right when I click the "view logs" button.

PXL_20240404_154108730

Maybe I have a setting screwed up or something for this 2023 image?

Here's some of the long list of logs from the node page:

0: Apr 04 11:58:17 node python[29356]: df: USB/ethoscope_data/tmp/: No such file or directory Apr 04 11:58:17 node python[29356]: df: /run/media/researcher/Memorex: No such file or directory Apr 04 11:58:14 node python[781]: ERROR:root:Could not generate backup path for device. Probably a MySQL issue Apr 04 11:58:14 node python[781]: WARNING:root:No information regarding backup file from the ethoscope Apr 04 11:58:09 node python[781]: ERROR:root:Could not generate backup path for device. Probably a MySQL issue Apr 04 11:58:09 node python[781]: WARNING:root:No information regarding backup file from the ethoscope Apr 04 11:58:03 node python[781]: ERROR:root:Could not generate backup path for device. Probably a MySQL issue Apr 04 11:58:03 node python[781]: WARNING:root:No information regarding backup file from the ethoscope Apr 04 11:57:59 node python[29352]: df: USB/ethoscope_data/tmp/: No such file or directory Apr 04 11:57:59 node python[29352]: df: /run/media/researcher/Memorex: No such file or directory Apr 04 11:57:58 node python[781]: ERROR:root:Could not generate backup path for device. Probably a MySQL issue Apr 04 11:57:58 node python[781]: WARNING:root:No information regarding backup file from the ethoscope Apr 04 11:57:53 node python[781]: ERROR:root:Could not generate backup path for device. Probably a MySQL issue Apr 04 11:57:53 node python[781]: WARNING:root:No information regarding backup file from the ethoscope Apr 04 11:57:48 node python[781]: ERROR:root:Could not generate backup path for device. Probably a MySQL issue Apr 04 11:57:48 node python[781]: WARNING:root:No information regarding backup file from the ethoscope Apr 04 11:57:43 node python[781]: ERROR:root:Could not generate backup path for device. Probably a MySQL issue Apr 04 11:57:43 node python[781]: WARNING:root:No information regarding backup file from the ethoscope Apr 04 11:57:38 node python[781]: ERROR:root:Could not generate backup path for device. Probably a MySQL issue Apr 04 11:57:38 node python[781]: WARNING:root:No information regarding backup file from the ethoscope Apr 04 11:57:35 node python[781]: Exception: Could not get image for ip = 192.168.0.101 (id = 0051509b9ef4423c8a249ad5a81b0c59) Apr 04 11:57:35 node python[781]: raise Exception("Could not get image for ip = %s (id = %s)" % (self._ip, self._id)) Apr 04 11:57:35 node python[781]: File "/opt/ethoscope-node/node_src/ethoscope_node/utils/device_scanner.py", line 438, in last_image Apr 04 11:57:35 node python[781]: ^^^^^^^^^^^^^^^^^^^ Apr 04 11:57:35 node python[781]: file_like = device.last_image() Apr 04 11:57:35 node python[781]: File "/opt/ethoscope-node/node_src/scripts/server.py", line 264, in get_device_last_img Apr 04 11:57:35 node python[781]: ^^^^^^^^^^^^^^^^^^^^^ Apr 04 11:57:35 node python[781]: return func(*args, *kwargs) Apr 04 11:57:35 node python[781]: File "/opt/ethoscope-node/node_src/scripts/server.py", line 43, in func_wrapper Apr 04 11:57:35 node python[781]: Traceback (most recent call last): Apr 04 11:57:35 node python[781]: During handling of the above exception, another exception occurred: Apr 04 11:57:35 node python[781]: urllib.error.HTTPError: HTTP Error 404: Not Found Apr 04 11:57:35 node python[781]: raise HTTPError(req.full_url, code, msg, hdrs, fp) Apr 04 11:57:35 node python[781]: File "/usr/lib/python3.11/urllib/request.py", line 643, in http_error_default Apr 04 11:57:35 node python[781]: ^^^^^^^^^^^ Apr 04 11:57:35 node python[781]: result = func(args) Apr 04 11:57:35 node python[781]: File "/usr/lib/python3.11/urllib/request.py", line 496, in _call_chain Apr 04 11:57:35 node python[781]: ^^^^^^^^^^^^^^^^^^^^^^^ Apr 04 11:57:35 node python[781]: return self._call_chain(*args) Apr 04 11:57:35 node python[781]: File "/usr/lib/python3.11/urllib/request.py", line 563, in error Apr 04 11:57:35 node python[781]: ^^^^^^^^^^^^^^^^^^ Apr 04 11:57:35 node python[781]: response = self.parent.error( Apr 04 11:57:35 node python[781]: File "/usr/lib/python3.11/urllib/request.py", line 634, in http_response Apr 04 11:57:35 node python[781]: ^^^^^^^^^^^^^^^^^^^ Apr 04 11:57:35 node python[781]: response = meth(req, response) Apr 04 11:57:35 node python[781]: File "/usr/lib/python3.11/urllib/request.py", line 525, in open Apr 04 11:57:35 node python[781]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Apr 04 11:57:35 node python[781]: return opener.open(url, data, timeout) Apr 04 11:57:35 node python[781]: File "/usr/lib/python3.11/urllib/request.py", line 216, in urlopen Apr 04 11:57:35 node python[781]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Apr 04 11:57:35 node python[781]: return urllib.request.urlopen(img_url,timeout=5) Apr 04 11:57:35 node python[781]: File "/opt/ethoscope-node/node_src/ethoscope_node/utils/device_scanner.py", line 435, in last_image Apr 04 11:57:35 node python[781]: ERROR:root:Traceback (most recent call last): Apr 04 11:57:35 node python[781]: ERROR:root:Could not get image for ip = 192.168.0.101 (id = 0051509b9ef4423c8a249ad5a81b0c59) Apr 04 11:57:33 node python[781]: ERROR:root:Could not generate backup path for device. Probably a MySQL issue

BrainFriedLab commented 7 months ago

Just came back and its still running. I'll report back in tomorrow morning. A few other things I noticed:

  1. Although I updated the node a month ago and burned a new sd card, I realized the system still needed more updates. So, I went ahead and updated the node and ethoscope.

  2. I also wonder if our IT specialist may have set up the backup location incorrectly and that's causing an issue for the ethoscope - somehow stopping tracking when it is confused with where to send the files? With our previous ethoscope (and this one), we would go to the tmp folder indicated in the image below that was pointed to memorex USB (which I believe is the actual sd card on the ethoscope because the folder disappears from my node when the ethoscope is off) and just copy the db and txt file to another folder on the computer. It seems like the backup directory is currently set to just store the data on the ethoscope itself instead of somewhere on the node? Honestly... It's questionable if I should even have a github account at this point. ha. I'm happy getting my .db and .txt files from each run, but I should also note that I can't find any results or video folders anywhere on this node or ethoscope. I only see a tmp folder.

image

ggilestro commented 7 months ago

Hi there!

The errors you are seeing in your logs just refer to the fact that the ethoscope suddenly goes offline. Sudden crashes like this one at variable times are normally due to a hardware problem. The most common culprit is the SD card, the second most common culprit is insufficient power from the ethoscope power supply but a worn out ethoscope is not to be excluded.

Regarding the data backup folder: if the data backup folder is not correctly configured the node may fail but the ethoscope should not. If you see that your data are not saved correctly, I just recommend using no blank spaces in the folder path. In your case, you should rename the mount point from "Memorex USB" to "Memorex_USB" or something like that. This implies changing the actual folder name, not just the configuration.

Hope this helps.

BrainFriedLab commented 7 months ago

I just got back to the lab this morning and it's still running! I guess just updating it helped do the trick. I have some back up SD cards, however, in case this happens again. And I guess I should make "building an ethoscope" a class activity. We're a lab of only undergrads here at Rutgers Camden, just outside Philly. This has been such a great system to help teach them troubleshooting, independence, and research methods. THANK YOU for building and supporting it!!!

I'll make the changes to the folder as recommended. I appreciate your help!