search

gilestrolab / ethoscope

a platform from monitoring animal behaviour in real time from a raspberry pi
http://lab.gilest.ro/ethoscope/
GNU General Public License v3.0
17 stars 25 forks source link

Data not being properly stored on node #105

Closed matt-p-brown closed 4 years ago

matt-p-brown commented 4 years ago

Hey Gilestro Lab,

I am having an issue with my ethoscopes not properly saving tracking data. I have the node pointed at the default directory, and I haven't altered any of the ethoscope_data folders. When I begin a tracking experiment, the dashboard displays the following:

DeepinScreenshot_select-area_20191112215859

Clicking on the file link brings up a tab with the address http://localhost/downloadNone (which gives a 404 error). Let me know if you need any further information. Thanks.

ggilestro commented 4 years ago

I cannot reproduce this so I will need more information to debug it. Find the ethoscope id: the id is a long hex string and you can find in the URL of that page. The first three digits are the same as the ethoscope number.

once you find the id, point your browser to the following page:

http://<node>/device/<id>/data

where is the address of your node and is the ethoscope id. We'll take it from there.

mmatell commented 4 years ago

Hi folks - As Giorgio knows, we are having the same problem. I think this is the information that is being requested. best, matt

browser address: http://node/device/007d951279b8425b915ce435be98a078/data resulting info on page: {"status": "stopped", "ip": "192.169.123.28", "time": 1573500238.6761742, "error": null, "log_file": "/ethoscope_data/results/ethoscope.log", "dbg_img": "/ethoscope_data/results/dbg_img.png", "last_drawn_img": "/tmp/ethoscope_intvvhjt/last_img.jpg", "id": "007d951279b8425b915ce435be98a078", "name": "ETHOSCOPE_007", "version": {"id": "55d60cc41a838cc69411687006133f0132e1b9cf", "date": "2019-11-07 09:35:45"}, "db_name": "ETHOSCOPE_007_db", "monitor_info": {"last_positions": null, "last_time_stamp": 0, "fps": 0}, "experimental_info": {}, "current_timestamp": 1573744472.2112212, "CPU_temp": 44.0, "backup_path": "/ethoscope_data/results/007d951279b8425b915ce435be98a078/ETHOSCOPE_007/2019-11-10_16-22-01/2019-11-10_16-22-01_007d951279b8425b915ce435be98a078.db", "time_since_backup": 323505.19884872437}

matt-p-brown commented 4 years ago

Here's what I get:

{"status": "running", "ip": "192.168.1.103", "time": 1573585600.4688365, "error": null, "log_file": "/ethoscope_data/results/ethoscope.log", "dbg_img": "/ethoscope_data/results/dbg_img.png", "last_drawn_img": "/tmp/ethoscope_55ninr91/last_img.jpg", "id": "00255e9bdb8b4c9da51d74d2c0899f92", "name": "ETHOSCOPE_002", "version": {"id": "fc7ed1a52dfdcbbe338473aecea960648060bd59", "date": "2019-10-07 14:23:56"}, "db_name": "ETHOSCOPE_002_db", "monitor_info": {"last_time_stamp": 162411.049, "fps": 2.15}, "experimental_info": {"name": "mbrown", "location": "Incubator 1", "code": "", "sensor": ""}, "current_timestamp": 1573748011.946246, "CPU_temp": 73.6, "backup_path": "None", "time_since_backup": null}

Let me know if there is anything else you need me to provide.

mmatell commented 4 years ago

Giorgio - please note that the ethoscope message I reported above was from an ethoscope that was updated to the current ethoscope version from the 9/28 image, and was not from a blank install with the newest image. As such, the backup path indicates the date that the ethoscope was last run (and successfully saved) using the older image. Please let me know if you want me to report the output with a new image.

ggilestro commented 4 years ago

guys - probably a silly question but: is your node updated?

matt-p-brown commented 4 years ago

The update page said everything was updated when I posted the issue, but I checked after your comment and it was outdated. I've updated the node and all the ethoscopes and the same problem is still there unfortunately.

mmatell commented 4 years ago

yep - the node has been updated each time there was a new version.

pepelisu commented 4 years ago

I am having the same issue, Node and ethoscopes updated to: [python3.7], 55d60c... (2019-11-07 09:35:45) Looking up the issue I found that the backup service is not working because it cannot find the expected database name ethoscope_db This is the error of the ethoscope_backup.service:

Nov 22 11:53:53 node python[528949]: ERROR:root:Could not generate backup path for device. Probably a MySQL issue

Looking in the ethoscope I discover that now the database are not named ethoscope_db, but ``ETHOSCOPE_007_db``` in my case.

Somehow the code to create the database has been changed and the name is not right. @ggilestro could you tell me if this is right?

ggilestro commented 4 years ago

yes - that is correct. I am exploring the possibility of the ethoscope saving the working db remotely on a mysql different from localhost. This is to minimize the wear and tear of the SD card. To this end, each ethoscope now creates a database whose name is unique.

However, the ethoscope should correctly report the new name as part of its data and this should be picked up by device_scanner.py correctly at line 434

            self._ethoscope_db_credentials["db"] = self._info["db_name"]

I suspect this is a node issue. Perhaps python is failing to update its egg files upon update? Try running on the node the installation line again

pip install -e .

Maybe this is it?

pepelisu commented 4 years ago

that was it. After run pip install -e . in /opt/ethoscope-node/node_src/ (two times). Now the output after reinstall is this:

Nov 22 13:14:35 node python[906954]: INFO:root:Joining now
Nov 22 13:14:35 node python[906954]: INFO:root:Pool mapped
Nov 22 13:14:35 node python[906954]: INFO:root:Backup done for for device  0073a9945647437d926839ef4092cfe8
Nov 22 13:14:35 node python[906954]: INFO:root:Backup done for for device  0089f1eddc2a498fad9bfa8f2da4785b
Nov 22 13:14:35 node python[906954]: AttributeError: 'MySQLdbToSQlite' object has no attribute '_replace_img_snapshot_table'
Nov 22 13:14:35 node python[906954]:     self._replace_img_snapshot_table(table_name, src, dst)
Nov 22 13:14:35 node python[906954]:   File "/opt/ethoscope-node/node_src/ethoscope_node/utils/mysql_backup.py", line 133, in _copy_table
Nov 22 13:14:35 node python[906954]:     self._copy_table(t, src, conn, dump_in_csv=False)
Nov 22 13:14:35 node python[906954]:   File "/opt/ethoscope-node/node_src/ethoscope_node/utils/mysql_backup.py", line 104, in __init__
Nov 22 13:14:35 node python[906954]:     remote_user=self._db_credentials["user"])
Nov 22 13:14:35 node python[906954]:   File "/opt/ethoscope-node/node_src/ethoscope_node/utils/backups_helpers.py", line 40, in run
Nov 22 13:14:35 node python[906954]: ERROR:root:Traceback (most recent call last):
Nov 22 13:14:35 node python[906954]: AttributeError: 'MySQLdbToSQlite' object has no attribute '_replace_img_snapshot_table'
Nov 22 13:14:35 node python[906954]:     self._replace_img_snapshot_table(table_name, src, dst)
Nov 22 13:14:35 node python[906954]:   File "/opt/ethoscope-node/node_src/ethoscope_node/utils/mysql_backup.py", line 133, in _copy_table
Nov 22 13:14:35 node python[906954]:     self._copy_table(t, src, conn, dump_in_csv=False)
Nov 22 13:14:35 node python[906954]:   File "/opt/ethoscope-node/node_src/ethoscope_node/utils/mysql_backup.py", line 104, in __init__
Nov 22 13:14:35 node python[906954]:     remote_user=self._db_credentials["user"])
Nov 22 13:14:35 node python[906954]:   File "/opt/ethoscope-node/node_src/ethoscope_node/utils/backups_helpers.py", line 40, in run
Nov 22 13:14:35 node python[906954]: ERROR:root:Traceback (most recent call last):
Nov 22 13:14:35 node python[906954]: INFO:root:Ensuring DAM file exists at /ethoscope_data/results/0089f1eddc2a498fad9bfa8f2da4785b/ETHOSCOPE_008/2019-11-22_13-11-54/2019-11-22_13-11-54_0089f1eddc2>
Nov 22 13:14:35 node python[906954]: INFO:root:Ensuring DAM file exists at /ethoscope_data/results/0073a9945647437d926839ef4092cfe8/ETHOSCOPE_007/2019-11-22_13-07-11/2019-11-22_13-07-11_0073a994564>
Nov 22 13:14:35 node python[906954]: INFO:root:Success
Nov 22 13:14:35 node python[906954]: INFO:root:Success
Nov 22 13:14:35 node python[906954]: INFO:root:Making parent directories
Nov 22 13:14:35 node python[906954]: INFO:root:Making parent directories
Nov 22 13:14:35 node python[906954]: INFO:root:Initializing local database static tables at /ethoscope_data/results/0089f1eddc2a498fad9bfa8f2da4785b/ETHOSCOPE_008/2019-11-22_13-11-54/2019-11-22_13->
Nov 22 13:14:35 node python[906954]: INFO:root:Initializing local database static tables at /ethoscope_data/results/0073a9945647437d926839ef4092cfe8/ETHOSCOPE_007/2019-11-22_13-07-11/2019-11-22_13->
Nov 22 13:14:35 node python[906954]: INFO:root:Running backup for device  0073a9945647437d926839ef4092cfe8
Nov 22 13:14:35 node python[906954]: INFO:root:Initiating backup for device  0073a9945647437d926839ef4092cfe8
Nov 22 13:14:35 node python[906954]: INFO:root:Running backup for device  0089f1eddc2a498fad9bfa8f2da4785b
Nov 22 13:14:35 node python[906954]: INFO:root:Initiating backup for device  0089f1eddc2a498fad9bfa8f2da4785b

Still an error is shown due to an attribute that does not exists.

ggilestro commented 4 years ago

Fixed by https://github.com/gilestrolab/ethoscope/commit/0567585158ff3a51ea998faa97a8d60285711163 , I believe. Please confirm.

mmatell commented 4 years ago

Hi folks - The most recent update has fixed the problem. I believe that the approach listed above "After run pip install -e . in /opt/ethoscope-node/node_src/ (two times)." was necessary to fix. FYI, need to run the pip as sudo. thanks matt