No video files saved upon tracking

[defra1]

Dear Gilestro lab, I am using a RPI 3b+ for my ethoscope with a RPI4 as the node. Upon updating everything, I ran a test experiment. I was able to download a zip file of the .txt and .db file, but I was not able to get a video of the tracking.

There was also an error in the experiment downloads.

I have also included a screenshot of the log files.

I wanted to find where the videos were, I looked on the node:

`Last login: Wed Dec 1 22:44:11 2021 from 192.168.1.8

[node@node ~]$ su root

Password:

[root@node node]# find /ethoscope_data/videos

/ethoscope_data/videos

[root@node node]# ls

[root@node node]# ls /ethoscope_data/videos # I did this to make sure that I really am in the right directory

[root@node node]# cd /ethoscope_data/

[root@node ethoscope_data]# ls

results tmp videos

[root@node ethoscope_data]# cd /ethoscope_data/videos

[root@node videos]# ls

** No videos listed here

[root@node videos]# cd

[root@node ~]# cd /ethoscope_data/tmp

[root@node tmp]# ls

[root@node tmp]# cd /ethoscope_data/results

[root@node results]# ls

001f51ac632d444bba30e8776e4caa6c results_230607_235029.zip

** The results here are the ones that I can see on the GUI, just the .db files and the .txt files that I downloaded.

` Then I looked for the videos on the ethoscope:

`Last login: Sun Mar 5 01:33:38 2023 from 192.168.68.105

[alarm@ETHOSCOPE001 ~]$ cd /ethoscope_data/

[alarm@ETHOSCOPE001 ethoscope_data]$ ls #Shouldn’t there be a video folder too?

results

[alarm@ETHOSCOPE001 ethoscope_data]$ cd /ethoscope_data/results/

[alarm@ETHOSCOPE001 results]$ ls

001f51ac632d444bba30e8776e4caa6c index.html

[alarm@ETHOSCOPE001 results]$ `

Thank you

[pepelisu]

Hi, If you are using the "Dev" branch, there is a bug in the ethoscope_video_backup.py file. I just discovered the in line 9 the import fails due to receive_devices, it is used only in line 134 in the same file. I think this is something from an old version that got oversaw.

Could @ggilestro check it? Either the function receive_devices is missing in backup_utils.py or should be removed from ethoscope_video_backup.py

Maybe this is the new function find_devices and the name was not updated in the video backup.

[pepelisu]

I think the problem is deeper. with the rewrite of the backup helper the class BackupClass has only a method, backup where it calls the MySQLToSqlite class, so it does not backup any videos, just the tracking databases.

A new class VideoBackupWrapper should be implemented (inheriting from GenericBackupWrapper) in the backup_helpers.py, with a new backup function for videos. that way the ethoscope_backup_video.py will be able to call gbw = VideoBackupWrapper(...).

Is that so @ggilestro ? I am not familiar with the new backup daemon.

[defra1]

@pepelisu Thank you for the response. The initial OS for the ethoscope was 20230306_ethsocope_000. Since I am using a RPI4 for my node I burned the 20211203_ethoscope-node_pi4 OS onto my SD card.

When I started things up, the dates and times were not matching between the node and my computer (off by months and years). Once this was fixed:

systemctl stop ntpd ntpdate 0.au.pool.ntp.org systemctl start ntpd

I was able to update both the node and ethoscope. I use the Master branch of the node for my update. I cannot remember which branch I used for the ethoscope, I suppose it was whatever is the default?

[pepelisu]

Just to quickly clarify your issue. If you started a tracking, then you would only get a db and txt files. Inside the db some images are saved every 5 min to keep some sort of control, but the entire experiment video is not saved. If you want to save the video and not the tracking parameters, you have to use the video->record menu. If you did that then, most probably the video is not saved due to the bug in the code.

[defra1]

@pepelisu I haven't tried to record a video without tracking, but I can try it. What you are saying is correct, I can download the .txt and .db files and there is data in them. But the video of the experiment is not saved.

[defra1]

@pepelisu I have now tried the recording feature. Here is a screenshot of what happens (there is no video image):

Here are my settings:

Here is the log during the recording:

Here is what I see when streaming (nothing wrong with the camera):

I hope that this is helpful. Thank you, Dena

[pepelisu]

Could you please add a picture of the "Update system" website to check which branch are you using? [master] or [dev]

From your previous screenshots, I can only say that something is happening with the preview of the image, that does not mean that the system is not recording and I would urge you to check in the /ethoscope_data/videos folder in the node.

[defra1]

@pepelisu , I have looked at the updates and I see that the ethoscope is again out of date and it is the dev branch.

I also tried to find the video recording (not the trakcing) that should have been stored from yesterday and I still do not see the videos file on the ethoscope:

[alarm@ETHOSCOPE001 ~]$ cd /ethoscope_data/
[alarm@ETHOSCOPE001 ethoscope_data]$ ls
results
[alarm@ETHOSCOPE001 ethoscope_data]$ cd
[alarm@ETHOSCOPE001 ~]$ cd /ethoscope_data/results
[alarm@ETHOSCOPE001 results]$ ls
001f51ac632d444bba30e8776e4caa6c  index.html
[alarm@ETHOSCOPE001 results]$

Then I updated the ethoscope with the master branch. Once I did this and rebooted the node and the ethoscope, the ethoscope was no longer showing up on the node GUI.

Seems that the status of the updated ethoscope_001 was degraded.

`[root@ETHOSCOPE001 alarm]# systemctl status
* ETHOSCOPE001
    State: degraded
    Units: 234 loaded (incl. loaded aliases)
     Jobs: 0 queued
   Failed: 1 units
    Since: Thu 1970-01-01 00:00:05 UTC; 53 years 5 months ago
  systemd: 253.1-1-arch
   CGroup: /
           |-init.scope
           | `-1 /sbin/init
           |-system.slice
           | |-avahi-daemon.service
           | | |-224 "avahi-daemon: running [ETHOSCOPE001.local]"
           | | `-229 "avahi-daemon: chroot helper"
           | |-dbus.service
           | | `-225 /usr/bin/dbus-daemon --system --address=systemd: --nofork >
           | |-ethoscope_GPIO_listener.service
           | | `-226 /usr/bin/python3 /opt/ethoscope-device/src/ethoscope/hardw>
           | |-ethoscope_update.service
           | | `-245 /usr/bin/python3 /opt/ethoscope_updater/update_server.py ->
           | |-mariadb.service
           | | `-285 /usr/bin/mariadbd
           | |-ntpd.service

` I checked the systemctl to see which service was affected: Looks like it is the systemd-journal-upload.service see below:

[root@ETHOSCOPE001 alarm]# systemctl
  UNIT                                                                         >
  proc-sys-fs-binfmt_misc.automount                                            >
  sys-devices-platform-serial8250-tty-ttyS0.device                             >
  sys-devices-platform-soc-3f201000.serial-tty-ttyAMA0.device                  >
  sys-devices-platform-soc-3f202000.mmc-mmc_host-mmc0-mmc0:0001-block-mmcblk0-m>
  sys-devices-platform-soc-3f202000.mmc-mmc_host-mmc0-mmc0:0001-block-mmcblk0-m>
  sys-devices-platform-soc-3f202000.mmc-mmc_host-mmc0-mmc0:0001-block-mmcblk0.d>
  sys-devices-platform-soc-3f300000.mmcnr-mmc_host-mmc1-mmc1:0001-mmc1:0001:1-n>
  sys-devices-platform-soc-3f980000.usb-usb1-1\x2d1-1\x2d1.1-1\x2d1.1:1.0-net-e>
  sys-devices-virtual-block-ram0.device                                        >
  sys-devices-virtual-block-ram1.device                                        >
  sys-devices-virtual-block-ram10.device                                       >
  sys-devices-virtual-block-ram11.device                                       >
  sys-devices-virtual-block-ram12.device                                       >
  sys-devices-virtual-block-ram13.device                                       >
  sys-devices-virtual-block-ram14.device                                       >
  sys-devices-virtual-block-ram15.device                                       >
  sys-devices-virtual-block-ram2.device                                        >
  sys-devices-virtual-block-ram3.device                                        >
  sys-devices-virtual-block-ram4.device                                        >
  sys-devices-virtual-block-ram5.device                                        >
  sys-devices-virtual-block-ram6.device                                        >
  sys-devices-virtual-block-ram7.device                                        >
  sys-devices-virtual-block-ram8.device                                        >
  sys-devices-virtual-block-ram9.device                                        >
  sys-devices-virtual-misc-rfkill.device                                       >
  sys-devices-virtual-tty-ttyprintk.device                                     >
  sys-module-configfs.device                                                   >
  sys-module-fuse.device                                                       >
  sys-subsystem-net-devices-eth0.device                                        >
  sys-subsystem-net-devices-wlan0.device                                       >
  -.mount                                                                      >
  boot.mount                                                                   >
  dev-mqueue.mount                                                             >
  run-credentials-systemd\x2dresolved.service.mount                            >
  run-credentials-systemd\x2dsysctl.service.mount                              >
  run-credentials-systemd\x2dtmpfiles\x2dsetup.service.mount                   >
  run-credentials-systemd\x2dtmpfiles\x2dsetup\x2ddev.service.mount            >
  run-user-1000.mount                                                          >
  sys-fs-fuse-connections.mount                                                >
  sys-kernel-config.mount                                                      >
  sys-kernel-debug.mount                                                       >
  sys-kernel-tracing.mount                                                     >
  tmp.mount                                                                    >
  systemd-ask-password-console.path                                            >
  systemd-ask-password-wall.path                                               >
  init.scope                                                                   >
  session-1.scope                                                              >
  avahi-daemon.service                                                         >
  dbus.service                                                                 >
  ethoscope_device.service                                                     >
  ethoscope_GPIO_listener.service                                              >
  ethoscope_listener.service                                                   >
  ethoscope_update.service                                                     >
  getty@tty1.service                                                           >
  kmod-static-nodes.service                                                    >
  mariadb.service                                                              >
  ntpd.service                                                                 >
  serial-getty@ttyAMA0.service                                                 >
  sshd.service                                                                 >
  systemd-journal-flush.service                                                >
* systemd-journal-upload.service                                               >
  systemd-journald.service                                                     >
  systemd-logind.service                                                       >
  systemd-modules-load.service                                                 >
  systemd-network-generator.service                                            >
  systemd-networkd-wait-online.service                                         >
  systemd-networkd.service                                                     >
  systemd-random-seed.service                                                  >
  systemd-remount-fs.service                                                   >
  systemd-resolved.service                                                     >
  systemd-sysctl.service                                                       >
  systemd-tmpfiles-setup-dev.service                                           >
  systemd-tmpfiles-setup.service                                               >
  systemd-udev-trigger.service                                                 >
  systemd-udevd.service                                                        >
  systemd-update-utmp.service                                                  >
  systemd-user-sessions.service                                                >
  user-runtime-dir@1000.service                                                >
  user@1000.service                                                            >
  wpa_supplicant.service                                                       >
  wpa_supplicant@wlan0.service                                                 >
  -.slice                                                                      >
  system-getty.slice                                                           >
  system-modprobe.slice                                                        >
  system-serial\x2dgetty.slice                                                 >
  system-wpa_supplicant.slice                                                  >
  system.slice                                                                 >
  user-1000.slice                                                              >
  user.slice                                                                   >
  avahi-daemon.socket                                                          >
  dbus.socket                                                                  >
  dm-event.socket                                                              >
  systemd-coredump.socket                                                      >
  systemd-journald-dev-log.socket                                              >
  systemd-journald.socket                                                      >
  systemd-networkd.socket                                                      >
  systemd-rfkill.socket                                                        >
  systemd-udevd-control.socket                                                 >
  systemd-udevd-kernel.socket                                                  >
  basic.target                                                                 >
  cryptsetup.target                                                            >
  getty.target                                                                 >
  graphical.target                                                             >
  integritysetup.target                                                        >
  local-fs-pre.target                                                          >
  local-fs.target                                                              >
  UNIT                                                                         >
  proc-sys-fs-binfmt_misc.automount                                            >
  sys-devices-platform-serial8250-tty-ttyS0.device                             >
  sys-devices-platform-soc-3f201000.serial-tty-ttyAMA0.device                  >
  sys-devices-platform-soc-3f202000.mmc-mmc_host-mmc0-mmc0:0001-block-mmcblk0-m>
  sys-devices-platform-soc-3f202000.mmc-mmc_host-mmc0-mmc0:0001-block-mmcblk0-m>
  sys-devices-platform-soc-3f202000.mmc-mmc_host-mmc0-mmc0:0001-block-mmcblk0.d>
  sys-devices-platform-soc-3f300000.mmcnr-mmc_host-mmc1-mmc1:0001-mmc1:0001:1-n>
  sys-devices-platform-soc-3f980000.usb-usb1-1\x2d1-1\x2d1.1-1\x2d1.1:1.0-net-e>
  sys-devices-virtual-block-ram0.device                                        >
  sys-devices-virtual-block-ram1.device                                        >
  sys-devices-virtual-block-ram10.device                                       >
  sys-devices-virtual-block-ram11.device                                       >
  sys-devices-virtual-block-ram12.device                                       >
  sys-devices-virtual-block-ram13.device                                       >
  sys-devices-virtual-block-ram14.device                                       >
  sys-devices-virtual-block-ram15.device                                       >
  sys-devices-virtual-block-ram2.device                                        >
  sys-devices-virtual-block-ram3.device                                        >
  sys-devices-virtual-block-ram4.device                                        >
  sys-devices-virtual-block-ram5.device                                        >
  sys-devices-virtual-block-ram6.device                                        >
  sys-devices-virtual-block-ram7.device                                        >
  sys-devices-virtual-block-ram8.device                                        >
  sys-devices-virtual-block-ram9.device                                        >
  sys-devices-virtual-misc-rfkill.device                                       >
  sys-devices-virtual-tty-ttyprintk.device

Indeed:

systemd-journal-upload.service - Journal Remote Upload Service
     Loaded: loaded (/usr/lib/systemd/system/systemd-journal-upload.service; enabled; preset: disabl>
    Drop-In: /etc/systemd/system/systemd-journal-upload.service.d
             `-override-timing.conf
     Active: failed (Result: exit-code) since Sun 2023-03-05 01:33:33 UTC; 3 months 18 days ago
   Duration: 307ms
       Docs: man:systemd-journal-upload(8)
    Process: 321 ExecStart=/usr/lib/systemd/systemd-journal-upload --save-state (code=exited, status>
   Main PID: 321 (code=exited, status=1/FAILURE)
     Status: "Shutting down..."
      Error: 5 (Input/output error)
        CPU: 271ms

Mar 05 01:33:32 ETHOSCOPE001 systemd[1]: systemd-journal-upload.service: Failed with result 'exit-co>
Mar 05 01:33:33 ETHOSCOPE001 systemd[1]: systemd-journal-upload.service: Scheduled restart job, rest>
Mar 05 01:33:33 ETHOSCOPE001 systemd[1]: Stopped Journal Remote Upload Service.
Mar 05 01:33:33 ETHOSCOPE001 systemd[1]: systemd-journal-upload.service: Start request repeated too >
Mar 05 01:33:33 ETHOSCOPE001 systemd[1]: systemd-journal-upload.service: Failed with result 'exit-co>
Mar 05 01:33:33 ETHOSCOPE001 systemd[1]: Failed to start Journal Remote Upload Service.

Anyways, this is probably too much code. So after the update to the master ethoscope_src the ethoscope is offline. Not sure how much help this is... Thanks

[pepelisu]

Hi, Ok so if the node is in the branch dev, then you are affected by a bug that will be solved in the next commit. I would recommend to keep the node in the dev and the ethoscopes in the dev branch. What is does not work or should not work is if you have the node and the ethoscopes in different branches.

If you can revert the ethoscope to dev branch and update the node (also in dev branch) when the new commit is available.

[defra1]

@pepelisu the node is not in branch dev at the moment, I updated it as master after I set it up. I can change it to dev now. However, since the ethoscope is not being picked up by the node, I might have to re-image the SD card again. Unless there is another way to update the ethoscope to the dev branch via ssh through the node. Thank you

[pepelisu]

you could try to manually add the ip of the ethoscope to the node (in the node main website) if that does not work, then ssh into the ethoscope and perform git pull in the following folder: /opt/ethoscope-device/

[ggilestro]

Hi @defra1 the commit I just pushed contains fixes that should sort out the video recording backup issue. The easiest is to reburn an SD card. This will automatically get you started on dev , which is the branch I recommend you use.

[defra1]

@ggilestro thank you for doing this. I have now re-imaged the sd card and updated the ethoscope and the node to the dev version (the latest one):

Everything was going well and I did a test-tracking experiment for a short time. However, it still looks like there is an issue with the data backup.

Then I tried the video recording, which showed an image this time (unlike the previous times).

I tried to look for the video files on the ethoscope again

[alarm@ETHOSCOPE001 ~]$ cd /ethoscope_data/
[alarm@ETHOSCOPE001 ethoscope_data]$ ls
results

But there is no videos folder in the ethoscope_data directory. I am not sure if there is something I am missing. All of the backup services are active (except the virtuascope). The other thing to note in the pic below, is that when it is tracking, there is no information about the last backup. There is also a lightning bolt that says there is not enough power (very helpful). I am using the anker usb charging station (as per recommended in the manual).

I looked on the node now for the video files:

[node@node ~]$ cd /ethoscope_data/videos/
[node@node videos]$ ls
[node@node videos]$ cd
[node@node ~]$ cd /ethoscope_data/tmp/
[node@node tmp]$ ls
[node@node tmp]$

Thanks

[pepelisu]

Ok, I see at least one thing that is not alright. The ethoscope is not updated to the last commit on dev, I know it says otherwise, but please reload the update manager and check it again. Commit number should be c291352

After that the video backup should be working.

Just to clarify, the videos in the ethoscope are saved in /ethoscope_data/results/ethoscope_id/... they are saved as .avi in from the last version in dev, but it was saved as .h264 before. To download the videos, the ethoscope needs to create a index.html file that contains the videos index, however the filter to look up for video files was set up to ".h264" instead of ".avi", now corrected in the [c291352] commit.

Data for tracking should still work, so please if you can try to record and also track for some time (more than 10 min) and then check the data and the logs.

[defra1]

@pepelisu I am having some issues with backing up the ethoscope. See the pic below. I attempted to back up the ethoscope, but it says the software is broken. However, when I restart the ethoscope after the backup, it reverts to the previous dev from June 2nd and says that it is up to date. I have also attached a picture of the journal log.

Thanks

[pepelisu]

I am not sure what is happening, it could be that there is a typo in the last version or something is wrong. Just a heads up, the software broken appears always after an update, you should reload the website to ask for a rescan. The reason is that the system becomes unresponsive for a short time while the system is being updated.

Maybe @ggilestro could check if there is something wrong with the code. I cannot test it at the moment since my ethoscopes are running an experiment.

[defra1]

Hi @pepelisu I have been away teaching for the last three months and have now started up the ethosocope again. I still cannot update the ethoscope to the newest version via the node. Would you advise to just re-image the ethoscope SD card with the latest update? Thanks

[pepelisu]

That is exactly what I would suggest. Better start with the last available image, and see what happens. Otherwise we could ne missing something that was changed or that we ask you to change.

[defra1]

@pepelisu Thanks, 1) when you say the last available image do you mean the 20230306.img file? 2) If I am to use the latest dev version (I think it is now faa... something), I cannot just download that off of your github site as is but would need to create the SD image from scratch.

[defra1]

@pepelisu I was able to run the ethoscope over the weekend and I can now see that there is an avi file from the experiment on the node. Thank you

[defra1]

Hi, I just ran another experiment using that latest dev commit for the node and ethoscopes and I have lost the videos and I had to force backup my data from the ethoscopes onto the node.

Node [Up to Date] NA [dev] 13ddfe... (2023-10-13 11:49:00)

ETHOSCOPE_001 [Up to Date] running [dev] 13ddfe... (2023-10-13 11:49:00)

ETHOSCOPE_002 [Up to Date] running [dev] 13ddfe... (2023-10-13 11:49:00)

However it looks like there is a backup issue with the node. I checked the ethoscope_video_backup and the ethoscope_backup status and get the same thing as this example:

root@node node]# systemctl status ethoscope_video_backup
* ethoscope_video_backup.service - Ethoscope node server
     Loaded: loaded (/usr/lib/systemd/system/ethoscope_video_backup.service; enabled; vendor preset: disabled)
     Active: active (running) since Wed 2021-12-01 00:41:14 UTC; 1 year 11 months ago
   Main PID: 371 (python)
      Tasks: 86 (limit: 4915)
        CPU: 19h 29min 34.735s
     CGroup: /system.slice/ethoscope_video_backup.service
             |- 371 /usr/bin/python /opt/ethoscope-node/node_src/scripts/video_backup_tool.py
             `-9290 /usr/bin/python /opt/ethoscope-node/node_src/scripts/video_backup_tool.py

Nov 28 06:30:58 node python[371]: WARNING:root:error while scanning: Errortimed out
Nov 28 06:30:58 node python[371]: WARNING:root:ethoscope not reachable, previous state running
Nov 28 06:30:59 node python[371]: WARNING:root:error while scanning: Errortimed out
Nov 28 06:30:59 node python[371]: WARNING:root:ethoscope not reachable, previous state running
Nov 28 06:31:02 node python[371]: WARNING:root:error while scanning: Errortimed out
Nov 28 06:31:02 node python[371]: WARNING:root:stethoscope not reachable, previous state running
Nov 28 06:31:08 node python[371]: WARNING:root:error while scanning: Errortimed out
Nov 28 06:31:08 node python[371]: WARNING:root:ethoscope not reachable, previous state running
Nov 28 06:40:58 node python[371]: WARNING:root:error while scanning: Errortimed out
Nov 28 06:40:58 node python[371]: WARNING:root:ethoscope not reachable, previous state running

Once again, I checked the node for results

[root@node node]# cd /ethoscope_data/
[root@node ethoscope_data]# ls
results  tmp  videos
[root@node ethoscope_data]# cd /ethoscope_data/videos/
[root@node videos]# ls

I also had to do a forced backup from the ethoscopes to get my .db and .txt files onto the node. The .avi files that I saw from the previous commit have disappeared. I am not sure what happened to the data? I am connecting the ethoscopes to the router via ethernet cable, by the way. Thank you!

[defra1]

I have a quick update on this issue. I checked my .db files when I tried to load them into rethomics and although it looks like there is something there (187 MB) worth of data, I got an error:

> dt <- load_ethoscope(metadata, verbose=FALSE)
Error: no such table: VAR_MAP

Does this mean that there was no actual tracking data stored? I tried the code on .db files from previous runs and had no issues. I am concerned that there was no tracking data at all from this week-long experiment.

[ggilestro]

I am afraid these look like symptoms that the SD card has failed and needs to be replaced. Failing SD files are by far the most common reason of why ethoscopes sometimes fail. I am working to change the platform to use SSD drives instead of SD cards. I anticipate this will be available in January.

[defra1]

@ggilestro thank you for the information. I only use the samsung evo 32GB SD cards. I will re-image a new card and try that out. I am also using a RPI4 as the node and I am thinking of switching to an actual computer with more power, since the RPI4 on the node has been problematic for me.

[ggilestro]

The node certainly is not guilty in this particular case but yes, I fully agree: running the node on a proper computer is recommended.

[defra1]

Hi @ggilestro , I have burned a new SD card for the ethoscope and I am running the node off of a computer instead of the RPI4. I am still having issues with updating to the latest commit for the ethoscope. The node is up to date:

Node [Up to Date] | NA | [dev] 00d389... (2024-02-08 18:53:32) | http://localhost:8888 | N/A |   ETHOSCOPE_001 [Up to Date] | stopped | [dev] 331267... (2023-02-06 11:46:58) | http://192.168.1.9 | 001879 |