Closed antortjim closed 2 years ago
antortjim
commented
4 years ago Also, the only way I found that supports systemd services running in Docker containers requires running the --privileged flag in docker run. Ideas on how to run these services without this not recommended flag would be welcome ;)
ggilestro
commented
4 years ago Dear @antortjim, thanks for your very welcome contribution. We did discuss having the node running in docker container before. Regarding the SQL related error messages: when an ethoscope is first initialized and recognized, the node will throw those SQL errors until tracking starts for the first time. If you start a dummy tracking on the ethoscope, a database will be generated and those errors should disappear.
antortjim
commented
4 years ago Dear @ggilestro, you are welcome, I would be very happy if it can be used by others as well. I suspected this error might solve itself upon starting a tracking job. Nice :)
Regarding the start of tracking, since I am running a virtual ethoscope, I can only use a video as input. Indeed, I can select the MovieVirtualCamera as input for the tracking algorithm. But how do I choose a video? Clicking on 'Available movies' does not display anymore options. What folder should I put my videos in?
I have explored the API and I see if I enter the url
localhost:<port>/device/<id>/videofiles
I should get the videofiles or something leading to it, but instead I get
{"error": "Traceback (most recent call last):\n File \"/opt/ethoscope-git/node_src/scripts/server.py\", line 61, in func_wrapper\n return func(*args, **kwargs)\n File \"/opt/ethoscope-git/node_src/scripts/server.py\", line 188, in get_device_videofiles\n return device.videofiles()\n File \"/opt/ethoscope-git/node_src/ethoscope_node/utils/device_scanner.py\", line 308, in videofiles\n videofiles_url = \"http://%s:%i/%s/%s\" % (self._ip, self._port, self._remote_page['videofiles'], self._id)\nAttributeError: 'Device' object has no attribute '_remote_page'\n"}Which indicates that the _remote_page attribute maybe should be made an attribute of the Device instances and not just a class attribute? At least that's what the rest of the code in https://github.com/gilestrolab/ethoscope/blob/57b01fa6d25e656c21cb8d8e620e9fd801247080/node_src/ethoscope_node/utils/device_scanner.py#L165 seems to assume?
Thanks again! Antonio
ggilestro
commented
4 years ago Videos should be on the ethoscope, in the folder:
/ethoscope_data/upload/video/
extensions allowed are mp4 and avi
antortjim
commented
4 years ago I hardcoded a path to a .mp4 file inside my ethoscope Docker container
I also redesigned the containers so the --privileged flag is not required. I now run a mysql server on a separate container and accordingly map the mysql.connectors to that IP (different from that of the ethoscope). This is achieved by having an environment variable defined when running the repository inside Docker
https://github.com/antortjim/docker-ethoscope/blob/b1e8d60edf2484e28038a9e345476d7a394924f7/ethoscope_base/Dockerfile#L105
https://github.com/antortjim/docker-ethoscope/blob/b1e8d60edf2484e28038a9e345476d7a394924f7/ethoscope_base/upstream/node_src/ethoscope_node/utils/device_scanner.py#L437
The whole thing can be run by executing a bash script provided in the repo. It is, however, error-prone because the mysql server is not accessible from the moment the container is run. Some seconds (~30 ) are needed before running the node and ethoscope containers, but that might change. Also, the database needs to be reset after each run, this is done automatically if the ethoscope container is restarted. I have a README.md file that shows these steps with figures. I have some todos listed there too...
The images can be pulled from DockerHub and should work out of the box (haven't tested it on a new PC yet).
Hope it's useful for someone besides me :smile:
Antonio
ggilestro
commented
4 years ago Thanks a lot for this (with some decent delay on my side....). I need to migrate the whole ethoscope manual to another site at one point because gitbook is no longer an option. As soon as I do, I will add a link to your docker container. For now I am leaving the issue open so that I'll remember to do it. Thanks!
antortjim
commented
4 years ago No problem!
I haven't touched it since then and for sure there are things that could be better designed, so it still needs testing and some improvements before it's completely ready.
I have however by now figured out it's more convenient for me to simply set up a conda environment and install all the dependencies needed for the ethoscope package *. Then I start offline tracking by calling the device_server.py script with the --run flag. The result is similar to running an actual ethoscope, to the point that it even registers in the network. I made some modifications to the script and it can be found here https://gitlab.com/antortjim/ethoscope/-/blob/rich_adaptive_bg_tracker/src/scripts/device_server.py
A full CLI I have recently done to do offline analysis:
python device_server.py \
#path to the .mp4 video I want to analyze offline
--input /ethoscope_data/videos/007aad42625f433eb4bd2b44f811738e/ETHOSCOPE_007/2020-06-26_12-30-11/whole_2020-06-26_12-30-11_007aad42625f433eb4bd2b44f811738e__1280x960@12_00000.mp4 \
# path to a pickle file containing a serialized list of ROI instances created upon starting recording (so I can verify the ROIs are good at the beginning of the experiment and not when it is too late)
--rois-pickle-file /ethoscope_data/videos/007aad42625f433eb4bd2b44f811738e/ETHOSCOPE_007/2020-06-26_12-30-11/2020-06-26_12-30-11_007aad42625f433eb4bd2b44f811738e_rois.pickle \
# my videos are 12 FPS but I only want the tracker to "see" 2 FPS so it's similar to the FPS of a tracking ethoscope. I still want videos of 12 FPS so an annotating human can see as much as possible
--drop-each 6 \
# the name the virtual ethoscope instance should get (not a random VIRTUASCOPE)
--name FLYSLEEPLAB_CV1 \
# the ip the emulated ethoscope should use when registering in the network using Zeroconf. This is to done to make sure any other internet interfaces are ignored
--address 192.169.123.10 \
# and this is to actually start tracking and not just "spin up a bottle server and wait for more user input via the JS GUI"
--runI save the results of the offline analysis using the SQLiteResultWriter class instead of the default ResultWriter (which uses MySQL). This way I get directly the sqlite3.db file without the need of a MySQL server.
I think that rather than a docker image we need more documentation on how to perform offline analysis of videos, which is very useful for debugging purposes as well as to try multiple different analyses without the need of repeating the experiments. I hope this helps take us there.
Best regards, Antonio
picamera but because it is lazily imported, as long as one just does offline analysis, it is not needed and thus no problem.ggilestro
commented
2 years ago There is now a section on the ethoscope manual explaining how to run offline tracking from the GUI too.
Dear GilestroLab
One of the main issues I face when working with the ethoscope is how difficult it is to deploy. It is great news that you released a pacman package for that reason. On the other hand, I have been working for a few weeks now on dockerizing the software, for the following reasons:
The Dockerfiles I wrote for this together with a README file on how to run them are here https://gitlab.com/antortjim/docker-ethoscope/
Briefly, I have an ethoscope_base image based on
archlinux/base, containing all the pacman and python packages as well as most time-consuming configurations. From it, I created two more, one for the node and one for an ethoscope device.I can spin up containers and for example experience the new GUI without the need to update our production node server.
However, I am encountering a MySQL issue in the node container
also, I need to provide the option to select a video as input for the recording/tracking options. Do you know a fast way to do this that you can help with? Thank you beforehand!
Antonio