PhytO-ARM is robot operating system (ROS)-based toolkit for integration of Imaging FlowCytobot (IFCB) and other ocean sensors that are commonly deployed by the Brosnahan Lab. The supported ROS release is Noetic Ninjemys.
There are instructions for assembling a profiling configuration at our accompanying PhytO-ARM website. The profiling system consists of an IFCB, an AML ctd, and a winch built from a JVL servo motor and parts from Conedrive and Waterman Industries. Once assembled, code provided here enables a variety of continuous sampling behaviors including continuous CTD profiling and targeted collection of IFCB samples at chlorophyll maxima. Other scientific payloads are also supported, and PhytO-ARM can coordinate an arbitrary number of winches running concurrently.
PhytO-ARM observations of location, depth, water properties, etc. are written to IFCB metadata files at time of collection for distribution through an existing webservices system called [IFCB dashboard][ifcbdb]. Additional scripts are provided for export to CF-compliant NetCDF data formats and upload to repositories like the NOAA ERDDAP system.
During and after deployments, system operators can review system data using standard ROS tools like FoxGlove Studio.
Three types of ROS nodes are provided. Device nodes enable integration of IFCB, CTD, GPS, servo, and IP camera systems. Behavior nodes interpret device node data streams and direct changes in system behavior. System nodes log ROS traffic and make it available for real-time interactive display (Rosbridge).
The following are required for PhytO-ARM to run:
The following devices are optional:
example.yaml
for how to enable/disable)First ensure docker
is installed:
docker -v
If this does not print a version number, install with:
sudo apt-get update && sudo apt-get install docker-io
Running Docker will also require sudo
unless you add your user to the docker
group:
# Adds user 'ifcb' to 'docker' group
sudo usermod -aG docker ifcb && newgrp docker
Finally, pull the phyto-arm
image. Container images are built for x86_64
and aarch64
and published automatically on Docker Hub by the continuous integration system.
docker pull whoi/phyto-arm:latest
Alternatively, the container image can be built with
docker build --tag whoi/phyto-arm .
These instructions are for install of nodes on an IFCB running Debian 10 ("Buster") OS. Additional instructions are provided for installation as dockerized components that may be distributed across several systems on a network.
These steps assume that ROS Noetic has been installed already.
source /opt/ros/noetic/setup.bash
# Install apt package dependencies
sed 's/#.*//' deps/apt-requirements.txt | envsubst \
| xargs sudo apt install -y
# Clone source dependencies
vcs import src < deps/deps.rosinstall
# Install ROS dependencies
sudo rosdep init
rosdep update
rosdep install --default-yes --from-paths ./src --ignore-src
# Patch build configuration for cv_bridge
# https://github.com/ros-perception/vision_opencv/issues/345
sudo sed -i 's,/usr/include/opencv,/usr/include/opencv4,g' \
/opt/ros/noetic/share/cv_bridge/cmake/cv_bridgeConfig.cmake
# Create Python virtual environment
python3 -m virtualenv .venv --system-site-packages
. venv/bin/activate
python3 -m pip install -r deps/python3-requirements.txt
# Create Catkin workspace
catkin init
# Build
catkin build phyto_arm
# Add user to dialout group (takes effect on next login)
sudo usermod -a -G dialout $USER
sudo ln -sf $(pwd)/phyto-arm.service /etc/systemd/system/phyto-arm.service
sudo systemctl daemon-reload
sudo systemctl enable phyto-arm
sudo systemctl start phyto-arm
The phyto-arm
tool starts all of the ROS nodes and loads the provided configuration file.
phyto-arm start [-h] [--config_schema <schema file>] [--skip_validation] \
<launchfile name> <config file>
positional arguments:
<launchfile name>
Name of ROS launchfile to use in the phyto_arm package, excluding extension.
Currently one of 'main', 'arm_ifcb' or 'arm_chanos'.
<config file>
Path to YAML config file.
options:
-h, --help
Show help message.
--config_schema <schema file>
File to validate config against.
--skip_validation
Skip validation check altogether.
If running natively, you may kill the process with:
phyto-arm stop
Below is an example Docker run script which includes both the phyto-arm
command as well as the necesssary docker parameters to access devices, ports, and directories on the host. For your convenience, this command is also available as a script in scripts/docker_run.sh
.
docker run --rm -it \
# Name of the container in docker
--name phyto-arm \
# Expose whatever port is being used by Foxglove
--publish 9090:9090/tcp \
# Expose whatever port is being used by web_node
--publish 8098:8098/tcp \
# If using an RBR CTD streaming over UDP, open the necessary port
--publish 12345:12345/udp \
# Bind config directory host:container. ':ro' = read-only
--volume "$(pwd)"/configs:/configs:ro \
# Bind IFCBAcquire routines directory host:container
--volume /home/ifcb/IFCBacquire/Host/Routines:/routines:ro \
# host:container bind for saving files to host, in this case
# for the log_dir and rosbag_prefix settings in example.yaml
--volume /mnt/data:/mnt/data \
# If using an AML CTD over serial, give access to device
--device /dev/ttyS3 \
# Name of Docker image:tag to use
whoi/phyto-arm:latest \
# Start command. Change to use desired config file, should be
# in the container config directory mapped above.
./phyto-arm start main /configs/config.yaml
The above script will start the main PhytO-ARM processes, but not launch any of the winch "arms" where science missions are defined. Currently there are two science payload implementations, for IFCB missions and Chanos sensor missions, called arm_ifcb
and arm_chanos
respectively.
Launching the arms independently means they can also be stopped, configured, and relaunched without interrupting other PhytO-ARM missions in progress.
After launching main
with the script above, you can launch the arms with the following commands:
docker exec -it phyto-arm ./phyto-arm start arm_ifcb /configs/config.yaml
and
docker exec -it phyto-arm ./phyto-arm start arm_chanos /configs/config.yaml
To launch all three simultaneously as separate panes in a tmux session, run the scripts/tmux_run.sh
script. Running this a second time will attach you to the existing session. To kill the session and all processes running within, use the scripts/tmux_kill.sh
script.
The phyto-arm
tool starts the main ROS nodes and loads the provided configuration file.
$ ./phyto-arm start main config/config.yaml
$ ./phyto-arm stop
In seperate terminal sessions, launch one or more arms:
$ ./phyto-arm start arm_chanos config/config.yaml
$ ./phyto-arm start arm_ifcb config/config.yaml
The ROS nodes will be run in the background (so that you can disconnect from the system, for example).
Note: All instruments must already be logging data. Some notes on configuring instruments are included below.
Note that there are also launch files which substitute the IFCB, CTDs and motors for mock versions, suitable for local development and testing. These can be launched as:
docker exec -it phyto-arm start mock_arm_ifcb /configs/config.yaml
and
docker exec -it phyto-arm start mock_arm_chanos /configs/config.yaml
To run unit tests, execute
docker run -it --rm whoi/phyto-arm:latest /bin/bash -c "source devel/setup.bash && catkin test phyto_arm"
Configuration files live in configs/
and use the YAML format. It is recommended that each deployment get its own configuration. An example config file is provided in example.yaml
.
A strict YAML schema is not defined, instead another YAML file is used as a schema to compare against at runtime. configs/example.yaml
is used by default, and validation is done by comparing both structure and data types.
You can skip validation by passing the --skip_validation
argument to phyto-arm
.
You can also provide your own schema by passing in the --config_schema <file>
argument to phyto-arm
. A few rules on how validation is completed:
#optional
are not required.The config validation script includes a separate set of unit tests that can be run with
python3 scripts/config_validation.test.py
When running in a container, ensure the config YAML refers to the Docker host's IP address of 172.17.0.1
instead of using localhost
when trying to access host services, such as in the config for /gps/host
and /ifcb/address
.
The entries in the config file are loaded into the ROS Parameter Server. Some parameters can be dynamically changed while the nodes are running:
$ source devel/setup.bash
$ rosparam get /conductor/schedule/every
60
$ rosparam set /profiler/data_topic /ctd/aml/port3/chloroblue
Be sure to make the corresponding edits to the config file to persist changes beyond the current session.
Use picocom
to to talk to the appropriate serial device. Press Enter first to interrupt any current logging and get a prompt. Run the commands below in picocom to configure the AML output correctly.
$ picocom -b 115200 /dev/ttyS3
> set derive depth y
> set scan dep
> set derive salc y
> set scan sal
> set derive density y
> set scan den
> set scan raw
> mmonitor
...
Press Ctrl-A, Ctrl-X to exit picocom
while the device is logging.
The time must be set to UTC. To sync the clock, you can use:
(echo; echo "set fulltime $(date -u "+%Y-%m-%d %H:%M:%S")") \
| picocom -b 115200 /dev/ttyS3
The current setup of the Chanos arm assumes the RBR CTD data is being transmitted to the host over UDP. In our setup, a proxy is running on a Raspberry Pi Zero which has a serial connection to the RBR, and forwards the CTD messages over UDP to the host.
The proxying script used is provided in rbr_relay.sh
, a systemd service equivalent is also provided in rbr_relay.service
.
The udp_to_ros.py
node is used to read the proxied UDP packets and publish them on a ROS topic to be used by PhytO-ARM.
GPS tracking is provided via gpsd.
sudo apt install -y gpsd gpsd-clients
On Ubuntu, edit /etc/default/gpsd
to configure the GPS device or network source. For example, to listen for UDP packets broadcast on the local network:
DEVICES="udp://192.168.13.255:22335"
Monitor that GPS updates are being received using gpsmon
.
When running in a container, the gpsd service on the host needs to be modified to accept inbound connections from the container. Use systemctl edit gpsd.socket
to create an override file:
# Allow clients to connect to gpsd from Docker.
# Based on https://stackoverflow.com/q/42240757
[Socket]
ListenStream=
ListenStream=/var/run/gpsd.sock
ListenStream=0.0.0.0:2947
Additional Ubuntu configuration that may be necessary to ensure GPS stays powered on and available.
Adding to the IP above, change /etc/default/gpsd
to:
# Default settings for the gpsd init script and the hotplug wrapper.
# Start the gpsd daemon automatically at boot time
START_DAEMON="true"
# Use USB hotplugging to add new USB devices automatically to the daemon
USBAUTO="false"
# Devices gpsd should collect to at boot time.
# They need to be read/writeable, either by user gpsd or the group dialout.
DEVICES="udp://192.168.13.255:22335"
# Other options you want to pass to gpsd
GPSD_OPTIONS=""
Then use systemctl edit gpsd.service
(may have to invoke sudo) and set the override file to:
# Always start GPSD, even if no one is connected to the socket
[Install]
WantedBy=multi-user.target
example.yaml
picocom
nc -lup 12345
gpsd
installed and runninglaunch_args/ifcb_winch
set to true if used, false otherwiselaunch_args/chanos_winch
set to true if used, false otherwiseTo develop in PhytO-ARM, it is recommended to use a development container with the provided development container configuration file. This can be done locally or on a remote machine (e.g. an IFCB).
.devcontainer/devcontainer.json
configuration file. You may need to comment out any lines which depend on hardware (e.g., /dev/ttyS3
) or folders (e.g. /home/ifcb
) not present on your development machine.devcontainer.json
configuration and prompt you to reopen the project as a development container.The development container allows you to develop and test from an environment identical to that used in production. If developing locally, launch arms with mocked hardware (see Running section above).
The .git
repository folder is also mounted in the development container, making it possible to pull/commit/push as needed from within the container.
PhytO-ARM has evolved to handle multiple mission types and deployment configurations. Deployments so far include single-winch on a stationary platform, winchless on an autonomous surface vehicle, and multi-winch on a stationary platform.
PhytO-ARM has been modularized to support any configuration of winch and scientific payload. Scientific payloads are isolated into distinct "arms", each of which may have a winch providing movement for that arm. All arms should implement ArmBase
, an interface class which handles winch movement logic and coordinates movement between arms by communicating with a central lock_manager
node.
lock_manager
is primarily to limit how many winches can move concurrently, as configured by the parameter /lock_manager/max_moving_winches
. This is primarily to accommodate missions which may require inter-arm coordination, or to prevent overdraw of power on battery-powered setups.
PhytO-ARM comes with two arm implementations included, see arm_ifcb.py
and arm_chanos.py
in the phyto_arm
package. Any number of arms is supported however, as illustrated below. Each box represents a set of nodes with an independent launch file.
graph TD
subgraph PhytO-ARM
subgraph Main
cameras(Cameras)
lm(Lock Manager)
gps(GPS)
end
subgraph Arm1
w1(Winch1)
subgraph Payload1
ifcb(IFCB)
c1(AML CTD)
end
end
subgraph Arm2
w2(Winch2)
subgraph Payload2
chanos(Chanos)
c2(RBR CTD)
end
end
subgraph ArmN
w3(WinchN)
subgraph PayloadN
unknown(?)
c4(? CTD)
end
end
end
lm <--> w1
lm <--> w2
lm <--> w3
Each arm implementing ArmBase
must override get_next_task
, which is the basis of an event loop where all decisions are made on which task to perform next. get_next_task
is passed the previous task as context, and should return a new task containing movement instructions and a callback that will be executed when the movement completes.
graph
subgraph Arm Execution Loop
subgraph Main
lm(Lock Manager)
end
subgraph ArmBase
loop
winch(send_winch_goal)
snt(start_next_task)
subgraph Arm Implementation
gnt(get_next_task)
cb(Task callback)
end
end
end
loop -- Await clearance --> lm
lm -- Get task --> gnt
gnt -- Move to depth --> winch
gnt -. Skip if no winch .-> cb
winch -- Execute --> cb
cb --> snt
snt --> loop
In all deployments thus far, we have used the IFCB as the primary host to run PhytO-ARM and coordinate movements and sampling. The hardware architecture diagram below illustrates this.
graph
subgraph Platform
subgraph IFCB Arm
motor1(JVL motor)
aml(AML CTD)
subgraph IFCB
acquire{IFCB Acquire}
pa{PhytO-ARM}
end
end
subgraph Chanos Arm
pi(Rapsberry Pi)
motor2(JVL Motor)
chanos(Chanos)
rbr(RBR CTD)
end
end
subgraph Key
hw(Hardware)
sw{Software}
end
pa -- Drive (TCP) --> motor1
pa -- Drive (TCP) --> motor2
aml -- Serial --> pa
rbr -- Serial --> pi
pi -- UDP --> pa
chanos -- Timed sampling -->chanos
pa -- Trigger (TCP) --> acquire
Note that ROS path conventions are used, such that paths beginning with /
are absolute, paths beginning with ~
start at the node, and paths without a prefix are relative to their namespace. Global namespace is used for main nodes, while arm_ifcb
and arm_chanos
namespaces are used for the IFCB and Chanos arms, respectively.
These nodes implement the core PhytO-ARM "algorithm" for sampling and are specific to the design of the platform.
lock_manager
: Winch traffic control center, limits number of concurrent winch movements.
~acquire
for arms to request permission to move their winches~release
for arms to release permission to move~check
for arms to check whether they might have a dangling lock (e.g. after a crash or user stop)arm_ifcb
: Orchestrates IFCB payload behaviors
ifcb_runner/sample
to run IFCB sampling actionsprofiler
for profile data, especially phy peakwinch/move_to_depth/result
to determine the success of a transitlock_manager/acquire
for getting clearance to move a winch. Some configurations may not permit simultaneous winch movementslock_manager/release
to inform the semaphore that movement has completedlock_manager/check
check for dangling locksifcb_runner
: Provides one high-level action for running IFCB samples
/ifcb/in
for IFCB status messages~state
current state for debugging purposes~sample/goal
action server for running IFCB sampling routines~sample/feedback
sampling action feedback~sample/result
sampling action resultmock_ifcb_runner
: Mock version of ifcb_runner
~state
current state for debugging purposes~sample/goal
action server for running mock sampling routines~sample/feedback
sampling action feedback~sample/result
sampling action resultarm_chanos
: Orchestrates Chanos sensor payload behaviors
winch/move_to_depth/result
to determine the success of a transitlock_manager/acquire
for getting clearance to move a winchlock_manager/release
to inform the semaphore that movement has completedlock_manager/check
to check for dangling locksprofiler
: Creates profiles of CTD data during a cast
~
with resampled profile data~downcast
as above, but only downcasts~downcast
as above, but only upcastsweb
: Web API for attaching metadata to IFCB bins
/gps/fix
for GPS fixeswinch
: Controls depth using the motor
ctd/depth
for monitoring depthmotor/motion
for monitoring motor statewinch/move_to_depth/goal
for setting the goal depthwinch/move_to_depth/cancel
for canceling the current goalwinch/move_to_depth/feedback
with progress updateswinch/move_to_depth/status
with the status of the current goalwinch/move_to_depth/result
with the result of the goalmock_winch_node
: Mock version of winch
winch/move_to_depth/feedback
with progress updateswinch/move_to_depth/status
with the status of the current goalwinch/move_to_depth/result
with the result of the goalThese nodes perform lower-level interactions with hardware components. These nodes are designed to be portable to future projects.
camera
: Video stream
~image/compressed
with the compressed video feedctd
: Driver for the AML or RBR maestro3 CTD
ctd_comms/in
for receiving data from CTD.~ctd
with conductivity, temperature, and pressure data~depth
with depth and pressure data~aml/derive/xyz
with derived values (for AML only)~aml/port#/xyz
with measured values (for AML only)ctd_comms
: Bridge for CTD serial communications
ctd_comms/in
for messages received from the CTD's serial portctd_comms/out
for messages sent to the CTD's serial portudp_to_ros
: Translates UDP messages to ROS topics. Used for receiving proxied RBR data.
udp_stream
a continuous stream of all received UDP messages.gps
: Provides GPS fixes to ROS from gpsd
/gps/fix
and /gps/extended_fix
ifcb
: Bridge for the IFCB websocket API
~in
for messages received from the IFCB~out
for messages sent to the IFCB~image
for full-frame images~roi/image
for detected ROI images only~roi/image/raw
republished ROI as raw (if classifier enabled)~roi/markers
with rectangular bounds of ROIs~command
to send a message to the IFCB~routine
to send a routine to the IFCBmotor
: Driver for the JVL motor
~electrical
with electrical registers~error
with error registers~motion
with motion registers~set_position
to set the motor's position~set_position_envelope
to set position limits~set_velocity
to set the motor's velocity~stop
to stop the motormock_ctd
: Mock version of a CTD publisher
~ctd
with conductivity, temperature, and pressure data~depth
with depth and pressure dataThese nodes provide functionality for recording data and connecting to other tools like Foxglove Studio.
rosbag_record
: Records ROS traffic to ROS bag files
/begin_write
when a new bag file is createdrosbridge_websocket
: Rosbridge server for Foxglove Studio
rosout
: Logging mechanism
/rosout
for log messages from each node/rosout_agg
with copies of all log messages
# First comment
val1 = 'some value'
val2 = 'next value'
A style guide has not yet been rigorously defined, this section will expand as new policies are adopted.
## Observing with Foxglove Studio
[Foxglove Studio][] is the recommended software for monitoring the PhytO-ARM system. To connect to a live system, use the "ROS Bridge WebSocket" connection type.
The ROS Bridge node uses TCP port 9090. Ensure this port is forwarded in your router settings *for trusted clients only*, or use SSH port forwarding:
$ ssh -NL 9090:localhost:9090 <PhytO-ARM host address>
In Studio, the connection address is `ws://<PhytO-ARM host address>:9090` if connecting directly or `ws://localhost:9090` if using the SSH tunnel.
[Foxglove Studio]: https://foxglove.dev
**Tip:** For the most accurate time series plots, configure plot lines to use a message's `header.stamp` rather than its receive time, which is affected by network latency.
## Data Logs
Data is logged to the ROS bag format using the `rosbag record` command. Bag files can also be viewed with Foxglove Studio.
Almost all internal ROS traffic is logged, with the exception of the camera feed (due to storage requirements) and ROS service calls (due to technical limtations).
The config file controls where the bag files are written:
launch_args:
rosbag_prefix: /mnt/data/rosbags/phyto-arm_hades
This results in files in the `/mnt/data/rosbags/` directory with names like `phyto-arm_hades_2022-04-01-14-47-11_0.bag`. The name includes the timestamp of when PhytO-ARM was started, plus a numeric counter. A new bag is created, incrementing the counter, when a configured size limit is reached.
A bag file with the suffix `.active` is currently being written, or was left in an incomplete state by a previous session that was terminated abruptly. Such files can sometimes be repaired.