Fly misdetection issues: fly in cotton and micromovements

[antortjim]

Dear Gilestrolab, I am Antonio, a technician at the Liu Lab @ KU Leuven. Thank you for producing the Ethoscope platform. We are using them and we are learning lots with them :). I would like to report a couple of issues we are facing.

I have made a Google doc with them in pdf format because since it has several figures, it becomes cumbersome to read in an issue page. I would really appreciate if you could have a look. https://docs.google.com/document/d/14pOcsv-8JEvTPEs99kLh23z_F4BiTK2JkVG-OTcCm6E/edit?usp=sharing Here is the pdf: Ethoscopes report.pdf

TL;DR The gist of the issue is is that we are detecting many movements in our flies that lead to the production of flat sleep traces at 0 i.e. our flies look like they don't sleep at all in the whole dark phase. We were wondering if you or anyone else has encountered similar problems and how you solved them. Take a look at the Google doc for more details and our proposed solutions.

1. Some of these movements are likely to be artifacts because the distance is too big to be produced in a single frame. See how in Figure A ROI 16 has detected the fly on the black cotton. We believe that misdetections like this can happen sporadically on only one frame at a time, followed by correct detections creating a "spike". FIgure B displays spikes like this on ROIs 1,6 or 14 for example.

A Annotated frame displaying misdetected fly on ROI 16

B Coordinates on the x axis of 20 flies over 2 LD cycles

2. Most of them however, could be micromovements, as shown in the gif below (the gif in the google doc has more frames due to file size)

C 1 min animation at 0.2 FPS displaying possible micromovements of the flies. The distance they produce in the adaptive bg tracker is more than > 0.003 (the default velocity correction coefficient), so they are by default flagged as movement. However, a coefficient of 0.01 would flag this as asleep.

PS I accidentally submitted a preliminary version of the issue a few mins ago. Sorry for that! 0:-)

Thank you very much for your time and for making the ethoscopes free and libre!!!! I am learning so much Best, Antonio

[pepelisu]

Hi Antonio, Many thanks for this exhaustive report. From the hardware side I can tell you that another light_box has been develop with the aim to improve the light conditions. We use now a custom made guide panel with the LED on one side. First tests show a promising improvement. You can find the onshape file here: https://cad.onshape.com/documents/98512e7af7f4bb54b04f14ad/w/45b95c21c6563124c1461db2/e/a742f8bda55848f2cf1a9ce2 The hole thing, once printed and the acrylic guide panel added should look like this: You can use 3 or 6 led strip, the design can be improved to use a dissipator for the leds or a longer space between flies and leds. If you tests it please let us know what do you think.

[antortjim]

Dear @pepelisu

Thank you for your prompt reply and your insights regarding the lightbox! I think it's a great improvement. For now, we will not print light boxes with the new version. Instead, we will replicate your design by placing 2 strips of 3 LEDs each on the short sides of the light box. We will use no dedicated diffuser other than the arena itself.

We connect the 2 strips in series, so it looks like this, using the old light box

I took a picture of how the illumination looks like on one ethoscope in the D phase with A) the old design (6 LEDs on the led holder) vs B the new design (the picture above).

A B

The improvement is clear on the 4 central ROIs (5,6,15,16). However it seems a bit darker. I am running a sleep deprivation experiment with this new setup now. I will let you know if the results!

PS I think on the other hand, more sophisticated image processing methods could be implemented, so the system is more robust to the lighting conditions? As long as the RPi can take it... or maybe we can try the Jetson Nano (<100 Euros) in future versions?

Thank you once again!! Antonio

[antortjim]

Dear Gilestro lab

I have a follow up on tracking issues. We replaced the LED strips on all our ethoscopes as shown above. The amount of issues has diminished, but we still encounter tracking issues. We observe a strange behavior by which the tracker places flies closer and closer to the left end of the tube (closer to 0). The resulting shape of the x trace is a 'right triangle' shape. We firmly believe this is a mistracking as a fly is highly unlikely to display such a strange moving pattern. Below you can see the x coordinate trace for all flies, and a zoom-in between hours 41 and 42 for flies 1 and 16. For fly 16 for example, I would say the fly did not move at all from around 0.62, but the tracker registers this increasingly bigger spikes towards 0, trumping the motion_detector in R.

PS This issue is the reason why I posted issue #104

[ggilestro]

I agree that this shape looks artifactual. It also appears to be synced in time across flies in the same tube. To me, this may suggest some external light disturbance coming from the incubator such as visible LED flickering in the incubator or someone opening the door. Have you considered these trivial possibilities first? At any rate, my recommendation is the same as the other issue you opened: record a 24 or 48 hours long video then do offline tracking of the same. Please let me know.

[antortjim]

I ran an ethoscope with modifications in the code to save the intermediate steps of the processing and the result. As you mentioned in #104, doing this greatly slows down tracking from ~10FPS to ~0.3 FPS. I had already tried the video but I didn't get any result. Fortunately, in this case, I was able to replicate the error:

and I was able to playback the processing steps of the tracker, as seen in the gif below. The spacer in the middle is placed there to improve the contact between the tubes and the shafts of the motors in the optomotor. I have a bug in the code that lost ROI 1 and shifted everything one position, sorry about that :innocent: Its frames and many more can be found here: https://mega.nz/#F!8hFywaTR!IRVA8cwAZU1L4iLQokSDMQ

Indeed, the flickering of the IR light is very significant and I think it's driving the tracking errors. I can see that the flickering is translated into the edges of the ROIs being sometimes included in the foreground mask (black&white figure). More concretely, spurious pixels above the threshold of the mask (20/255) seem to lie around the edge of the spacer. The edge of the fly food in the tube also has this effect but less pronounced I would say. This seems to happen only when the fly has not moved for a while. Since the fly is quiet, it produces no changes in the foreground and thus smaller changes in the background caused by light flickering become big enough to confuse the tracker. But if the fly moves, even if the light flickers, the signal produced by the fly offsets this noise and the tracker succeeds.

Have you ever faced such an issue before? Did you redesign the lightbox for a similar reason?

A list of things I think can be done to attempt to solve this.

Hardware

1. Make lighting more robust. Maybe the IR strips are not reliable, or maybe the RPi cannot provide a stable energy supply. In that case, a more robust PSU would be the solution.

Software

1. Increase the foreground masking threshold, so it becomes more difficult for spurious pixels to be flagged as foreground. https://github.com/gilestrolab/ethoscope/blob/f692e7c973b54e1aa1580f198aa836fb7faa99b6/src/ethoscope/trackers/adaptive_bg_tracker.py#L383

2. Define a threshold for feasible velocities between frames (taking into account the time between them), so huge velocities i.e. 'teleportations across the tube' due to mistracking is filtered out.

3. Make sure the x,y coordinates that are saved are actually the center of mass of the ellipse and not the center of mass of the foreground-background (I can see it's a todo) https://github.com/gilestrolab/ethoscope/blob/f692e7c973b54e1aa1580f198aa836fb7faa99b6/src/ethoscope/trackers/adaptive_bg_tracker.py#L468 This is achievable if the ellipse is drawn on black canvas (i.e. np.zeros_like(self._buff_fg) and not the self._buff_fg object, which contains the spurious pixels)

4. An ideal solution at the software level would be to upgrade the tracking algorithm to make it more robust, maybe with a CNN but probably it would be too much for the RPi to take.

Thank you for your help beforehand! Antonio

[antortjim]

One sidenote! I checked that the slowdown effect is due to me producing the figures shown in the gif. If I just change the period of ImgToMySQLHelper to 0.1, i.e. I save all snapshots but I don't save intermediate steps, the resulting FPS is ~3, which I think is fine (way better than 0.3).

[antortjim]

I created this gif with the snapshots I saved during the debugging run. Every snapshot represents 1-3 seconds. It's clear that the overall light changes a lot between frame, but I don't know the cause yet. But at least this confirms the flickering is not an artifact of the algorithm i.e. it could be due to malfunctioning of either the camera, the IR strip or the PSU (the RPi and in turn an Anker PowerBank).

[ggilestro]

I think there are two possibilities here:

1) a hardware issue. Perhaps the LEDs are defective or your system does not provide enough current. Make sure you use a power supply that is powerful enough (at least 2A) and maybe try powering the IR LEDs not through the PI. We use 12V rated IR LEDs powered with the 5V coming from the PI

2) a software issue. The picamera library got some recent changes to solve compatibility issues with the PI4. This created some sporadic issues with the piNOIR camera. You may want to try disabling the auto white balance in the python code. If you do, you need at the same time to adjust gains otherwise the image will be black. Check the API and the recipes.

If you see a purple image under white light, this may be due to the software issue. I'd rather be oriented to think it is a hardware one, though. Please let me know how this goes.

[antortjim]

Dear @ggilestro

Thanks for your response. We are now encountering new artifacts in our data, as seen on this plot which shows the x column of the db file for a single fly in the y axis and time on the x axis. The plot is showing 1/10 of an hour (6 mins) and 1/10 of the ROI.

Small fluctuations of the position of the fly are detected when using the python3.7 branch. They are very small (< 1/100 the width of the tube) but the detected corrected velocity becomes greater than 0.003 (the default coefficient in the motion_detector function), which thus causes the data to be annotated as awake, even though the data is not moving. We do not want to increase the correction coefficient for consistency with previous experiments, but I have instead tried:

linear_filter <- function(x, kernel) {
  result <- as.integer(filter(x, kernel))
  result[is.na(result)] <- x[is.na(result)]
  return(result) 
}

dt_raw <- load_ethoscope(
  metadata = metadata,
  result_dir = ETHOSCOPE_RESULTS_FOLDER
)

length_kernel <- 30
linear_kernel <- rep(1/length_kernel, length_kernel)

dt_raw[,xy_dist_log10x1000 := as.integer(linear_filter(x = xy_dist_log10x1000, kernel = linear_kernel)), by = 'id']

dt <- sleep_annotation(dt_raw, 0.003)

Which smoothens the data and fixes the issue of these small fluctuations. I am not sure how correct this procedure is though. Have you experienced a similar problem when running the python 3 version of the code?

EDIT These fluctuations are equivalent to 1 pixel i.e. the tracker is detecting the fly 'fluctuates' between two contiguous pixels. This happens mostly in the night.

[antortjim]

Following up on this, I just checked the 6000th datapoint of the annotated dataset for region id 1 in this dataset. You can see here that even though x and y don't change in many contiguous rows, the distance indeed changes and sometimes becomes higher than log10(0.003) * 1000, which is the default threshold of the max_velocity_detector function -> https://github.com/rethomics/sleepr/blob/cb4efa653100d215f1cb31ad7da4a505d266dd3d/R/motion_detectors.R#L32

*See for example the 8th and 9th rows. x and y do not change but the distance becomes greater than -2522.879 (log10(0.003) 1000). I don't understand how is this possible since the distance only relies on x, y and the width of the ROI** https://github.com/gilestrolab/ethoscope/blob/0567585158ff3a51ea998faa97a8d60285711163/src/ethoscope/trackers/adaptive_bg_tracker.py#L455 which stays between 465-466 pixels.

The xy_dist_log10x1000_manual column is manually created by me, following the formula of the Euclidean distance and assuming x and y are already normalized with w_im (as x and y oscillate between 0 and 1 on this table).

w_im <- 465
points_list <- as.list(dt_raw_point[, .(x, y)])
points_list <- lapply(1:length(points_list[["x"]]), function(i) c(points_list$x[[i]], points_list$y[[i]]))

xy_dist_log10x1000_manual <- c()
for (i in 2:length(points_list)) {
  difference_between_points <- (points_list[[i]] - points_list[[i-1]])
  # print(difference_between_points)
  res <- sqrt(sum(difference_between_points**2))
  res <- round(log10(1/w_im + res) * 1000)
  xy_dist_log10x1000_manual[i] <- res
}

As I understand max_velocity_detector, it:

• takes this xy_dist_log10x1000
• transforms it back into a distance,
• computes the time difference between frames
• divides distance with the time difference to get the velocity
• corrects the velocity, which means multiplying again by the time difference (thus getting the distance again), and then divides by the velocity correction coefficient (0.003)
• selects the maximum over a block of a given time (by default 10 seconds)
• if the maximum velocity is greater than 1 i.e. the max distance > 0.003, the block of 10 seconds is annotated as moving.

Thank you very much for your time.

[qgeissmann]

Hi there, thanks for reporting your issue!

See for example the 8th and 9th rows. x and y do not change but the distance becomes greater than -2522.879 (log10(0.003) * 1000). I don't understand how is this possible since the distance only relies on x, y and the width of the ROI

Keep in mind that, internally (before the saving to the db), x and y are subpixel accurate as they are computed from the centre of mass of the greyscale detected object (y,x = ndimage.measurements.center_of_mass(self._buff_fg_backup)). However, to save space in the database, y and x are saved as integers (x_var = XPosVariable(int(round(x)))). Therefore, in the DB, xy_dist_log10x1000 can change even when x and y don't.

You said the problem is in the python3 branch... I wonder if this could be due the default behaviour of a division of ints in python2 vs 3 (/ vs //)?

[antortjim]

Hi @qgeissmann thanks for your reply

Indeed, I did not think of the difference between /and // in python2 and python3. However, a glance at the line that computes the distance and in general the function it is in tells me that should not be a problem

https://github.com/gilestrolab/ethoscope/blob/0567585158ff3a51ea998faa97a8d60285711163/src/ethoscope/trackers/adaptive_bg_tracker.py#L470

as the code enforces the division between float types. I don't see any other place where a conflict between // and / could arise.

Could you list the specifications of the ethoscopes you are using with your python3.7 branch? That would help us a lot! We are using:

• python 3.7.4
• picamera 1.13
• scipy 1.3.1
• opencv 4.1.1
• numpy 1.17.2
• mysql_connector_python-8.0.18
• RaspberryPi 3B
• Pi camera V2.1
• add anything you may think could be a culprit

this is the output of uname -a for our ETHOSCOPE_001

Linux ETHOSCOPE001 4.19.75-2-ARCH #1 SMP PREEMPT Tue Oct 1 00:34:07 UTC 2019 armv7l GNU/Linux

The software is that provided by your image from November available in this link: https://imperialcollegelondon.box.com/shared/static/hins96jdnzfaqd33s8y0o01oitejos17.zip

Our code is that in your python3.7 branch with 2 modifications: I uncommented this line https://gitlab.com/antortjim/ethoscope/blob/fsl_python3/src/scripts/device_server.py#L447 because the ethoscope 'kept waiting for a network'

And I added a method to the ControlThread class to handle an error with the camera that causes it to be 'out of resources' i.e. taken by a previous process.

Both changes are listed on this commit: https://gitlab.com/antortjim/ethoscope/commit/c0c95fad4b9dc1ccea3e34a3b58a7507e3b66c1c

This is the picamera error. It eventually causes the frame grabber to timeout, which is the error we get in the GUI.

Jan 21 13:40:50 ETHOSCOPE013 python3[611]: picamera.exc.PiCameraMMALError: Failed to enable connection: Out of resources
Jan 21 13:40:50 ETHOSCOPE013 python3[611]:     raise PiCameraMMALError(status, prefix)
Jan 21 13:40:50 ETHOSCOPE013 python3[611]:   File "/usr/lib/python3.7/site-packages/picamera/exc.py", line 184, in mmal_check
Jan 21 13:40:50 ETHOSCOPE013 python3[611]:     prefix="Failed to enable connection")
Jan 21 13:40:50 ETHOSCOPE013 python3[611]:   File "/usr/lib/python3.7/site-packages/picamera/mmalobj.py", line 2212, in enable
Jan 21 13:40:50 ETHOSCOPE013 python3[611]:     self.renderer.inputs[0].connect(source).enable()
Jan 21 13:40:50 ETHOSCOPE013 python3[611]:   File "/usr/lib/python3.7/site-packages/picamera/renderers.py", line 558, in __init__
Jan 21 13:40:50 ETHOSCOPE013 python3[611]:     self, self._camera.outputs[self.CAMERA_PREVIEW_PORT])
Jan 21 13:40:50 ETHOSCOPE013 python3[611]:   File "/usr/lib/python3.7/site-packages/picamera/camera.py", line 513, in _init_preview
Jan 21 13:40:50 ETHOSCOPE013 python3[611]:     self._init_preview()
Jan 21 13:40:50 ETHOSCOPE013 python3[611]:   File "/usr/lib/python3.7/site-packages/picamera/camera.py", line 433, in __init__
Jan 21 13:40:50 ETHOSCOPE013 python3[611]:     capture = PiCamera()
Jan 21 13:40:50 ETHOSCOPE013 python3[611]:   File "/opt/ethoscope-device/src/ethoscope/hardware/input/cameras.py", line 353, in run
Jan 21 13:40:50 ETHOSCOPE013 python3[611]:     self.run()
Jan 21 13:40:50 ETHOSCOPE013 python3[611]:   File "/usr/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
Jan 21 13:40:50 ETHOSCOPE013 python3[611]: Traceback (most recent call last):

Again thanks for your help!

[ggilestro]

Hi Antonio

Do I understand correctly that you are using a version that was forked from GitHub to GitLab in November? I made several hundred commits in the past 4 months so it is going to be difficult for me to reproduce this or future problems unless we converge again!

Having said this: I do not remember having touched anything in the tracking algorithm when migrating from python 2 to python 3. Given you see one-pixel fluctuations I suppose this may be either a consequence of rounding a value that is close to 0.5 (stochastically floored or ceiled) or again due to some lighting artifact (the fact you see this only at night makes me opt for the latter hypothesis).

If it is an issue with rounding, it may be related to the intrinsic limitations of floating-point arithmetic.

We could fix that by fixing the rounding procedure to one direction.

How frequent is the artifact?

[antortjim]

Hi @ggilestro Thank you for looking into it :)

Yes, I am using a forked version, but I merge your changes on my fork when I see a significant milestone is achieved. We don't update very frequently because we want to be conservative. The last commit I have from you is

commit 0567585158ff3a51ea998faa97a8d60285711163
Author: ggilestro <giorgio@gilest.ro>
Date:   Fri Nov 22 17:58:41 2019 +0000

    Added back an internal function that was removed by mistake. This handles the special occurrence of the img table

Indeed, we also thought these two factors could be contributing to the problem. However, at the turn of the new year I updated our ethoscopes and also those used in the Verstreken la, which were fabricated independently. And then both labs started experiencing the problem very frequently, to the point of

preventing us from doing more experiments. The traces indicated that the flies were barely sleeping at all in the night. The fact that all their ethoscopes were exhibiting the same problem at the same time, made me think a buggy IR strip is unlikely to be the reason.

A look at the detected max_velocityes showed that these were somehow systematically getting shifted upwards, just enough to overcome the velocity threshold of 0.003. You can see that more than half of the background ‘non-moving’ max_velocities are > 1 in the dark phase, indicating that the non corrected velocity was > 0.003. This did not happen with ethoscopes running the python2 version, and it explains the observed traces that indicate lack of sleep in the night.

We came up with an easy solution, which was adding a high pass filter (only let through > 730 nm) to the RPi camera, so the lighting conditions in L and D were identical in terms of what the camera ‘sees’. We cropped this filter https://www.robertwhite.co.uk/lee-filters-100mm-system-polyester-87-infrared-filter.html into tiny pieces and mounted it on top of the camera, like this

This has the effect of making the background velocities similar again in L and D i.e. no more upwards shift. We combined this with a change in the software in Python and R, so the ethoscope saves x and y with float precision in the SQL database and R uses this offline to recompute the distance, as indicated here R: https://github.com/rethomics/sleepr/issues/7#issuecomment-588143906 Python:

• Float variables definition https://gitlab.com/antortjim/ethoscope/-/blob/fsl_python3_precise/src/ethoscope/core/variables.py#L136
• Usage in the adaptive_bg_tracker class: https://gitlab.com/antortjim/ethoscope/-/blob/fsl_python3_precise/src/ethoscope/trackers/adaptive_bg_tracker.py#L492

It is working fine for us like this now, even though the analysis exactly as we did before i.e. using the distance stored in the sqlite3 file and 0.003 as threshold still shows this problem. Using the sqlite3 distance and 0.006 as threshold rescues the data in some cases, I don’t know what fraction. I explain that by the fact that the distance in R is always gonna be 1/roi_width less than in Python (I have another workaround for the log(0) error), so it is more easily scored as non movement. We will run calibration experiments to verify that these modified settings are correct, similar to what you did before you found 0.003 was the best threshold. For us it seems it is not, and we haven’t found the reason yet. Hope I was clear enough!

Best regards, Antonio

[ggilestro]

Hi @antortjim

thank you again for reporting bugs and changes in such a careful manner. This is extremely useful. After looking into this, I think this is actually a bug that affects ethoscopes that are using the newest camera version (pinoir v2) on python3.

This probably all originated from this external issue: https://github.com/raspberrypi/firmware/issues/1167

One thing we can try, is to turn autowhitebalance off on these lines

https://github.com/gilestrolab/ethoscope/blob/9ae58e8fdabeb694c1d4968baa921c534a3e140f/src/ethoscope/hardware/input/cameras.py#L356-L362

Would you like to give it a try and let me know?

[antortjim]

Hi @ggilestro You are welcome, I am very happy I can be of any help. Do you actually use pinoir v1 on your ethoscopes? In that case it would definitely explain our problems. I will definitely give a try to this atuowhitebalance thing. I must say that while the tracking and recording live feed look normal, the stream feed looks purplish, similar to

I will let you know ASAP

[antortjim]

This is how our frames look on an ethoscope in the L phase without the IR filter I posted above if I don't convert to gray

and the D phase

placing the IR filter makes the L phase image look like in D. The image is purple in the D phase, but I don't think that's hardware malfunction, just the nature of IR light, right?

I tried setting

capture.awb_mode = 'off' 

and indeed the output is a black square. I am not setting custom gains, as explained in the links you provided, due to lack of time (for now).

On a RPi 3B+ with camera Noir v2.1 running python-3.7.4-2 picamera 1.13

I got the above pictures running the script below and accessing http://192.169.123.61:9000/static/tmp/last_img.png (that's the IP of ETHOSCOPE_011). Streaming by clicking on the button available in the Python3 GUI I get a similar picture.

import picamera
from picamera.array import PiRGBArray
import logging
import cv2

target_fps=20
target_resolution=(1280, 960)

with picamera.PiCamera() as capture:

    logging.warning(capture)

    capture.resolution = target_resolution
    capture.awb_mode = 'auto'
    #capture.awb_mode = 'off'

    capture.framerate = target_fps
    raw_capture = PiRGBArray(capture, size = target_resolution)
    for frame in capture.capture_continuous(raw_capture, format="bgr", use_video_port=True):

        try:
            raw_capture.truncate(0)
            #out = cv2.cvtColor(frame.array, cv2.COLOR_BGR2GRAY)
            out = frame.array
            cv2.imwrite('/tmp/last_img.png', out)

        except Exception as e:
            logging.warning(e)
            break

[antortjim]

Just remembered I did not report on our solution to the problems here:

• We put the IR filter on the camera so data from L and D phase have similar properties.
• We power our IR strip with an external PSU to avoid flickering effects caused by unstable current provided by the RPi. I don't know what causes this unstability, but an external PSU is a way around the issue anyway/ It means though the ethoscope now needs two cables: one to power the RPi and this new cable to power the strip.
• We use a different velocity correction coef. Apparently the switch to Python 3 caused something in the algorithm that processes the frames and yields arrays in Python to change. The Python code and the setup (ethoscope code and lighting) are the same but the output (np.array) is not. @ggilestro mentioned adding this line

  awb_auto_is_greyworld=1

  to /boot/config.txt reverts the algorithm back to the same as in Python2. That is the same solution they found here https://github.com/raspberrypi/firmware/issues/1167 That should also work