Pupil labs eye cameras - Githubissues

so, long story short, re: pupil camera software - three options ranging from easiest/least-reliable - harder/more-reliable

Level 0 - Download their standard software here: https://github.com/pupil-labs/pupil/releases/tag/v3.5 and try to connect to the eye camera via Pupil Capture (however it might take some finagling to get it to connect properly)

Level 1 - run their code from source ( https://github.com/pupil-labs/pupil?tab=readme-ov-file#installing-dependencies-and-code )

Level 2 - write our own custom scripts using a combination of their pupil-core API/codebase, skellycam code, etc

again, we can talk about it in person and come up with a plan to move us through those levels in a way that will allow early progress without getting locked into local tech maxima

(copied from scalloncu's discord message: https://discord.com/channels/1191349041267752980/1191349041821384747/1259912772829974569)

I grabbed some dimensions off of the cameras. This file can be found in Fusion at /BenandAlonHeadfix/Cameraholder/PupilLabCamera. I don't know what dim are important to you now, so I didn't include any here, but sceenshot below. Also, here are some notes/thoughts/ramblings as I look at this:

Each camera has three boards, separated by flex. I will call them LED board, Camera board, and Data board here. a. The two cameras feed into one cable which goes into presumably another board (to send over USBC). It is in some 3D printed housing. I would like to get in there, but it doesn’t readily disassemble, so I will pause b. There is opportunity to fold the flex and minimize space. But we should consider where we want the boards relative to the animal. It may be good to put boards further away (posteriorly) rather than just minimize space.
There are two LEDs attached to each camera. This means there is enough power to supply LEDs of our choosing directly from this board. a. There is a resistor connected to these, presumably to correctly use LEDs, and set the brightness. But I don’t think brightness is something you should think about (if you see it, AI can see it)
The camera appears to have a removable lens. FYI
The camera board each has four wires. The connector is a large molex, this could be reduced and wires could be soldered directly on the board. a. Also, I would imagine two of the wires are just power, and the other two are data. Putting the Camera boards next to each other, we could further minimize wiring b. The wires are wrapped in some sort of electrostatic shielding. This makes them beefy. I wonder if off the shelf coaxial cabling would achieve the same results in a much smaller package

Some thoughts:

This file can be found in Fusion at /BenandAlonHeadfix/Cameraholder/PupilLabCamera. I don't know what dim are important to you now, so I didn't include any here, but sceenshot below.

This is fine for now, but we should find some FOSS-friendly way to backup/store the relevant base files in whatever (non-proprietary) format we like.

Each camera has three boards, separated by flex. I will call them LED board, Camera board, and Data board here.

For the record, the 'camera board' holds the lens/sensor assembly and the 'Data board' is probably better thought of as the 'processing board'

a. The two cameras feed into one cable which goes into presumably another board (to send over USBC). It is in some 3D printed housing. I would like to get in there, but it doesn’t readily disassemble, so I will pause

The data leaves the camera via a USB-C connection (the 4-pin connector) - the place they come togther is a USB-C hub

b. There is opportunity to fold the flex and minimize space. But we should consider where we want the boards relative to the animal. It may be good to put boards further away (posteriorly) rather than just minimize space.

Yeah, lot of options there

a. There is a resistor connected to these, presumably to correctly use LEDs, and set the brightness. But I don’t think brightness is something you should think about (if you see it, AI can see it)

IRED brightness is/will be critical to consider, as it will define the corneal reflection and general illumination of the scene. Also, we will not be using AI to track the pupil - we might add it later, but the first pass will be old-school computer vision methods :)

The camera appears to have a removable lens. FYI

Yes, that can be swapped out to change the FOV, or adjusted to set the focus (i.e. adjust the effective focal length of the sensor/lens assembly within narrow range)

The camera board each has four wires. The connector is a large molex, this could be reduced and wires could be soldered directly on the board.

That's USB - the other wires are for power (5V+/-) and the inner wires are for data (3.3v+/-)

a. Also, I would imagine two of the wires are just power, and the other two are data. Putting the Camera boards next to each other, we could further minimize wiring

Better not to f with things at that level - let the USB be USB 😌

b. The wires are wrapped in some sort of electrostatic shielding. This makes them beefy. I wonder if off the shelf coaxial cabling would achieve the same results in a much smaller package

Worth playing around with options, but also might be a rabbit hole - Remember Rule #1 - Don't waste your time improving things that aleady exists/work while there are other things that don't!

Hrm, I just saw this. I came to dump some info, maybe for my own sake. I'll post it below. It is info on the cameras and settings that we can play with, per v4l2.

But back to this issue of capturing video. This is a basic webcam. I wrote some code based heavily on vidgears. It's great, check em out. I added on top of this code to detect connected arduinos (for real time stuffs). I also detect all cameras, allow you to chose which to view/record, but also timestamp each frame (cvs or whatever) (when recieved by the computer), put the timestamp right in the frame (if you want and don't trust your data, it's always there), and do some analysis on the frames. Not that I love my code, I just didn't know you would want this. I ran through the code again, and noted that my envirment variables were screwy (with a new version of ubuntu?). I updated the site, so thanks there.

https://optogeneticsandneuralengineeringcore.gitlab.io/ONECoreSite/projects/streamboat/

Camera info (all these can be played with in v4l2). Maybe more can be played with with in vidgears. Note that these do not currently work with this code, I will have to update how they are called, as they show up as three seperate cameras (v4l2-ctl --list-devices). But I have other projects that require my attention. Sigh....

v4l2-ctl --all Driver Info: Driver name : uvcvideo Card type : Pupil Cam2 ID0: Pupil Cam2 ID0 Bus info : usb-0000:00:14.0-6.2.1.3 Driver version : 6.8.4 Capabilities : 0x84a00001 Video Capture Metadata Capture Streaming Extended Pix Format Device Capabilities Device Caps : 0x04200001 Video Capture Streaming Extended Pix Format Media Driver Info: Driver name : uvcvideo Model : Pupil Cam2 ID0: Pupil Cam2 ID0 Serial : Bus info : usb-0000:00:14.0-6.2.1.3 Media version : 6.8.4 Hardware revision: 0x00002702 (9986) Driver version : 6.8.4 Interface Info: ID : 0x03000002 Type : V4L Video Entity Info: ID : 0x00000001 (1) Name : Pupil Cam2 ID0: Pupil Cam2 ID0 Function : V4L2 I/O Flags : default Pad 0x01000007 : 0: Sink Link 0x02000010: from remote pad 0x100000a of entity 'Extension 3' (Video Pixel Formatter): Data, Enabled, Immutable Priority: 2 Video input : 0 (Camera 1: ok) Format Video Capture: Width/Height : 400/400 Pixel Format : 'YUYV' (YUYV 4:2:2) Field : None Bytes per Line : 800 Size Image : 320000 Colorspace : sRGB Transfer Function : Rec. 709 YCbCr/HSV Encoding: ITU-R 601 Quantization : Default (maps to Limited Range) Flags : Crop Capability Video Capture: Bounds : Left 0, Top 0, Width 400, Height 400 Default : Left 0, Top 0, Width 400, Height 400 Pixel Aspect: 1/1 Selection Video Capture: crop_default, Left 0, Top 0, Width 400, Height 400, Flags: Selection Video Capture: crop_bounds, Left 0, Top 0, Width 400, Height 400, Flags: Streaming Parameters Video Capture: Capabilities : timeperframe Frames per second: 25.000 (25/1) Read buffers : 0

User Controls

                 brightness 0x00980900 (int)    : min=-64 max=64 step=1 default=0 value=0
                   contrast 0x00980901 (int)    : min=0 max=95 step=1 default=48 value=48
                 saturation 0x00980902 (int)    : min=0 max=128 step=1 default=64 value=64
                        hue 0x00980903 (int)    : min=-2000 max=2000 step=1 default=0 value=0
    white_balance_automatic 0x0098090c (bool)   : default=1 value=1
                      gamma 0x00980910 (int)    : min=100 max=300 step=1 default=144 value=144
                       gain 0x00980913 (int)    : min=0 max=100 step=1 default=0 value=0
       power_line_frequency 0x00980918 (menu)   : min=0 max=2 default=1 value=1 (50 Hz)
            0: Disabled
            1: 50 Hz
            2: 60 Hz
  white_balance_temperature 0x0098091a (int)    : min=2800 max=6500 step=1 default=4600 value=4600 flags=inactive
                  sharpness 0x0098091b (int)    : min=1 max=7 step=1 default=2 value=2
     backlight_compensation 0x0098091c (int)    : min=0 max=3 step=1 default=2 value=121

Camera Controls

              auto_exposure 0x009a0901 (menu)   : min=0 max=3 default=3 value=1 (Manual Mode)
            1: Manual Mode
            3: Aperture Priority Mode
     exposure_time_absolute 0x009a0902 (int)    : min=1 max=32 step=1 default=32 value=32
 exposure_dynamic_framerate 0x009a0903 (bool)   : default=0 value=1
               pan_absolute 0x009a0908 (int)    : min=-57600 max=57600 step=3600 default=0 value=0
              tilt_absolute 0x009a0909 (int)    : min=-57600 max=57600 step=3600 default=0 value=0
             focus_absolute 0x009a090a (int)    : min=1 max=100 step=1 default=1 value=1
 focus_automatic_continuous 0x009a090c (bool)   : default=0 value=0
              zoom_absolute 0x009a090d (int)    : min=0 max=3 step=1 default=0 value=0
            zoom_continuous 0x009a090f (int)    : min=0 max=0 step=0 default=0 value=0
                    privacy 0x009a0910 (bool)   : default=0 value=0
              iris_absolute 0x009a0911 (int)    : min=100 max=1000 step=100 default=100 value=100
              iris_relative 0x009a0912 (int)    : min=0 max=0 step=0 default=0 value=0
                  pan_speed 0x009a0920 (int)    : min=0 max=0 step=0 default=0 value=0
                 tilt_speed 0x009a0921 (int)    : min=0 max=0 step=0 default=0 value=0

freemocap / bs

Pupil labs eye cameras #11