Mismatch between firmware and software header data sizes

[MarcelMB]

Getting the following error while running miniscope_io @t-sasatani @sneakers-the-rat

miniscope-io-py3.11mbrosch@RELMNGYFAC31424 miniscope-io % mio stream capture -c /Users/mbrosch/Documents/GitKraken_mac/miniscope-io/miniscope_io/data/config/WLMS_v02_200px.yml -o test.avi [24-08-12T15:57:20] INFO [miniscope_io.okDev] Connected to XEM7310-A75 opalkelly.py:34 Process _buffer_to_frame: Traceback (most recent call last): File "/Users/mbrosch/anaconda3/lib/python3.11/multiprocessing/process.py", line 314, in _bootstrap self.run() File "/Users/mbrosch/anaconda3/lib/python3.11/multiprocessing/process.py", line 108, in run self._target(*self._args, **self._kwargs) File "/Users/mbrosch/Documents/GitKraken_mac/miniscope-io/miniscope_io/stream_daq.py", line 351, in _buffer_to_frame serial_buffer, self.buffer_npix[header_data.frame_buffer_count], locallogs

IndexError: list index out of range
[24-08-12T15:57:23] WARNING  [miniscope_io.streamDaq] Termination timeout: force terminating process _fpga_recv.                                                            stream_daq.py:583
                    INFO     [miniscope_io.streamDaq] Child processes joined. End capture.           

I checked frame width and height 200 px, header len 12x32 bit, buffer block length 10, num buffer: 32
8 MHz for 200x200 pixel at framerate 20 also fits

Block size was the problem that in the config file of miniscope io was set to 512 but the firmware is on 32 bit mode making it 128 bit

**What could be nice to add as a feature is receiving the header_file from the miniscope that is currently in the firmware and displaying it to the user. Could then show the mismatch to the miniscope_io config**  

[MarcelMB]

after changing block_size to 128 still getting 'list index out of range' error. Which doesn't really make sense to me yet?!

[MarcelMB]

It runs with the newest commits from firmware a781dbf and miniscope_io a541fe8. I am not exactly sure what has changed.

but still having lots of errors with the buffer reading. not because the signal is bad. Hardware wise this looks all fine to me.

just sending test frames out and it looks bad. video attached to slack message of issue here

[MarcelMB]

WARNING [miniscope_io.streamDaq.buffer] Frame 1215; Buffer 9723 (#3 in frame) stream_daq.py:176 Expected buffer data length: 1232, got data with shape (5072,).
Padding to expected length

it seems like the buffer is constantly to large for what is expected

[t-sasatani]

I'm running the exact same version codes and getting test slide images, so this is confusing. I think you should first run the code in commits as it is if you haven't done that yet. If it still doesn't work, it might be worth running the same firmware on different hardware (the dev board for this case) or reverting firmware for testing.

• It runs with the newest commits from firmware a781dbf and miniscope_io a541fe8.
• What could be nice to add as a feature is receiving the header_file from the miniscope that is currently in the firmware and displaying it to the user.

• 35 is supposed to solve this, so working correctly when using this is expected. Also for displaying metadata, this PR should do metadata export to a .csv file in the same directory as the video file. I'm wondering what amount of information should be presented in realtime, because users shouldn't modify this config.

after changing block_size to 128 still getting 'list index out of range' error. Which doesn't really make sense to me yet?!

This should be 512 and making this 128 with the current firmware will fill 1/4 of the pixels in a image. Sorry that it's confusing because it's 32-bit defined in FW and 8-bit defined in firmware. I'll pull out the block size to config and unify these definitions in next PR.

[sneakers-the-rat]

Ideally we use the metadata reported by the device instead of just show it? Sorry havent had time to work on this in awhile but will return soon, I think thats the eventual goal tho right? Either software controls config and can configure the device or device has fixed config and can tell software, one or the other

[t-sasatani]

No rushes at all! There are only project people in this repo, and the commits will be linear by nature.

Ideally we use the metadata reported by the device instead of just show it?

Ideally, yes, but practically unsure. We'll likely have bit errors for a while, and letting a single bit flip in the metadata break the buffer structure is pretty risky. Also, these buffer structure-related metadata will likely be fixed for each device, so manually selecting a device on mio might be a middle ground.

We're still a bit far from this, but I think the actual ideal would be statistically detecting metadata from the first multiple transmissions.

[sneakers-the-rat]

oh ya totally, not variable per frame, i meant on initial connection <3

so we have it for the docs, would it be possible to link to the relevant place in the firmware code where these config values get set? would be good to have a page per device that lists stuff like all the related code and hardware and papers and whatnot

[t-sasatani]

would be good to have a page per device that lists stuff like all the related code and hardware and papers and whatnot

Yes, for sure. Structure-wise, I already added something similar in the latest PR, though there is not much information yet.

• https://miniscope-io--35.org.readthedocs.build/en/35/device/test_device.html

The firmware portion is here, and I'll add a link to the above page later. It's private, and I think some project names people don't want to expose yet, so access will be restricted for now.

• https://github.com/Aharoni-Lab/Miniscope-SAMD-Framework/blob/main/include/MS_config.h#L39C1-L68C7

Now that the firmware is also in alpha, we might want to start thinking about documenting this in general, too. Once it's public, it'll be easy to host Doxygen on GitHub pages, but it's not as nice as Sphinx, so it's kind of undetermined here, too.

[MarcelMB]

I just did another test with the feedback provided

I see that a lot of the buffers are now correct but still getting a pretty bad signal. Block_size in miniscope_io config is set to 512 as it is 4 times the firmware block_size of 128. So that is correct now. All is what Takuya used without any manual changes to the commits from my side. firmware commit: abbaedd53eb67a8665d8acc2cee0cab64b42f211 miniscope-io commit: a541fe8444fa05d27f71f32177beed826979d77d

Despite many buffers now fitting I still get a lot of buffers wrong. And this is not related to hardware since I watch the signal on the oscilloscope and it looks quite good. Except of the gap in between the buffers when the signal goes crazy.

Video stream is attached to the slack

issue

next test was going back to dev board and using same firmware and miniscope-io commits as mentioned in my first post here

and I get the same it feels of some buffers work almost other not at all worse is I don't even get a video stream or saved video

So I am puzzled. Both WL MS PCB and WL MS dev board wont work. Header structure seems to be fine now. Wondering why I cant reproduce what Takuya did with the same commit IDs

[t-sasatani]

hmm, very interesting. I'm still curious if this is a hardware or software/firmware issue. Could you try this with combinations of old/new software/firmware in the same physical conditions (hardware, position, excitation LED brightness if that's exposed, etc.) to know which pairs cause problems? At least the last working one on your end should be compatible with the current software/firmware. Doing this only on the dev board is enough.

[t-sasatani]

Also, it's not critical here, but the warning you always get in the last buffer in the frame (so _#7 in the frame for this case) is a minor bug, and I'll address it soon.

[MarcelMB]

hmm, very interesting. I'm still curious if this is a hardware or software/firmware issue. Could you try this with combinations of old/new software/firmware in the same physical conditions (hardware, position, excitation LED brightness if that's exposed, etc.) to know which pairs cause problems? At least the last working one on your end should be compatible with the current software/firmware. Doing this only on the dev board is enough.

I did this and went back into software testing with the WL MS dev board for now

f11d9018e8bde5587664c0575bc51cec2a7e1892 using this miniscope-io commit v.0.3.0 release I'm getting the same error as reported here: https://github.com/Aharoni-Lab/miniscope-io/issues/34 I am using 52f31a WL MS firmware

it should theoretically be nothing hardware-related because I am using the dev board again that worked perfectly fine before the new miniature PCB (of course we cant completely rule out hardware issues)

[MarcelMB]

so I went here because this bug should have been fixed in this commit a541fe8444fa05d27f71f32177beed826979d77d

when I do this the code is running. not having the error as described in previous comment. but no image appears in the little window (I posted this already in a picture its just gray), and Expected buffer data length: 4496, got data with shape (5072,).
Padding to expected length
buffer seems to always be slightly off, despite not even showing the frames

so I am very puzzled which combination of framework and miniscope-io worked, I don't actually remember exactly, tried different versions but all seems to not work

[sneakers-the-rat]

Ill start digging into this on Monday, there is something wrong with detecting buffer headers if the first processing method is passing humongous buffers through to the second method

[t-sasatani]

tried different versions but all seems to not work

It feels very likely that this is a hardware or physical setup issue now. I've been going back and forth between SW/FW versions, and it works finely. It's hard to rule these physical issues out because they're hard to reproduce, but it might be good to find if there is a hardware/physical setup that works fine to start with (e.g., changing the LED position, etc.).

Expected buffer data length: 4496, got data with shape (5072,). Padding to expected length

FYI, this warning with these specific numbers (5072, 4496) wouldn't negatively affect image recovery. This was a minor bug on the data validation side, and is fixed in https://github.com/Aharoni-Lab/miniscope-io/pull/35/commits/bf0a519f15176392b9f6d31ff3bb96e0421649e4.

there is something wrong with detecting buffer headers if the first processing method is passing humongous buffers through to the second method

Large buffers are usually caused if the preambles in the next buffers are missed. Since the detector software logic is not very complex here, I'm assuming actual bit errors cause this. Passing down long buffers should be an expected behavior in these cases, but won't produce images if some values in metadata are apparently wrong.

[t-sasatani]

I feel that a part of this issue is that we don't have a diagnostic method to evaluate bit errors yet. Physical communication shouldn't usually be classified into binary working/non-working states, so I will think about this.

[MarcelMB]

Adding binary files here for working conditions and non-working conditions to better understand what is going on:

miniature WL MS PCB, wired communication (directly plugged into FPGA, no wireless), this is the PCB that is working with the new commits of FM/SW: Drive link

Ill add other non working recordings tomorrow

[MarcelMB]

Today I repeated the exact same recording. Nothing has changed ( literally just turned on board and fpga from last evening again) and miniscope io doesn't show a live video stream or save the video file. So reporting this as a potential software bug (but again not excluding hardware).

Drive link Binary, csv and failed avi.

Frame_buffer_count looks different. Doesn't start from 0

[MarcelMB]

funnily when I do this all wireless we get a video stream and it seems to work mostly smoothly, and importantly video file is created which wasn't the case for wired recording from today

here is a recording where I moved the board inside the box slightly (which introduced errors) Drive link

and one where I didn't move it and it was just sitting below the SiPM in good conditions Drive link

(P.S. let me know if this is a new issue because I am not sure if I should actually still post this here or somewhere else, but always happy to learn about good github practice also let me know if I should record more detailed binary files etc.)

still using a541fe8444fa05d27f71f32177beed826979d77d and FM: e18464

[MarcelMB]

I also want to add the recording from the development board (v1.0 , april 18 2023) since all the earlier recordings were from the miniature WL MS PCB this board has worked fine previously with FM/SW commits but is now 'not working; with the new versions (same commits as my previous commit used)

No live video stream in miniscope io, lots of 'broken' buffers, however, a few buffers seem ok Drive link

I'm also happy to contribute for test of the software part. Just don't know how this all work yet. But as Jonny described would be great to have test for many cases so we don't run into the same problems of not knowing where the bug is software or hardware. So I am happy to learn here hope to avoid this

[MarcelMB]

and last but not least I also supply files for the dev board (v1.0 , april 18 2023) but wired. Here I cant supply many files because

After this: miniscope-io-py3.11mbrosch@RELMNGYFAC31424 miniscope-io % mio stream capture -c /Users/mbrosch/Documents/GitKraken_mac/miniscope-io/miniscope_io/data/config/WLMS_v02_200px.yml -b -o wired_devboard [24-08-21T15:44:35] INFO [miniscope_io.okDev] Connected to XEM7310-A75

nothing happens. Just saves the files empty except binary has some I assume. Drive link

in summary, we have files for miniature PCB and dev board in wired and wireless condition.

[sneakers-the-rat]

This is super good. Can you also post logs? They should be in ~/.config/miniscope-io by default. If you just zip that directory that would b great

[MarcelMB]

logs.zip

[MarcelMB]

I would close this issue. We did resolve the initially mentioned bug.