ZDA analysis reproducibility

[john-judge]

The more I dig through ZDA files (in this script) the more I’m convinced there are fairly extensive secret artifact corrections happening within old PhotoZ

I tried more different orderings (e.g. data inversing before baseline, etc), and this is the closest I was able to get with attempted Python reproductions of the old PhotoZ analysis (all are the same 06_01_01.zda with good signal -- Wen-Chi’s data):

Attempt to see if Python scripting can find signal in TSM data, which may have fewer artifacts:

I think the next attempt would be to write a TSM-to-ZDA converter and see if I can load binned TurboSM data straight into the old PhotoZ.

[john-judge]

Successfully created TSM-to-ZDA load script

Immediately realized was that PhotoZ's original polynomial fit breaks down with TurboSM’s dramatic low-frequency baseline. So I had to do baseline correction outside of PhotoZ to avoid letting PhotoZ blow apart the data.

We sometimes see weird artifacts in the temporal dimension on the old rig. At least some of them are very possibly bugs in PhotoZ's polynomial-8 baseline correction, which was written from scratch in C++ in early 2000's.

Two test TurboSM recordings showed no signal in PhotoZ

Consistent with the results in Python scripting:

Will look through my old TSM data for something with good signal.

[john-judge]

Looked at TSM data from 1/17/22 in PhotoZ. My notes on this were that we had good signal around the stimulating electrode but none spatially distributed (that I could find).

In PhotoZ, we can see a bit of reasonable signal in ~spatially distributed areas~ (see the pixel indices): Updated: found an issue (see May 23 post) that invalidates this conclusion

[john-judge]

New pipline: TurboSM -> TSM-to-ZDA Python script -> C++ PhotoZ

New rig computer uses https://github.com/john-judge/PhotoZ_upgrades/tree/load-hacked-zda to load hacked ZDA files. Install instructions on new rig:

• Shortcut on desktop is called PhotoZ-TSM-compatible.exe
• Source code is in Desktop/PhotoZ_old/PhotoZ_Jan2021/PhotoZ_upgrades
• git checkout load-hacked-zda
• move .exe from x64/Debug to PhotoZ folder (to use correct .dll paths)
• create shortcut for Desktop

Example workflow on new rig:

[john-judge]

Data arrangement is incorrect, as revealed when validating with Masoud's data. The following is more than just an indexing error (swapping axes or transposing or reshaping will not solve this issue) -- instead it looks like a binary packing alignment issue, because the bright spot that should be in the middle is exploded at the bottom of the PhotoZ RLI image.

Similar can be noticed for the MaxAmp/SD frame, but it is more subtle and may be only an indexing issue, not a byte alignment issue.

One solution would be to write an interface to C++ for ZDA writing, so we can directly pass data to a modified FileController:saveData

[john-judge]

Analysis of 2/7/22 TSM data -- single trial with signal from 200 ua stim

[john-judge]

02_01_03.zda from 02-09-22, signal from 200 ua stim, single trial only

Suggestion is that new rig has higher SNR?

[john-judge]

Breakthrough after 2 days of struggling with TSM-to-ZDA load. At first I thought the way PhotoZ treats a 'C_CONTIGUOUS' numpy array was incorrect. Instead, I realized that PhotoZ does not tolerate non-square images well. PhotoZ stores images as 1-D arrays, but not in an intuitive way. The typical way would be the simple C convention of row-major, but this assumption only holds true for square arrays. Rather than try to hack PhotoZ to be more flexible, we can proceed with only square arrays and still do all that is needed with just some cropping.

Added an exception raise to prevent TSM-to-ZDA from proceeding if images are not square.

RLI correct load with square images does not need any extra byte-alignment or transposition handling:

As opposed to non-square arrays, regardless of attempts at transpose in numpy, transpose in C++, adjust offsets, etc --

PhotoZ analysis of a TSM file

TO-DO: PhotoZ analysis of recent TSM data

[john-judge]

5/25/22 TurboSM recordings processed in PhotoZ

200 ua stim 512x160 at 2000 Hz

Slice 1 Location 1

Recording 5

Recording 7

Slice 2 Location 1

Recording 15

Recording 16

Recording 17

Slice 2 Location 2

Recording 19

Recording 20

Recording 21