Spike2photom

[kouichi-c-nakamura]

This pull request adds an ability to export Photometry continuous data into Spike2 waveform channels. Only tested for one session so far. There may be cases where photometry data are missing or specified photometry_keys is not found. Do we want to see some warning or issue an error in such cases? Take a look at #TODO signs.

Exported Spike2 data itself appeared OK. There may be an issue if the Photometry file is shorter than the PyControl file. If you can find such a case with an error, I think I can find a way to handle it.

[teristam]

I notice an error on the following file: Julien/Data/head-fixed/by_sessions/reaching_go_spout_bar_nov22/RE604-2023-03-22-120336

lueError                                Traceback (most recent call last)
/home/MRC.OX.AC.UK/ndcn1330/code/trialexp/workflows/scripts/03_export_spike2.py in line 2
      [58](file:///home/MRC.OX.AC.UK/ndcn1330/code/trialexp/workflows/scripts/03_export_spike2.py?line=57) #%%
----> [59](file:///home/MRC.OX.AC.UK/ndcn1330/code/trialexp/workflows/scripts/03_export_spike2.py?line=58) export_session(df_pycontrol, keys, 
      [60](file:///home/MRC.OX.AC.UK/ndcn1330/code/trialexp/workflows/scripts/03_export_spike2.py?line=59)              data_photometry = data_photometry,
      [61](file:///home/MRC.OX.AC.UK/ndcn1330/code/trialexp/workflows/scripts/03_export_spike2.py?line=60)              photometry_times_pyc = photometry_times_pyc,
      [62](file:///home/MRC.OX.AC.UK/ndcn1330/code/trialexp/workflows/scripts/03_export_spike2.py?line=61)              photometry_keys = photometry_keys,
      [63](file:///home/MRC.OX.AC.UK/ndcn1330/code/trialexp/workflows/scripts/03_export_spike2.py?line=62)              smrx_filename=soutput.spike2_file)

File [~/code/trialexp/trialexp/process/pycontrol/utils.py:435](https://vscode-remote+ssh-002dremote-002b7b22686f73744e616d65223a224a616465227d.vscode-resource.vscode-cdn.net/home/MRC.OX.AC.UK/ndcn1330/code/trialexp/~/code/trialexp/trialexp/process/pycontrol/utils.py:435), in export_session(df, keys, export_state, print_expr, event_ms, smrx_filename, verbose, print_to_text, data_photometry, photometry_times_pyc, photometry_keys)
    [433](file:///home/MRC.OX.AC.UK/ndcn1330/code/trialexp/trialexp/process/pycontrol/utils.py?line=432) y_index += 1
    [434](file:///home/MRC.OX.AC.UK/ndcn1330/code/trialexp/trialexp/process/pycontrol/utils.py?line=433) if name in data_photometry:
--> [435](file:///home/MRC.OX.AC.UK/ndcn1330/code/trialexp/trialexp/process/pycontrol/utils.py?line=434)     write_photometry(name)
    [436](file:///home/MRC.OX.AC.UK/ndcn1330/code/trialexp/trialexp/process/pycontrol/utils.py?line=435) else:
    [437](file:///home/MRC.OX.AC.UK/ndcn1330/code/trialexp/trialexp/process/pycontrol/utils.py?line=436)     # name is not found in data_photometry
    [438](file:///home/MRC.OX.AC.UK/ndcn1330/code/trialexp/trialexp/process/pycontrol/utils.py?line=437)     #TODO issue a warning or error or nothing? 
    [439](file:///home/MRC.OX.AC.UK/ndcn1330/code/trialexp/trialexp/process/pycontrol/utils.py?line=438)     ...

File [~/code/trialexp/trialexp/process/pycontrol/utils.py:426](https://vscode-remote+ssh-002dremote-002b7b22686f73744e616d65223a224a616465227d.vscode-resource.vscode-cdn.net/home/MRC.OX.AC.UK/ndcn1330/code/trialexp/~/code/trialexp/trialexp/process/pycontrol/utils.py:426), in export_session..write_photometry(name)
    [423](file:///home/MRC.OX.AC.UK/ndcn1330/code/trialexp/trialexp/process/pycontrol/utils.py?line=422) if len(Y_no_nan) == len(T_no_nan):
    [424](file:///home/MRC.OX.AC.UK/ndcn1330/code/trialexp/trialexp/process/pycontrol/utils.py?line=423)     new_Y = np.interp(new_T, T_no_nan, Y_no_nan)
--> [426](file:///home/MRC.OX.AC.UK/ndcn1330/code/trialexp/trialexp/process/pycontrol/utils.py?line=425)     spike2exporter.write_waveform(new_Y, name, y_index, multiplier)
    [427](file:///home/MRC.OX.AC.UK/ndcn1330/code/trialexp/trialexp/process/pycontrol/utils.py?line=426) else:
    [428](file:///home/MRC.OX.AC.UK/ndcn1330/code/trialexp/trialexp/process/pycontrol/utils.py?line=427)     # the length mismatch, maybe a wrong data type (not a time series?)
...
--> [229](file:///home/MRC.OX.AC.UK/ndcn1330/code/trialexp/trialexp/process/pycontrol/spike2_export.py?line=228) AdcData_int = np.int16([(adc - offset)*6553.6/scale for adc in AdcData]) # np.ndarray
    [231](file:///home/MRC.OX.AC.UK/ndcn1330/code/trialexp/trialexp/process/pycontrol/spike2_export.py?line=230) #AdcData is meant to be in int16 (-32,768 to 32,767) or short in C
    [232](file:///home/MRC.OX.AC.UK/ndcn1330/code/trialexp/trialexp/process/pycontrol/spike2_export.py?line=231) self.MyFile.SetChannelYRange(y_index, np.max(AdcData_int), np.min(AdcData_int))

ValueError: cannot convert float NaN to integer

[kouichi-c-nakamura]

When Photometry signals are saturated, scale becomes 0 and when dividing a value with scale you get NaN or inf, causing an error when converting float to int16

test run worked

[kouichi-c-nakamura]

I made lots of changes because I found that Spike2 file lacked some of the important channels and that it contained many empty channels.

I assumed that there is just one pycontrol txt file in a folder. If not, this will create some errors.

Test run on a single session looked good.

[teristam]

I would suggest merging it to the dev branch, as the master branch doesn't support task-based folder structure, so there is no reliable way of testing it on the current folder structure. Please resolve the conflict before merging.

[teristam]

I will help resolve the conflict with dev. Please wait

[kouichi-c-nakamura]

Thanks! What now? Do you run it for everything at some point?

[teristam]

Thanks! What now? Do you run it for everything at some point?

it is now merged to the dev branch. I tried running the pipeline on all our dataset but encountered errors due to the "ghost file" problem. I don't think that's related to this export feature. Guess we will have to find some other ways to solve the ghost file problem on the network drive..

[kouichi-c-nakamura]

Encountered an error when exporting 'reaching_go_spout_bar_all_reward_dec22', 'JC317L-2022-12-16-173145'

pulse_times_A len:90 pulse_times_B len: 16

RsyncError: Two pulse times differ too much. Results may not be correct

line68 in 03 script

Maybe need try to handle the error

It's likely that pycontrol and pyphotometry files are paired wrongly.

[teristam]

Encountered an error when exporting 'reaching_go_spout_bar_all_reward_dec22', 'JC317L-2022-12-16-173145'

pulse_times_A len:90 pulse_times_B len: 16

RsyncError: Two pulse times differ too much. Results may not be correct

line68 in 03 script

Maybe need try to handle the error

It's likely that pycontrol and pyphotometry files are paired wrongly.

It is deliberately done that way so that the pipeline will refuse to proceed for the wrong file, otherwise, everything will be messed up. We can add the -k option when running snakemake to skip that particular recording. I suggest we open another issue for this as this is not related to this PR directly.