processed FIP timeseries have double timestamps

[alexpiet]

from aind_dynamic_foraging_data_utils import nwb_utils as a
nwb = a.load_nwb_from_filename(NEW_NWB_FILES[1])

fib_df_tidy = a.create_fib_df(nwb, tidy=True)

Note that G_2_preprocessed has two entries with the same timestamp, but different data

This is present in the NWB file itself:

stamps = nwb.acquisition['G_2_preprocessed'].timestamps[:]
data = nwb.acquisition['G_2_preprocessed'].data[:]

[alexpiet]

Spot checking this issue in different data assets:

• behavior_705599_2024-06-26_13-04-27.nwb
• behavior_700708_2024-05-30_11-16-22.nwb
• behavior_717531_2024-08-23_08-58-49_processed_2024-08-24_10-37-56
• behavior_711042_2024-08-23_13-27-08_processed_2024-08-24_07-39-12

The doubling of timestamps appears to happen during preprocessing, because G_2 is not effected when G_2_preprocessing is

[alexpiet]

@j-friedrich says this is an artifact of pandas combining two preprocessing methods, instead of separating them into two different columns

[jeromelecoq]

Ahad needs to test stability of this issue.

[alexpiet]

Test this pull request: https://github.com/AllenNeuralDynamics/aind-fip-dff/pull/7

[Ahad-Allen]

Will test with a copy of the pipeline using that pr's branch! :)

[Ahad-Allen]

Looks resolved, just note that this results in multiple fields being formed instead of one 'preprocessed' one I'll give the pr a review :)

[Ahad-Allen]

I think this just needs to be merged, I'll hand it off to ya'll for that! :)

[alexpiet]

@Ahad-Allen can you point me to the output of the test? I want to double check that the double timestamps issue has been resolved

[Ahad-Allen]

Just shared :)

[alexpiet]

Confirmed this problem is resolved by the PR.