search

AllenNeuralDynamics / aind-fip-dff

This capsule takes an input raw NWB file of fiber data and creates processed DFF traces
MIT License
0 stars 0 forks source link

NWB files tables in the files need to be reorganized to follow some standardized convention #4

Closed alexpiet closed 2 weeks ago

alexpiet commented 3 months ago

EDIT: Updated description of NWB file, including outstanding tasks, here: https://alleninstitute-my.sharepoint.com/:w:/g/personal/alex_piet_alleninstitute_org/Eeep60T6VnZBuzW7SA0ujrUBM6mRdNSPVVohIFDJly84Yw?e=65DvHk

hanhou commented 3 months ago

scratch['metadata'] should have all the information in the aind-metadata session.json, right?

Actually the opposite is true. Scratch is my temporary way of storing meta data, so we should remove it and make sure that all information in the scratach exists in the json metadata files.

Ahad-Allen commented 3 months ago

  • nwb.stimulus should contain:

    • go Cue (currently only in trials)

    • rewards (currently in acquisitions, and only has time of rewards, not size)

      • data should contain the reward volume, instead of "1" which is the current value

    • scratch['metadata'] should have all the information in the aind-metadata session.json, right?

    • nwb.experimenter is none

    • nwb.session_description is the string "Test File"

    • nwb.lab is none

    • nwb.notes is none (we have notes in the metadata)

    • nwb.protocol is none

    • nwb.surgery is none

    • nwb.virus is none

Most of this makes sense to update! We can pull a lot of the missing metadata from the json files. The scratch section should be removed, as Han pointed out. I'm a little confused on the changes you'd like to do to the stimulus. Are you suggesting we add the Stim table with reward times and go Cues in it(I am not sure what a go Cue is, but that is probably just my ignorance on experimental design here)? That shouldn't be too large of a change, if so!

alexpiet commented 3 months ago

Yes, we should add a stimulus table with reward times (water delivery, currently in acquisition), and go cue (sound played to the mouse, currently only in the trials table)

Ahad-Allen commented 2 months ago

@rachelstephlee here's the metadata objects we should be adding

rachelstephlee commented 2 months ago

where was the session_description set? I can't re-set it. NWB gives me an error that the attribute is already set.

rachelstephlee commented 2 months ago

@rachelstephlee will followup with @Ahad-Allen on this.

Ahad-Allen commented 2 months ago

I believe @rachelstephlee already did some work on adding most of this metadata, and I have some code that should move where the gocues are present. We still need to add reward volume and remove scratch. I can work on these now

Ahad-Allen commented 2 months ago

Good news, the reward volume size is already in the trials table as reward_size_left and reward_size_right. I added the gocues in the feat-4 branch. I'll push a pr now

Ahad-Allen commented 2 months ago

PR: https://github.com/AllenNeuralDynamics/NWB-Packaging-FiberPhotometry-Base-Capsule/pull/21

alexpiet commented 2 months ago

EDIT: see updated specs in top comment

alexpiet commented 2 months ago

https://github.com/AllenNeuralDynamics/NWB-Packaging-FiberPhotometry-Base-Capsule/pull/16 adds nwb.surgery, nwb.Experimenter, nwb.protocol, and nwb.lab

alexpiet commented 2 months ago

Need to define an NWB specification here

jeromelecoq commented 2 months ago

Assigning to Alex to finish document. @Ahad-Allen can handle once done.

alexpiet commented 2 months ago

This NWB extension is for adding AIND metadata to NWB files https://github.com/AllenNeuralDynamics/ndx-aind-metadata

alexpiet commented 1 month ago

Feedback needed:

rachelstephlee commented 1 month ago

would we want a version # of the pipeline added in "NWB file generated by AIND pipeline"? othewise we are guessing when the data is being processed + the state of the pipeline at that time

alexpiet commented 1 month ago

would we want a version # of the pipeline added in "NWB file generated by AIND pipeline"? othewise we are guessing when the data is being processed + the state of the pipeline at that time

I think that should be taken care of in the processing metadata

alexpiet commented 1 month ago

I believe I have finished reviewing how to organize the NWB files, and generate many specific issues.

Details here: https://alleninstitute-my.sharepoint.com/:w:/g/personal/alex_piet_alleninstitute_org/Eeep60T6VnZBuzW7SA0ujrUBM6mRdNSPVVohIFDJly84Yw?e=dAPmUI

I would like feedback on these specific issues, as well as reviewing the file above:

Ahad-Allen commented 4 weeks ago

Session_Description could maybe be generated using the processing.json to add a little more time? Not sure on the reference time, perhaps we could ask someone who's worked on Harp on this.

jeromelecoq commented 3 weeks ago

session_description should include the task + modalities recorded. nwb.timestamps_reference_time is absolute to world clock.

We agreed to Reset Harp timestamps at beginning to task for new experiments and calculate the value the field should be for past experiments, accepting some imprecision (seconds off is ok).

jeromelecoq commented 3 weeks ago

Related issue : https://github.com/AllenNeuralDynamics/dynamic-foraging-task/issues/319

jeromelecoq commented 3 weeks ago

@alexpiet will coordinate there.

alexpiet commented 3 weeks ago

I made separate issues for the session_description and timestamps_reference_time.

If everyone can give feedback on the NWB planning document then we can close this issue and each individual item is a separate issue

https://alleninstitute-my.sharepoint.com/:w:/g/personal/alex_piet_alleninstitute_org/Eeep60T6VnZBuzW7SA0ujrUBM6mRdNSPVVohIFDJly84Yw?e=WhfsAP

dyf commented 2 weeks ago

has been broken down into sub-issues. more changes incoming once HED tags discussion resolves.