Metadata errors to fix

[alexpiet]

Rig

• [x] Rig ID uses YYYYMMDD, not YYYY_MM_DD
• [x] Tube manufacturer should be CUSTOM or OTHER, not null
• [x] Motorized stage needs a model name
• [x] Reward spouts needs a manufacturer. CUSTOM, OTHER, JANELIA
• [x] Solenoid - manufacturer, model, serial number?
• [x] How is there a lick_sensor_type without a lick_sensor?
• [x] Isn’t the lick_sensor the janelia lick detector rather than the spout?
• [x] Speaker Manufacturer, model, serial number?
  • [x] Need info
• [x] Camera lens:
  • [x] How is there a model number without a manufacturer?
  • [x] Do we need to change the focal length field if it doesn’t work for the lens?
• [ ] Add position of cameras
• [ ] Add enclosure
• [ ] Harp daq:
  • [ ] core_version for each harp device, should get dynamically from the devices. I hard-coded the value used in 447.
• [x] Calibrations: typo in lack spout right description
• [x] Beamsplitter:
  • [x] Confirm height of beamsplitter (24 or 25)
  • [x] This is more a choice thing, but if the notes contain “493/574 nm BrightLine dual-edge standard epi-fluorescence dichroic beamsplitter" then I would use a name like “beamsplitter” rather than duplicating half that information in the name. If these are details we want to track, let’s get it into the schema.

Session

• [x] Daq device names don’t match the names in the rig
• [ ] Dynamic foraging task software should be in the stimulus_epoch section
  • there are different stimulus_epoch (auditory or optogenetics). Which one to store the software?
• [x] Speaker name doesn’t match the name in rig
• [ ] How was there 343 uL water consumed in the session and none in the stimulus epoch?
  • I think it makes more sense to store the water in the session instead of a specific stimulus (we have different stimulus auditory tone occurs every trial, optogenetics occurs in some trials...).
• [ ] You list several patches, only one is listed in the Rig (and not with this name)
• [ ] These are surgically implanted fibers I assume?
• [ ] Add Behavior parameters in the stimulus software's parameters fields in the stimulus epoch.
• [ ] https://github.com/AllenNeuralDynamics/dynamic-foraging-task/issues/465

[alexpiet]

@XX-Yin Can you take care of the Session metadata updates?

[XX-Yin]

@XX-Yin Can you take care of the Session metadata updates?

I can handle the session metadata.

[hagikent]

I saw a couple behavior_json has erroneous entries such as animal_weight_post = 24..12 (should reas 24.12) Would it be possible to "force" input type?

This causes ValueError: could not convert string to float and would be a reason for session.json not being produced.

[XX-Yin]

I can add a format check when using the animal_weight_post in the session metadata. Please let me know if you find other edge cases.

[XX-Yin]

I saw a couple behavior_json has erroneous entries such as animal_weight_post = 24..12 (should reas 24.12) Would it be possible to "force" input type?

This causes ValueError: could not convert string to float and would be a reason for session.json not being produced.

This session should be an old session. The current GUI will check the format and the non-numeric expression is not allowed.

[alexpiet]

Additional feedback from Saskia about session metadata. @XX-Yin can you address these issues?

Notes are for behavior_719511_2024-08-05_09-33-05 (I believe):

• [x] Add project_name to data_description (Alex: this is resolved)

In Session:

• [x] "Lick sensor left" and "Lick sensor right" are not daq_names for your rig and thus can't be listed as under daq_names. (Xinxin: the new lick detector should be included in the daq_names?)
• [x] Camera names don't match the names in the rig. They must match. There is a utility function you can use to make sure that the names align between rig and session. (Xinxin: the camera name matches the names in the ephys rig. This seems to be an issue in the behavior rig)
• [ ] Reward delivery cannot be null. How are you delivery rewards without it? (Xinxin: we don't have the lickspout reference area)
• [ ] Your session_end_time is before the end of one of your data streams. The end time should be the last time of all data streams and stimulus epochs. (Xinxin: the session_end_time here is the end of the behavior session. For each data stream, there is a separate start and end time.) Stimulus Epoch
• [x] Missing stimulus_device_name
• [x] These output parameters and task parameters duplicate almost the entire metadata schema. Can we please please please not do this? Very seriously, this is going to make our DocDB painful and it ultimately undermines having the schema in the first place.
• [x] eg. calibrations should be in the calibration field (Xinxin: calibrations are in the calibration field. I don't know this issue)
• [ ] frame rate should be in the device (also, check the version of the schema you're using. The cameras in your device class have a max_frame_rate which was changed a while ago) (Xinxin: old version of the schema)
• [x] Your frame_num item specifically is blocking upload to DocDB. You can't have fields with ".csv" in the name. This information should live in the quality_control class (which was only recently added to the schema, but that's where it should live) (Alex: the .csv name is fixed, but not moved to quality_control)

[XX-Yin]

@alexpiet I do not have time to address these issues recently. Did you have a discussion with Saskia about these issues?

[alexpiet]

@alexpiet I do not have time to address these issues recently. Did you have a discussion with Saskia about these issues?

Saskia gave me feedback on the session.json while I was addressing the issue of ".csv" in the fieldnames blocking uploads to DocDB.

[XX-Yin]

The session metadata can be regenerated easily when we update the code.

[JeremiahYCohen]

Do we have an update here @XX-Yin ? Do you need help from one or two other folks to get this done?

[XX-Yin]

I don’t get a chance to update it. That would be great if some forks can help with this. Who should I contact for help?

Get Outlook for iOShttps://aka.ms/o0ukef

---

From: JeremiahYCohen @.> Sent: Thursday, October 31, 2024 8:24:20 AM To: AllenNeuralDynamics/dynamic-foraging-task @.> Cc: Xinxin Yin @.>; Mention @.> Subject: Re: [AllenNeuralDynamics/dynamic-foraging-task] Metadata errors to fix (Issue #510)

Do we have an update here @XX-Yinhttps://github.com/XX-Yin ? Do you need help from one or two other folks to get this done?

— Reply to this email directly, view it on GitHubhttps://github.com/AllenNeuralDynamics/dynamic-foraging-task/issues/510#issuecomment-2450161059, or unsubscribehttps://github.com/notifications/unsubscribe-auth/A2CTX5VS3RDXOSBL6TWECVTZ6JDSJAVCNFSM6AAAAABJGQB67CVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDINJQGE3DCMBVHE. You are receiving this because you were mentioned.Message ID: @.***>

[alakunina]

Can I help with getting this updated? I also really need the metadata issues fixed for my experiments.

[JeremiahYCohen]

@alakunina That would be great, thanks. @hanhou can you work with Anna on this?

[hanhou]

@alakunina That would be great, thanks. @hanhou can you work with Anna on this?

No problem.

[JeremiahYCohen]

@alakunina @hanhou @XX-Yin any updates here? There are nine unresolved items above. Happy to add other folks if this is a big lift.

[alexpiet]

Probably a good time to take care of this as well:

• [ ] https://github.com/AllenNeuralDynamics/dynamic-foraging-task/issues/1004

[hagikent]

@alexpiet FYI, I suggested David ingesting "requestor" from surgery form into procedure. Can inherit from there or manually inputting at GUI for each session. Latter maybe better as, in rare cases, requester and responsible-sci may be different.

[XX-Yin]

• https://github.com/AllenNeuralDynamics/dynamic-foraging-task/pull/1051
• [x] "Lick sensor left" and "Lick sensor right" are not daq_names for your rig and thus can't be listed as under daq_names. (Xinxin: the new lick detector should be included in the daq_names?)
• [x] Camera names don't match the names in the rig. They must match. There is a utility function you can use to make sure that the names align between rig and session.
• [ ] Reward delivery cannot be null. How are you delivery rewards without it? (Xinxin: we don't have the lickspout reference area)
• [ ] Your session_end_time is before the end of one of your data streams. The end time should be the last time of all data streams and stimulus epochs. (Xinxin: the session_end_time here is the end of the behavior session. For each data stream, there is a separate start and end time.) Stimulus Epoch
• [x] Missing stimulus_device_name
• [x] These output parameters and task parameters duplicate almost the entire metadata schema. Can we please please please not do this? Very seriously, this is going to make our DocDB painful and it ultimately undermines having the schema in the first place.
• [x] eg. calibrations should be in the calibration field
• [ ] frame rate should be in the device (also, check the version of the schema you're using. The cameras in your device class have a max_frame_rate which was changed a while ago) (Xinxin: not clear to me)
• [x] Your frame_num item specifically is blocking upload to DocDB. You can't have fields with ".csv" in the name. This information should live in the quality_control class (which was only recently added to the schema, but that's where it should live) (Alex: the .csv name is fixed, but not moved to quality_control)
• [x] adding software name to the stimulus_epochs

[XX-Yin]

I updated most fields. And there are some fields requiring more discussion tomorrow.

• [ ] Reward delivery cannot be null. How are you delivery rewards without it? (Xinxin: we don't have the lickspout reference area)
• [ ] Your session_end_time is before the end of one of your data streams. The end time should be the last time of all data streams and stimulus epochs. (Xinxin: the session_end_time here is the end of the behavior session. For each data stream, there is a separate start and end time.)
• [ ] frame rate should be in the device (also, check the version of the schema you're using. The cameras in your device class have a max_frame_rate which was changed a while ago) (Xinxin: not clear to me)

[XX-Yin]

• [ ] "Lick sensor left" and "Lick sensor right" are not daq_names for your rig and thus can't be listed as under daq_names. (Xinxin: the new lick detector should be included in the daq_names? Janelia should be removed from daq names)
• [x] Camera names don't match the names in the rig. They must match. There is a utility function you can use to make sure that the names align between rig and session.
• [ ] Reward delivery cannot be null. How are you delivery rewards without it? (Xinxin: we don't have the lickspout reference area; need a reference area and reference coordinates collected from the GUI)
• [ ] Your session_end_time is before the end of one of your data streams. The end time should be the last time of all data streams and stimulus epochs. (Xinxin: the session_end_time here is the end of the behavior session. For each data stream, there is a separate start and end time. Behavior should go to the stimulus_epoch) Stimulus Epoch
• [x] Missing stimulus_device_name
• [x] These output parameters and task parameters duplicate almost the entire metadata schema. Can we please please please not do this? Very seriously, this is going to make our DocDB painful and it ultimately undermines having the schema in the first place.
• [x] eg. calibrations should be in the calibration field
• [ ] frame rate should be in the device (also, check the version of the schema you're using. The cameras in your device class have a max_frame_rate which was changed a while ago) (Xinxin: not clear to me; change the rig about the max_frame_rate)
• [ ] Your frame_num item specifically is blocking upload to DocDB. You can't have fields with ".csv" in the name. This information should live in the quality_control class (which was only recently added to the schema, but that's where it should live) (Alex: the .csv name is fixed, but not moved to quality_control)
• [x] adding software name to the stimulus_epochs

[alexpiet]

• [ ] Reward delivery cannot be null. How are you delivery rewards without it?
  • We can record the (x,y,z) coordinates of the motor stage at the start of the session
  • variable_position will be True
  • make sure relative position between spouts is known
  • "reference position" should be set on the first day. in a neutral position

[alexpiet]

• [x] Your session_end_time is before the end of one of your data streams. The end time should be the last time of all data streams and stimulus epochs.
  • behavior start/stop is "stimulus epochs" (continued licking after session is still part of stimulus epochs)
  • video/ephys is a "data stream"
  • session start/stop is earliest data_stream/stimulus_epoch start, and latest stop

[XX-Yin]

• [x] "Lick sensor left" and "Lick sensor right" are not daq_names for your rig and thus can't be listed as under daq_names. (Xinxin: the new lick detector should be included in the daq_names? Janelia should be removed from daq names)
• [x] Camera names don't match the names in the rig. They must match. There is a utility function you can use to make sure that the names align between rig and session.
• [x] Reward delivery cannot be null. How are you delivery rewards without it? (Xinxin: we don't have the lickspout reference area; need a reference area and reference coordinates collected from the GUI)
• [x] Your session_end_time is before the end of one of your data streams. The end time should be the last time of all data streams and stimulus epochs. (Xinxin: the session_end_time here is the end of the behavior session. For each data stream, there is a separate start and end time. Behavior should go to the stimulus_epoch) Stimulus Epoch
• [x] Missing stimulus_device_name
• [x] These output parameters and task parameters duplicate almost the entire metadata schema. Can we please please please not do this? Very seriously, this is going to make our DocDB painful and it ultimately undermines having the schema in the first place.
• [x] eg. calibrations should be in the calibration field
• [x] frame rate should be in the device (also, check the version of the schema you're using. The cameras in your device class have a max_frame_rate which was changed a while ago) (Xinxin: not clear to me; change the rig about the max_frame_rate)
• [x] Your frame_num item specifically is blocking upload to DocDB. You can't have fields with ".csv" in the name. This information should live in the quality_control class (which was only recently added to the schema, but that's where it should live) (Alex: the .csv name is fixed, but not moved to quality_control; remove them from the session)
• [x] adding software name to the stimulus_epochs
• [x] mismatch harp names

[alexpiet]

• [ ] adopt compatibility check between session and rig (this tool already exists)