clean up metadata and user interface

[jihyunbak]

Description and related issues

• update metadata paths, to match a metadata PR BouchardLab/NSDSLab-NWB-metadata#9
• for legacy blocks, add old experiment note whenever available
• update bad block checking logic (#71)
• make sure that all metadata items are either used or kept as "extra metadata" (#103, #100)
• clean up electrode location metadata (#101)
• add missing source_script_file_name that previously gave a warning
• update generate_nwb.py and other test scripts

Close #71 Close #94 (this TIMIT test block is now detected as a bad block, and stopped before stim processing) Close #100 (temporary solution - for now adding as scratch data) Close #101 Close #103

Checklist:

• [ ] All tests pass on catscan: run pytest --basetemp=tmp -sv -n 8 tests on catscan from the root directory
  • with RESAMPLE_DATA = False, all test blocks in test_catscan.py passed
  • with RESAMPLE_DATA = True, RVG16_B06, RVG16_B07, R56_B10 failed with ArrayMemoryError (also see issue #93)
• [x] If needed, docs have been update: docs/source has been updated for any added, moved, or removed files
  • (N/A)
• [x] Docs build with no errors: run make clean & make html from the docs folder
• [ ] No python formatting errors: run flake8 nsds_lab_to_nwb tests from the root directory

[jthermiz]

@jihyunbak I get the below error when trying to run this branch. I don't get this error with main the branch.

Command

python scripts/generate_nwb.py --save_path /clusterfs/NSDS_data/jhermiz --block_metadata_path /clusterfs/NSDS_data/raw/RVG16/RVG16_B01/RVG16_B01.yaml RVG16_B01

Error

Traceback (most recent call last):
  File "scripts/generate_nwb.py", line 85, in <module>
    write_nwb=write_nwb)
  File "/home/jhermiz/software/nsds_lab_to_nwb/nsds_lab_to_nwb/convert_block.py", line 43, in convert_block
    use_htk=use_htk)
  File "/home/jhermiz/software/nsds_lab_to_nwb/nsds_lab_to_nwb/nwb_builder.py", line 82, in __init__
    self.metadata = self._collect_nwb_metadata()
  File "/home/jhermiz/software/nsds_lab_to_nwb/nsds_lab_to_nwb/nwb_builder.py", line 140, in _collect_nwb_metadata
    return self.metadata_manager.extract_metadata()
  File "/home/jhermiz/software/nsds_lab_to_nwb/nsds_lab_to_nwb/metadata/metadata_manager.py", line 384, in extract_metadata
    metadata = self._extract(metadata_input)
  File "/home/jhermiz/software/nsds_lab_to_nwb/nsds_lab_to_nwb/metadata/metadata_manager.py", line 414, in _extract
    self.__load_stimulus_info(value)
  File "/home/jhermiz/software/nsds_lab_to_nwb/nsds_lab_to_nwb/metadata/metadata_manager.py", line 453, in __load_stimulus_info
    stimulus_metadata.update(read_yaml(stim_yaml_path))
  File "/home/jhermiz/software/nsds_lab_to_nwb/nsds_lab_to_nwb/common/io.py", line 31, in read_yaml
    with open(file_path, 'r') as stream:
FileNotFoundError: [Errno 2] No such file or directory: '/home/jhermiz/software/NSDSLab-NWB-metadata/auditory/stimulus/wn2.yaml'

[jthermiz]

Should the sub-directories include a yaml folder?

What this branch expects: auditory/stimulus/wn2.yaml

What my NSDSLab-NWB-metadata path actually looks like: auditory/yaml/stimulus/wn2.yaml

[jihyunbak]

Yeah sorry for the inconvenience, I rearranged the metadata folder in a PR there which is currently open and ready for review (maybe I shouldn’t have changed the existing structure…)

On Sat, Jul 31, 2021 at 13:47 John Hermiz @.***> wrote:

Should the sub-directories include a yaml folder?

What this branch expects: auditory/stimulus/wn2.yaml

What my NSDSLab-NWB-metadata path actually looks like: auditory/yaml/stimulus/wn2.yaml

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/BouchardLab/nsds_lab_to_nwb/pull/105#issuecomment-890402336, or unsubscribe https://github.com/notifications/unsubscribe-auth/AJLF3437B3ZHOSZHKBLTU3TT2RONBANCNFSM5BBU2H6Q .

[jthermiz]

Ok checking out the open PR in the metadatafolder resolved it. Thanks! I won't be able to review the other aspects of this PR until next week.