[Bug]: Value of type `types.core.ElementIdentifiers` cannot be converted to type `types.hdmf_common.ElementIdentifiers`

[yarikoptic]

What happened?

in the fresh run of https://github.com/dandi/dandisets-healthstatus saw in the logs

following error

```shell Asset: sub-P10HMH/sub-P10HMH_ses-20060901_ecephys+image.nwb Output: Error using assert Unexpected properties {help}. Your schema version may be incompatible with the file. Consider checking the schema version of the file with `util.getSchemaVersion(filename)` and comparing with the YAML namespace version present in nwb-schema/core/nwb.namespace.yaml Error in types.util.checkUnset (line 13) assert(isempty(dropped),... ... ```

but when I reproduced "from scratch" using https://github.com/dandi/dandisets-healthstatus/blob/main/code/checknwb.sh got for both released and master version:

+ git describe --tags
v2.6.0.0-3-g16a5411
+ matlab -nodesktop -sd /mnt/backup/tmp/dandi/matnwb-fS6cSkk/matnwb -batch 'generateCore()'
+ MATLABPATH=/mnt/backup/tmp/dandi/matnwb-fS6cSkk/matnwb:/mnt/backup/tmp/dandi/matnwb-fS6cSkk/matnwb/../out
+ time matlab -nodesktop -batch 'nwb = nwbRead('\''../000004/sub-P10HMH/sub-P10HMH_ses-20060901_ecephys+image.nwb'\'', '\''savedir'\'', '\''../out'\'')'
Error using assert
Value of type `types.core.ElementIdentifiers` cannot be converted to type
`types.hdmf_common.ElementIdentifiers`.

Error in types.util.correctType (line 60)
        assert(isa(val, type), ...

Error in types.util.checkDtype (line 96)
correctedValue = types.util.correctType(val, type);

Error in types.hdmf_common.DynamicTable/validate_id (line 90)
        val = types.util.checkDtype('id', 'types.hdmf_common.ElementIdentifiers', val);

Error in types.hdmf_common.DynamicTable/set.id (line 46)
        obj.id = obj.validate_id(val);

Error in types.hdmf_common.DynamicTable (line 33)
        obj.id = p.Results.id;

Error in types.core.TimeIntervals (line 21)
        obj = obj@types.hdmf_common.DynamicTable(varargin{:});

Error in io.parseGroup (line 85)
    parsed = eval([Type.typename '(kwargs{:})']);

Error in io.parseGroup (line 38)
    subg = io.parseGroup(filename, group, Blacklist);

Error in io.parseGroup (line 38)
    subg = io.parseGroup(filename, group, Blacklist);

Error in nwbRead (line 59)
nwb = io.parseGroup(filename, h5info(filename), Blacklist);

Command exited with non-zero status 1

Does initial message state that we have something outdated while doing healthcheck sweep? (attn @jwodder)?

How to open such a file using matnwb?

Steps to Reproduce

get the script and datalad

./checknwb.sh 000004 sub-P10HMH/sub-P10HMH_ses-20060901_ecephys+image.nwb v2.6.0.0 master

Error Message

No response

Operating System

Linux

Matlab Version

R2022b Update 1 (9.13.0.2080170) 64-bit (glnxa64)

Code of Conduct

• [X] I agree to follow this project's Code of Conduct
• [X] Have you ensured this bug was not already reported?

[yarikoptic]

BTW, message says to look at nwb-schema/core/nwb.namespace.yaml but there we have all those versions under https://github.com/NeurodataWithoutBorders/matnwb/tree/master/nwb-schema, not core/ (per each version). So what should be checked -- just availability of such a version folder under nwb-schema/?

FWIW

(base) dandi@drogon:/mnt/backup/tmp/dandi/matnwb-fS6cSkk/matnwb/nwb-schema$  MATLABPATH=/mnt/backup/tmp/dandi/matnwb-fS6cSkk/matnwb:/mnt/backup/tmp/dandi/matnwb-fS6cSkk/matnwb/../out matlab -nodesktop -batch "util.getSchemaVersion('../../000004/sub-P10HMH/sub-P10HMH_ses-20060901_ecephys+image.nwb')"

ans =

    '2.1.0'

so there is https://github.com/NeurodataWithoutBorders/matnwb/blob/master/nwb-schema/2.1.0/core/nwb.namespace.yaml

[lawrence-mbf]

Hi @yarikoptic, Can you actually remove the generateCore from the read script and rerun? Sometime after version 2.1.0, the ElementIdentifiers class was moved from core to the hdmf-common namespace. This error is usually an indicator of a namespace version conflict which is probably what's happening here.

[yarikoptic]

yes, commenting that line out makes it pass:

```shell (base) dandi@drogon:~/cronlib/dandisets-healthstatus/code$ git diff checknwb.sh diff --git a/code/checknwb.sh b/code/checknwb.sh index 3919596..496906d 100755 --- a/code/checknwb.sh +++ b/code/checknwb.sh @@ -27,7 +27,7 @@ set -x for v in "$@"; do git checkout $v; echo "Running using: "; git describe --tags; - matlab -nodesktop -sd $PWD -batch 'generateCore()'; + # matlab -nodesktop -sd $PWD -batch 'generateCore()'; MATLABPATH=$PWD:$PWD/../out time matlab -nodesktop -batch "f='../$ds/$f'; disp(util.getSchemaVersion(f)); nwb = nwbRead(f, 'savedir', '../out')" || echo "Exited with $?" done (base) dandi@drogon:~/cronlib/dandisets-healthstatus/code$ ./checknwb.sh 000004 sub-P10HMH/sub-P10HMH_ses-20060901_ecephys+image.nwb v2.6.0.0 /tmp/matnwb-QUmF6dy Cloning into '000004'... remote: Enumerating objects: 2334, done. remote: Counting objects: 100% (2334/2334), done. remote: Compressing objects: 100% (1083/1083), done. remote: Total 2334 (delta 988), reused 2101 (delta 755), pack-reused 0 Receiving objects: 100% (2334/2334), 283.60 KiB | 2.31 MiB/s, done. Resolving deltas: 100% (988/988), done. Version of the dandiset: 0.220126.1852-3-gd96cb6e Remote origin not usable by git-annex; setting annex-ignore https://github.com/dandisets/000004/config download failed: Not Found get sub-P10HMH/sub-P10HMH_ses-20060901_ecephys+image.nwb (from web...) ok (recording state in git...) Cloning into 'matnwb'... remote: Enumerating objects: 8841, done. remote: Counting objects: 100% (2004/2004), done. remote: Compressing objects: 100% (816/816), done. remote: Total 8841 (delta 1328), reused 1619 (delta 1184), pack-reused 6837 Receiving objects: 100% (8841/8841), 27.43 MiB | 24.49 MiB/s, done. Resolving deltas: 100% (5703/5703), done. + for v in "$@" + git checkout v2.6.0.0 Note: switching to 'v2.6.0.0'. You are in 'detached HEAD' state. You can look around, make experimental changes and commit them, and you can discard any commits you make in this state without impacting any branches by switching back to a branch. If you want to create a new branch to retain commits you create, you may do so (now or later) by using -c with the switch command. Example: git switch -c Or undo this operation with: git switch - Turn off this advice by setting config variable advice.detachedHead to false HEAD is now at 4bb8ef2 Merge pull request #482 from NeurodataWithoutBorders/schema-260-release + echo 'Running using: ' Running using: + git describe --tags v2.6.0.0 + MATLABPATH=/tmp/matnwb-QUmF6dy/matnwb:/tmp/matnwb-QUmF6dy/matnwb/../out + time matlab -nodesktop -batch 'f='\''../000004/sub-P10HMH/sub-P10HMH_ses-20060901_ecephys+image.nwb'\''; disp(util.getSchemaVersion(f)); nwb = nwbRead(f, '\''savedir'\'', '\''../out'\'')' 2.1.0 nwb = NwbFile with properties: nwb_version: '2.1.0' file_create_date: [1x1 types.untyped.DataStub] general_devices: [1x1 types.untyped.Set] general_source_script_file_name: [] identifier: 'H10_7' session_description: 'New/Old recognition task for ID: 7. ' session_start_time: 2006-09-01T03:00:00.000000-04:00 timestamps_reference_time: 2006-09-01T03:00:00.000000-04:00 acquisition: [2x1 types.untyped.Set] analysis: [0x1 types.untyped.Set] general: [0x1 types.untyped.Set] general_data_collection: 'learning: 80, recognition: 81' general_experiment_description: 'The data contained within this file describes a new/old recogntion task performed in patients with intractable epilepsy implanted with depth electrodes and Behnke-Fried microwires in the human Medical Temporal Lobe (MTL).' general_experimenter: [] general_extracellular_ephys: [18x1 types.untyped.Set] general_extracellular_ephys_electrodes: [1x1 types.core.DynamicTable] general_institution: 'Hunigton Memorial Hospital' general_intracellular_ephys: [0x1 types.untyped.Set] general_intracellular_ephys_filtering: [] general_intracellular_ephys_sweep_table: [] general_keywords: [1x1 types.untyped.DataStub] general_lab: 'Rutishauser' general_notes: [] general_optogenetics: [0x1 types.untyped.Set] general_optophysiology: [0x1 types.untyped.Set] general_pharmacology: [] general_protocol: [] general_related_publications: [1x1 types.untyped.DataStub] general_session_id: [] general_slices: [] general_source_script: [] general_stimulus: [] general_subject: [1x1 types.core.Subject] general_surgery: [] general_virus: [] intervals: [0x1 types.untyped.Set] intervals_epochs: [] intervals_invalid_times: [] intervals_trials: [1x1 types.core.TimeIntervals] processing: [0x1 types.untyped.Set] scratch: [0x1 types.untyped.Set] stimulus_presentation: [1x1 types.untyped.Set] stimulus_templates: [0x1 types.untyped.Set] units: [1x1 types.core.Units] 15.22user 0.84system 0:15.82elapsed 101%CPU (0avgtext+0avgdata 911528maxresident)k 1880inputs+1152outputs (15major+196234minor)pagefaults 0swaps + pwd /tmp/matnwb-QUmF6dy/matnwb ```

[yarikoptic]

FWIW, the run is ongoing but so far I see following similar issue(s) in following dandisets:

(dandisets-2) dandi@drogon:~/cronlib/dandisets-healthstatus$ grep -h 'cannot be converted to type' results/*/2023.02.0[2-9]*.log | sort | uniq -c
    224     Value of type `types.core.ElementIdentifiers` cannot be converted to type
    100     Value of type `types.hdmf_common.VectorData` cannot be converted to type

(dandisets-2) dandi@drogon:~/cronlib/dandisets-healthstatus$ grep -l 'cannot be converted to type' results/*/2023.02.0[2-9]*.log
results/000004/2023.02.02.16.56.48_matnwb_nwbRead_errors.log
results/000010/2023.02.02.16.56.50_matnwb_nwbRead_errors.log
results/000011/2023.02.02.16.56.51_matnwb_nwbRead_errors.log
results/000039/2023.02.07.14.33.09_matnwb_nwbRead_errors.log

[yarikoptic]

are such errors of the same "type/cause"?

    Cannot define property 'strain' in class
    'types.ndx_aibs_ecephys.EcephysSpecimen' because the property has already been
    defined in the superclass 'types.core.Subject'.

(dandisets-2) dandi@drogon:~/cronlib/dandisets-healthstatus$ grep -l 'because the property has already been' results/*/2023.02.0[2-9]*.log
results/000021/2023.02.05.11.32.02_matnwb_nwbRead_errors.log
results/000022/2023.02.05.19.19.37_matnwb_nwbRead_errors.log

(dandisets-2) dandi@drogon:~/cronlib/dandisets-healthstatus$ grep -h 'because the property has already been' results/*/2023.02.0[2-9]*.log | sort | uniq -c
      8     'types.ndx_aibs_ecephys.EcephysProbe' because the property has already been
    371     'types.ndx_aibs_ecephys.EcephysSpecimen' because the property has already been

?

[lawrence-mbf]

are such errors of the same "type/cause"?

    Cannot define property 'strain' in class
    'types.ndx_aibs_ecephys.EcephysSpecimen' because the property has already been
    defined in the superclass 'types.core.Subject'.

(dandisets-2) dandi@drogon:~/cronlib/dandisets-healthstatus$ grep -l 'because the property has already been' results/*/2023.02.0[2-9]*.log
results/000021/2023.02.05.11.32.02_matnwb_nwbRead_errors.log
results/000022/2023.02.05.19.19.37_matnwb_nwbRead_errors.log

(dandisets-2) dandi@drogon:~/cronlib/dandisets-healthstatus$ grep -h 'because the property has already been' results/*/2023.02.0[2-9]*.log | sort | uniq -c
      8     'types.ndx_aibs_ecephys.EcephysProbe' because the property has already been
    371     'types.ndx_aibs_ecephys.EcephysSpecimen' because the property has already been

?

No this looks like either a schema error or something that requires a workaround in MATLAB.

[yarikoptic]

Please let me know if I should file them as a separate issue

[lawrence-mbf]

Hi, sorry for missing this yesterday. Yes, this looks like a separate issue, please submit a new one for this issue.

On 2023-02-08 19:57, Yaroslav Halchenko wrote:

Please let me know if I should file them as a separate issue