migration error - imageDataFormats

[martinschorb]

Hi,

I run into the following error when trying to migrate a project:

Traceback (most recent call last):
  File "/Users/schorb/miniconda3/envs/mobie/bin/mobie.migrate_project", line 33, in <module>
    sys.exit(load_entry_point('mobie', 'console_scripts', 'mobie.migrate_project')())
  File "/Users/schorb/code/mobie-utils-python/mobie/migration/migrate_project.py", line 35, in main
    migrate_project_v3(args.root)
  File "/Users/schorb/code/mobie-utils-python/mobie/migration/migrate_v3/migrate_project.py", line 27, in migrate_project
    validate_project(root, require_local_data=False, require_remote_data=False)
  File "/Users/schorb/code/mobie-utils-python/mobie/validation/project.py", line 42, in validate_project
    validate_with_schema(project_metadata, "project")
  File "/Users/schorb/code/mobie-utils-python/mobie/validation/utils.py", line 51, in validate_with_schema
    jsonschema.validate(instance=metadata, schema=schema)
  File "/Users/schorb/miniconda3/envs/mobie/lib/python3.8/site-packages/jsonschema/validators.py", line 1110, in validate
    raise error
jsonschema.exceptions.ValidationError: 'imageDataFormats' is a required property

Failed validating 'required' in schema:
    {'$id': 'https://raw.githubusercontent.com/mobie/mobie.github.io/master/schema/project.schema.json',
     '$schema': 'http://json-schema.org/draft-07/schema',
     'additionalProperties': False,
     'description': 'Schema describing a MoBIE project',
     'properties': {'datasets': {'description': 'The list of datasets '
                                                'available in this '
                                                'project.',
                                 'items': {'$ref': 'view.schema.json#/definitions/name',
                                           'type': 'string'},
                                 'minItems': 1,
                                 'type': 'array'},
                    'defaultDataset': {'description': 'The default dataset '
                                                      'for this project.',
                                       'type': 'string'},
                    'description': {'desription': 'The description.',
                                    'type': 'string'},
                    'imageDataFormats': {'description': 'The available '
                                                        'data formats for '
                                                        'this project.',
                                         'items': {'enum': ['bdv.hdf5',
                                                            'bdv.n5',
                                                            'bdv.n5.s3',
                                                            'bdv.ome.zarr',
                                                            'bdv.ome.zarr.s3',
                                                            'ome.zarr',
                                                            'ome.zarr.s3',
                                                            'openOrganelle.s3'],
                                                   'type': 'string'},
                                         'minItems': 1,
                                         'type': 'array'},
                    'references': {'description': 'Publication or other '
                                                  'references for this '
                                                  'project',
                                   'items': {'type': 'string'},
                                   'type': 'array'},
                    'specVersion': {'description': 'The MoBIE spec version '
                                                   'of this project.',
                                    'pattern': '0\\.2\\.\\d+',
                                    'type': 'string'}},
     'required': ['datasets',
                  'defaultDataset',
                  'specVersion',
                  'imageDataFormats'],
     'title': 'MoBIEProjectSchema'}

However, the project validates:

mobie.validate_project -i data -d 1 -r 0
...
The project at data is a valid MoBIE project.

and the imageDataFormat is present https://github.com/mobie/centriole-tomo-datasets/blob/main/data/project.json

[martinschorb]

The migration script then leaves a corrupted directory behind (which is OK if using git or a having made a backup).

[martinschorb]

Validation after running the migration (unsuccessfully) yields the exact same error.

Traceback (most recent call last):
  File "/Users/schorb/miniconda3/envs/mobie/bin/mobie.validate_project", line 33, in <module>
    sys.exit(load_entry_point('mobie', 'console_scripts', 'mobie.validate_project')())
  File "/Users/schorb/code/mobie-utils-python/mobie/validation/project.py", line 73, in main
    validate_project(
  File "/Users/schorb/code/mobie-utils-python/mobie/validation/project.py", line 42, in validate_project
    validate_with_schema(project_metadata, "project")
  File "/Users/schorb/code/mobie-utils-python/mobie/validation/utils.py", line 51, in validate_with_schema
    jsonschema.validate(instance=metadata, schema=schema)
  File "/Users/schorb/miniconda3/envs/mobie/lib/python3.8/site-packages/jsonschema/validators.py", line 1110, in validate
    raise error
jsonschema.exceptions.ValidationError: 'imageDataFormats' is a required property

Failed validating 'required' in schema:
    {'$id': 'https://raw.githubusercontent.com/mobie/mobie.github.io/master/schema/project.schema.json',
     '$schema': 'http://json-schema.org/draft-07/schema',
     'additionalProperties': False,
     'description': 'Schema describing a MoBIE project',
     'properties': {'datasets': {'description': 'The list of datasets '
                                                'available in this '
                                                'project.',
                                 'items': {'$ref': 'view.schema.json#/definitions/name',
                                           'type': 'string'},
                                 'minItems': 1,
                                 'type': 'array'},
                    'defaultDataset': {'description': 'The default dataset '
                                                      'for this project.',
                                       'type': 'string'},
                    'description': {'desription': 'The description.',
                                    'type': 'string'},
                    'imageDataFormats': {'description': 'The available '
                                                        'data formats for '
                                                        'this project.',
                                         'items': {'enum': ['bdv.hdf5',
                                                            'bdv.n5',
                                                            'bdv.n5.s3',
                                                            'bdv.ome.zarr',
                                                            'bdv.ome.zarr.s3',
                                                            'ome.zarr',
                                                            'ome.zarr.s3',
                                                            'openOrganelle.s3'],
                                                   'type': 'string'},
                                         'minItems': 1,
                                         'type': 'array'},
                    'references': {'description': 'Publication or other '
                                                  'references for this '
                                                  'project',
                                   'items': {'type': 'string'},
                                   'type': 'array'},
                    'specVersion': {'description': 'The MoBIE spec version '
                                                   'of this project.',
                                    'pattern': '0\\.2\\.\\d+',
                                    'type': 'string'}},
     'required': ['datasets',
                  'defaultDataset',
                  'specVersion',
                  'imageDataFormats'],
     'title': 'MoBIEProjectSchema'}

On instance:
    {'datasets': ['tomo'], 'defaultDataset': 'tomo', 'specVersion': '0.3.0'}

and indeed after the migration script, project.json looks like this:

{
  "datasets": [
    "tomo"
  ],
  "defaultDataset": "tomo",
  "specVersion": "0.3.0"
}

[martinschorb]

manually adding the missing lines to project.json and then validating gives me:

jsonschema.exceptions.ValidationError: '0.3.0' does not match '0\\.2\\.\\d+'

Failed validating 'pattern' in schema['properties']['specVersion']:
    {'description': 'The MoBIE spec version of this project.',
     'pattern': '0\\.2\\.\\d+',
     'type': 'string'}

On instance['specVersion']:
    '0.3.0'

Is the validation not updated yet? This is from the current master branch.

>>> mobie.__version__
'0.3.2'

[martinschorb]

OK, the good old "remove old cached specs" issue.

Have added a couple of lines here to give that hint...

https://github.com/mobie/mobie-utils-python/pull/90

[constantinpape]

Thanks for #90. I forgot to mention that.

The migration script then leaves a corrupted directory behind (which is OK if using git or a having made a backup).

Yes, that's unfortunately true. I recommend to run it in git or do a backup of the project beforehand in the README. For now, I have decided against making a backup file, since then this would need to be cleaned up as well... (And for me the migration script ran very stable.)