dandi / dandi-cli

DANDI command line client to facilitate common operations

https://dandi.readthedocs.io/

Apache License 2.0

21 stars 25 forks source link

Make validation on local assets not require identifier and contentUrl #369

Closed yarikoptic closed 3 years ago

yarikoptic commented 3 years ago

following brief (not-yet-)discussion with @satra on slack, our AssetMeta record already has some fields which are known/set only on the server side:

identifier: UUID4 = Field(readOnly=True, nskey="schema")
contentUrl: Optional[List[HttpUrl]] = Field(None, readOnly=True, nskey="schema")
... may be more ...

Also there is

path: str = Field(None, nskey="dandi") which we do not provide as part of the metadata record but rather as parameter to PUT call. Since in current state of things a new asset UUID is minted only upon publish, I think we should just upload metadata record with path, hence https://github.com/dandi/dandi-api/issues/91 but I could be shown that it is a bad idea

jwodder commented 3 years ago

Is is strictly necessary for these fields to be stored under "metadata" instead of at the top-level of the asset object alongside created etc.?

satra commented 3 years ago

we can add anything at the top level for archive interaction, but the metadata field has to correspond to our schema and should allow us to change the schema without database migrations.

yarikoptic commented 3 years ago

but it seems we need to "separate" / subclass, or otherwise annotate schema for a local asset -vs- asset on the server.

satra commented 3 years ago

you need to decide which fields to check for locally during validation. it will be a bit more complicated to separate out fields, since many are shared through the commonmodel between dandiset and asset.

yarikoptic commented 3 years ago

373 initiated the approach where could use BareAssetMeta (instead of AssetMeta) for validation of the local assets.

Otherwise validation of any asset would fail ATM, which leads to e.g.

dandi@drogon:/mnt/backup/dandi/dandi-api-datasets$ git grep -B1 'field required (type=value_error.missing)' | grep identifier | nl | tail
  6500  000053/VALIDATION.errors-identifier
  6501  000053/VALIDATION.errors-identifier
  6502  000053/VALIDATION.errors-identifier
  6503  000053/VALIDATION.errors-identifier
  6504  000053/VALIDATION.errors-identifier
  6505  000053/VALIDATION.errors-identifier
  6506  000053/VALIDATION.errors-identifier
  6507  000053/VALIDATION.errors-identifier
  6508  000053/VALIDATION.errors-identifier
  6509  000053/VALIDATION.errors-identifier

fails.