Table to produce

[yarikoptic]

Example/specification of the table to produce

dandiset	version	schema_version	api_status	modified	pydantic	linkml
000727	0.240106.0043	0.6.4	Valid	2024-01-06T00:43:28.815333Z	4 (1 value_error, 3 string_pattern_mismatch)	14 (1 + 13)
000727	draft	0.6.4	Invalid	2024-05-18T17:08:27.003540Z	???

• [x] "api_status" as API returns in the listing of dandisets, e.g. "Valid" in both cases here:

  "most_recent_published_version": { "version": "0.240106.0043", "name": "Mapping the Neural Dynamics of Locomotion across the Drosophila Brain", "asset_count": 18, "size": 609568504598, "status": "Valid", "created": "2024-01-06T00:43:26.749476Z", "modified": "2024-01-06T00:43:28.815333Z" }, "draft_version": { "version": "draft", "name": "Mapping the Neural Dynamics of Locomotion across the Drosophila Brain", "asset_count": 18, "size": 609568504598, "status": "Invalid", "created": "2023-12-10T10:22:46.559944Z", "modified": "2024-05-18T17:08:27.003540Z" }

• [x] if no most_recent_published_version -- only draft
• [x] "modified" - also from API
• [x] if draft and published version have the same "modified" and "status", then do not produce DRAFT version row
• [x] schema_version -- the version of schema from metadata.json schemaVersion
• [x] for linkml validation outputs, check "message" and do following transformation:
  • apply following regex substitution (here in sed format): "s,.*\(is .*\) in \S.*,\1,g" which ATM produces across all dandisets:

    
    ❯ grep -h '"message"' validation-reports-dandi/*/linkml_validation_errs.json | sed -e "s,.*\(is .*\) in \S.*,\1,g" | sort -n | uniq -c | sort -n
    1 is not valid under any of the given schemas
    1     "message": "Additional properties are not allowed ('@context', 'datePublished', 'publishedBy' were unexpected) in /",
    179     "message": "Additional properties are not allowed ('@context', 'datePublished', 'doi', 'publishedBy' were unexpected) in /",
    1399 is not of type 'string'

- [x] and then for linkml reference, just count the number of such different errors. Most of the time it would be 2 ATM since they all have that "is not of type string" and "Additional propert

[yarikoptic]

this regex "s,.*\(is .*\) in \S.*,\1,g" is the "GNU sed" regex, so it escapes the group etc, in python it would be the difference, not sure in OSX

[candleindark]

All features requested for the table are complete in https://github.com/dandi/dandisets-linkml-status/pull/1. However, I want to remove the following one. The required condition in the feature is never true for any of the existing dandisets.

• [x] if draft and published version have the same "modified" and "status", then do not produce DRAFT version row

The modified timestamp for a draft version seems to be never the same as the corresponding latest published version. Take a look at the "000017" dandisets @ https://github.com/candleindark/dandisets-linkml-status/blob/init/dandi-reports/summary.md, and you will that the draft version is essentially the same as the latest published version, but they have different timestamp.

In place of the condition you suggested, we may just take away a draft version if it has the status of Published. However, my recommendation would be not taking away any draft version at all. Even for a draft version with the status of Published, its metadata is not exactly the same as the metadata of the corresponding latest published version.

[candleindark]

FTR: (per @yarikoptic's approval) The outputting of validation reports on draft version is now unconditional, i.e. the validation reports of the draft version of all dandiset will be outputted.