Closed satra closed 2 years ago
@satra These errors are caught and responded to by falling back to get_default_metadata()
. Is attempting to open files as NWB by default actually a problem?
Also, I find it hard to believe that this is only happening on Python 3.7; I'm confident I've seen it on Python 3.9 as well.
i just saw this in the most recent PR for the CI test #694
@satra And the problem is...?
py3.7 fails a test with the error i posted in this issue.
@satra No, that error has nothing to do with the failure. The test failed because a to-be-published test Dandiset failed validation. (To be precise, what happened was that the server classified the Dandiset as "Invalid" with the message "1 assets have not been validated yet", which suggests a bug on the server's end.)
@jwodder - indeed - i did not scroll up beyond the ERROR
log.
regarding the failure, do you know if validation finished and then you tried to publish? posting an asset -> triggers validation -> which updates publishability. the question on the server's end is, is it validating? or is it stuck validating?
@satra The ValueError
is raised before actually trying to publish; the publish()
method starts out by querying the version's /info/
endpoint until the status is either "Valid" or "Invalid", and in the latter case it raises a ValueError
.
right, but do you know if the validation process has actually completed? the validation endpoint should give more info. as in do you:
POST asset wait for validation to complete (on a test file this should happen quickly, but not instantaneously) check for validity then try to publish
@satra Check for validity how? The only way I know is to query /dandisets/{dandiset__pk}/versions/{version}/info/
and check the "status" field, which is what the code does. If you're referring to the /uploads/{upload_id}/validate/
endpoint, I don't believe that's bothered with asset metadata validation for a while.
@dchiquito
/info/
endpoint returned a status of "Invalid" with a validation_error
of "1 assets have not been validated yet"; that seems to be a bug.@jwodder - did you see the example above?
@satra Yes, after I commented.
@satra dandi-cli now only treats files with .nwb
extension as NWB, and Dandiset metadata is checked for validity before publishing. Can this issue therefore be considered resolved?
perhaps this is already being tackled in a PR. given that this is specific to 3.7 may be worth checking what version is available. the code suggests that every file is being opened using nwb?
however, given that the blob store doesn't have any file extensions, we do need some better evaluation of a file before opening it. we can use the extension if available, but fall back on some other options if not.