At the end of the day, the written/human-readable specification should use the reserved terms MUST, SHOULD/MAY etc. should have exact parity with the levels of messages that Psych-DS validation produces (i.e. Error, Warning, Suggestion.)
We have also discussed just tentatively how to display these messages to users in ways that are both informative and not overwhelming, maybe using a "Levels" system, i.e.
Level 1: If a very basic thing is missing (no dataset_description.json, no data/ folder with at least one CSV inside), instruct the user how to do that thing rather than displaying all of the resulting downstream errors. Maybe you selected the wrong root directory, for instance!
Level 2: If the basics are present but off-kilter enough that very many downstream errors are likely, stop and help the user get them into shape before giving exhaustive errors. Examples:
"You have a dataset_description.json, but it has a syntax error so it cannot be parsed. Use [X tool] to help you fix this before proceeding."
"You have XLS and TSV files in your data/ directory and you might probably want them to be CSV. Fix this by..."
"You have IMG/other non-tabular files in your data/ directory and may want to move them or add them to .psychds-ignore. You can proceed, but Psych-DS will ignore these files from here on out."
Level 3: As a user, I am now equipped to understand the basic kinds of error messages I'm seeing, and if I am using a template I haven't modified/only changing some things so far, I can understand what those error messages mean.
At the end of the day, the written/human-readable specification should use the reserved terms MUST, SHOULD/MAY etc. should have exact parity with the levels of messages that Psych-DS validation produces (i.e. Error, Warning, Suggestion.)
We have also discussed just tentatively how to display these messages to users in ways that are both informative and not overwhelming, maybe using a "Levels" system, i.e.
Level 1: If a very basic thing is missing (no
dataset_description.json, nodata/folder with at least one CSV inside), instruct the user how to do that thing rather than displaying all of the resulting downstream errors. Maybe you selected the wrong root directory, for instance!Level 2: If the basics are present but off-kilter enough that very many downstream errors are likely, stop and help the user get them into shape before giving exhaustive errors. Examples:
"You have a dataset_description.json, but it has a syntax error so it cannot be parsed. Use [X tool] to help you fix this before proceeding."
"You have XLS and TSV files in your
data/directory and you might probably want them to be CSV. Fix this by...""You have IMG/other non-tabular files in your
data/directory and may want to move them or add them to.psychds-ignore. You can proceed, but Psych-DS will ignore these files from here on out."