ValueSet-Tools: When uploading to cset name that already exists, add new version ONLY IF DIFFERENT

[Sigfried]

I'm not sure what the current behavior is, but we need our software to do something reasonable if we try to add the same cset twice, if we add a different cset with the same name, or if we add a genuinely new version of an existing existing cset.

Should probably:

• [ ] Check if it's identical (codes and metadata the same) and return an explanatory error to the user.
• [ ] Check if it's different, return an error to the user along with a clear explanation of how the new cset differs from the old.
• [ ] If the user is sure they want to add a new version, give them a way to do that. Optionally return the same report of how it differs from the existing version.

[joeflack4]

Very good idea.

[Sigfried]

@stephanieshong, you asked me to wait on this work until Push enclave_wrangler code so Siggie can work is done. Is there anything I can do to help?

[stephanieshong]

I do not think you need to do this elaborate check prior to uploading since there is already a UI tool within the enclave to do this analysis. And it would very difficult to decide without a clinician looking at the expressionItem which descendant should be included or not. This is usually a clinician's review step with the clinical knowledge.

Final metadata check is done from the UI, currently user cannot make a draft version a non-draft version.

[joeflack4]

From my perspective, if we have a bunch of quality control checks we can run prior to uploading, that's not a bad idea in theory. From what I was thinking that Siggie meant, it would be easy to check if a new version we're uploading is exactly the same as an old version; wouldn't need a clinician to check; the script would just check the codes or any other fields we think are critical to check, and make sure that they're not exactly the same.

[Sigfried]

Yes. The point of this is just to make it so you can use this tool to add concept sets and not have to worry if you already added it; the software will figure it out for you. If you did already add it, it will tell you that and you're done. If not, you're probably trying to add a new version on purpose -- but, perhaps you didn't even know that a concept set with that name existed, so you can check and make sure you're not clobbering something you shouldn't... etc. So, as Joe said, this is all prior to looking at the clinical terminology issues.

[stephanieshong]

The following condition only applies to the enclave versions. Certain use cases that will be important to consider:

1. If the archive flag is set than cs is not displayed in the UI, however it still exist in the cs dataset in the enclave. So we would need to check for this flag not just against the cs container name(key field in the Enclave)
2. if the draft version exist, would it be possible to validated against a draft version? Not sure if you can check against the draft version If the draft version already exist should we validate against a draft version? or check against the none-draft version? Currently, you can add additional draft versions to the concept set.
3. I think the most useful test case would be to validate against the draft version that has been already uploaded such that we can just make the draft version to non-draft without adding additional concept sets.
4. Currently, UI confirmation is required before the draft version can become non-draft version.