Draft checksum documentation (to go into OCL Docs) - Githubissues

OpenConceptLab / ocl_issues

Issues for all OCL repos. NOTE: Install ZenHub Browser Extension and request access to the OCL Roadmap board to view all issues and to contribute

4 stars 2 forks source link

Draft checksum documentation (to go into OCL Docs) #1811

Open paynejd opened 7 months ago

paynejd commented 7 months ago

@snyaggarwal

[x] Fields for standard/smart checksums for both concepts and mappings
[x] Description of algorithm for resource-level checksum
[x] Description of algorithm for repo-level checksum

snyaggarwal commented 7 months ago

Fields:

Concepts

Standard
- concept_class <str>
- datatype <str>
- retired <bool>
- external_id <str>
- extras <json>
- parent_concept_urls <list of str>
- child_concept_urls <list of str>
- names (all names/synonyms) <list>
- locale <str>
- locale_preferred <bool>
- name <str>
- name_type <str>
- descriptions <list>
- locale <str>
- locale_preferred <bool>
- description <str>
- description_type <str>
Smart
- concept_class <str>
- datatype <str>
- retired <bool>
- names (only FSNs) <list>
- locale <str>
- locale_preferred <bool>
- name <str>
- name_type <str>
Mappings
- Standard
- map_type <str>
- from_concept_code <str>
- to_concept_code <str>
- from_concept_name <str>
- to_concept_name <str>
- retired <bool>
- external_id <str>
- extras <json>
- Smart
- map_type <str>
- from_concept_code <str>
- to_concept_code <str>
- from_concept_name <str>
- to_concept_name <str>
- retired <bool>

Source/Version

Standard
- source_type <str>
- canonical_url
- custom_validation_schema <str>
- default_locale <str>
- supported_locales <list of str>
- website <str>
- hierarchy_meaning <str>
- extras <json>
Smart
- source_type <str>
- canonical_url <str>
- custom_validation_schema <str>
- default_locale <str>
- released <bool>
- retired <bool>

Collection/Version

Standard
- collection_type <str>
- canonical_url <str>
- custom_validation_schema <str>
- default_locale <str>
- supported_locales <list of str>
- website <str>
- extras <json>
Smart
- collection_type <str>
- canonical_url <str>
- custom_validation_schema <str>
- default_locale <str>
- released <bool>
- retired <bool>

Algorithm:

Cleanup
- Remove None values
- Ignore is_active=True and retired=False
- Ignore empty extras
Serialize
- Consider a list with a single object equal to a single object without the list, e.g. [element] vs element
- Sort items in the list
- Sort keys in dict
- convert UUID to str (simple str conversion for UUID('xxx-xxxx-..') to 'xxx-xxxx-..')
- convert the entire payload to str (JSON dump)
Generate MD5 hash from serialized str:
- encode in UTF-8
- apply MD5 hash

paynejd commented 7 months ago

[ ] Which resources have checksums?
[ ] How/when are checksums generated?
[ ] Where are checksums made available in the API?