search

wmo-im / wis2-topic-hierarchy

WIS2 Topic Hierarchy
https://wmo-im.github.io/wis2-topic-hierarchy
Apache License 2.0
6 stars 4 forks source link

Modify workflow and/or scripts for uploading to the Codes Registry #175

Open amilan17 opened 2 months ago

amilan17 commented 2 months ago

Right now, all codes are updated when the "upload_changes.py" script and we should only update codes that have changed.

For WMDR codes, there is a "check_urls.py" script that will do a comparison of all differences and then generates JSON output of TTLs to POST and PUT.

amilan17 commented 2 months ago

Screen shot from ci.codes.wmo.int

image

josusky commented 3 weeks ago

I have added a "simple" comparison of local TTL files with the ones on the server. The result then looks like this:

Processing wis/topic-hierarchy.ttl Checking https://ci.codes.wmo.int/wis/topic-hierarchy/ - Existing entry, going to compare: Equal. Unchanged entry, nothing to do.

Note that this does not check for things that exist on the server (register) but are no longer in the local "repository", that is nothing is ever deleted.

amilan17 commented 2 weeks ago

@josusky thank you very much for this! Were you able to test it on ci.codes.wmo.int?

josusky commented 2 weeks ago

Yes, I was, the example of output above is from such testing. Though, feel free to re-test it yourself.

amilan17 commented 4 days ago

@josusky Thank you for working on this script. I was finally able to test it yesterday and confirm that only new and revised codes were included in the upload to the test codes registry.

However, I would like to request a modification to this workflow. Right now, the comparison and the post/put are both run in the same upload_changes.py script and I would like to have these two functions separated. In WMDR, there is an output that summarizes the codes that will be updated or added first. This is a useful step in the workflow, because sometimes the registry is difficult to modify and this allows for one more final review of what will be updated first.

josusky commented 3 days ago

I have slightly improved the text output of the script 'upload_changes.py'. A changed file in --dry-run mode now looks like this (showing the command and three files, the second one "earth-system-discipline.ttl" was changed locally):

python3 scripts/upload_changes.py <login> <password> test wis --status experimental --dry-run
Running upload against https://ci.codes.wmo.int
Authenticating at https://ci.codes.wmo.int/system/security/apilogin
Processing wis/topic-hierarchy.ttl
  Unchanged entry, nothing to do.
Processing wis/topic-hierarchy/earth-system-discipline.ttl
  Changed entry, will upload.
  HTTP PUT (dry run) to https://ci.codes.wmo.int/wis/topic-hierarchy/earth-system-discipline
Processing wis/topic-hierarchy/system.ttl
  Unchanged entry, nothing to do.

in verbose mode, the output for the changed file looks like this.

Processing wis/topic-hierarchy/earth-system-discipline.ttl
  Checking https://ci.codes.wmo.int/wis/topic-hierarchy/earth-system-discipline/ -  Existing entry, going to compare: Changed.
    http://purl.org/dc/terms/description: Seven high-level test change categories as defined by the WMO Unified Data Policy, Annex 1: 'atmospheric-composition', 'climate', 'cryosphere', 'hydrology', 'ocean', 'space-weather', or 'weather'
  Changed entry, will upload.
  HTTP PUT (dry run) to https://ci.codes.wmo.int/wis/topic-hierarchy/earth-system-discipline
    headers: {'Content-type': 'text/turtle; charset=UTF-8'}
    params: {'status': 'experimental', 'non-member-properties': 'true'}

that is, it also shows the changed element, in this case, it is "description".

amilan17 commented 9 hours ago

@josusky Thank for your help. Codes registry is now updated with the last WTH topics. http://codes.wmo.int/wis/topic-hierarchy