implement publication workflow to codes.wmo.int

tomkralidis commented 8 months ago

[x] update tables (@tomkralidis)
- add source column of a URL for values that are defined external to WMO
[x] investigate workflow to convert CSVs to RDF/TTL (@antje-s @josusky)
[x] update specification and examples to point to new/updated codelist values (@tomkralidis)

Examples:

josusky commented 8 months ago

I haven't fully figured out how the upload of TTL files to (ci).codes.wmo.int works, the script from "https://github.com/wmo-im/wmds" expects some JSON as input. On the other hand, I have figured out how to adjust makeWMDREntities.py to convert "our" CSVs. Problem is what parameters should we set in the TTL files. They are generated using a textual template that sets "identity", "notation", "label" and "description". So far I have used "Name" for all of them except "description", where I have used "Description" column. But I have no idea where to put our "Source" column.

josusky commented 8 months ago

Oh, only now have I noticed that @antje-s has created branch csv2registry, I am going to check what is new there.

josusky commented 8 months ago

OK, so she went further than me. Her version of the script puts "Source" into "skos:related" and she made custom versions of other WMDS scripts too.

tomkralidis commented 8 months ago

Great work here! Note that if the source column is not empty, this means that the codelist value is from an upstream controlled vocabulary. In which case we should use owl:sameAs instead.

tomkralidis commented 8 months ago

Notes from 2024-02-07 meeting with @antje-s / @josusky

TTL generation (@antje-s)
- CSV sources "Source" column: empty means local definition, else upstream/remote
- a non-empty "Source" column should produce a an owl:sameAs and skos:exactMatch or the URL in the concept template
- generated TTLs should not go into main branch, but a branch called publication
Upload to WMO Codes Registry (@josusky)
- determine interaction / payload specifics from registry API
@tomkralidis will review implementation thereafter

Next meeting is on 2024-02-09.

antje-s commented 8 months ago

Updated after our meeting today:

only add entry for source column if not empty or None
use "wis" as directory name in branch publication for TTLs
delete branch publication if exists, use always an new branch publication
deleted publish script files from branch "csv2registry"

I think we need for PR from csv2registry branch: 1).github/workflows/generateTTLandCommit.yml 2)scripts/makeWCMP2Entities.py 3)codelists/wcmp2-tables.csv

Many thanks for all the support, I have learned a lot again @josusky: if you want me to do anything further, please let me know

tomkralidis commented 8 months ago

Next steps:

consolidated PR with scripts to 1. generate TTLs and 2. publish to codes.wmo.int
clarify publication workflow
- main branch: CI/CD to ci.codes.wmo.int
- release: CI/CD or manual process? Needs discussion with @amilan17

tomkralidis commented 8 months ago

Implemented in #8. Excellent work @antje-s and @josusky ! Thank you for landing this so quickly.

amilan17 commented 8 months ago

The plan is to publish everything to the public codes.wmo.int registry with a status of experimental at the beginning of March. This will ensure that all of the links are active when members are reviewing the documents.

tomkralidis commented 8 months ago

PR update in #10 (see Notes section for required clarifications). Request for clarifications will be sent via email.

tomkralidis commented 8 months ago

Copying notes here for next steps:

GitHub Secrets for WMO Codes Registry publishing from GitHub Actions have not been set yet and need clarification/discussion
we need clarification on what/how to publish the "root" wis register. Doing so manually or automatically yields an "invalid" status

amilan17 commented 7 months ago

@tomkralidis

@david-i-berry suggests using something different than owl:sameAs when the code and definition are copied from another source. two ideas: https://www.w3.org/TR/rdf-schema/#ch_isdefinedby or https://www.dublincore.org/specifications/dublin-core/dcmi-terms/elements11/source/

tomkralidis commented 7 months ago

2024-03-11

implement category to enable visibility of codelists on codes.wmo.int homepage (UI)
CI to update ci.codes.wmo.int (status=experimental)
WMO to update codes.wmo.int (status=approved)

Actions:

@amilan17 @antje-s @josusky to meet with UK Met Office to discuss category lifecycle
@josusky @antje-s to change owl:sameAs to rdfs:isDefinedBy (https://github.com/wmo-im/wcmp2-codelists/blob/main/scripts/codelists2ttl.py)
@amilan17 should then be able to publish codes.wmo.int proper

tomkralidis commented 7 months ago

Update: this should be ready for publication to codes.wmo.int proper. @amilan17 let us know if this is successful, at which point we can close this issue.

david-i-berry commented 6 months ago

Note that the authenticate part of upload_changes needs to be modified to use port 443 at:

https://github.com/wmo-im/wcmp2-codelists/blob/6b9eab5173f0f798d56f329fba5eebcd743f69ac/scripts/upload_changes.py#L46

i.e.

url = f'{base_url}:443/system/security/apilogin'

tomkralidis commented 6 months ago

Thanks @david-i-berry. The result is the same whether 443 is explicitly set or not.

josusky commented 6 months ago

I agree with @tomkralidis, port 443 is default for HTTPS. Only if a non-standard port, let say 8181 is used, it needs to be explicitly set.

amilan17 commented 6 months ago

I was able to upload AFTER removing the global-service-type TTLs. For some reason, the software does not accept those, but the TTLS are identical to the accepted TTLS accept for the content referring to global-service-type... ?!

josusky commented 6 months ago

Hi Anna,

I am checking the issue with global-service type sub-register, the server is responding with with an internal error when I try to GET (read) the resource: $ curl https://ci.codes.wmo.int/wis/global-service-type/

Cannot invoke "com.epimorphics.registry.core.Description.getRoot()" because the return value of "com.epimorphics.registry.store.StoreAPI.getCurrentVersion (String)" is null

I will try to remove this sub-register and retry.

JanO

On Thu, Mar 28, 2024 at 4:13 PM Anna Milan @.***> wrote:

I was able to upload AFTER removing the global-service-type TTLs. For some reason, the software does not accept those, but the TTLS are identical to the accepted TTLS accept for the content referring to global-service-type... ?!

— Reply to this email directly, view it on GitHub https://github.com/wmo-im/wcmp2-codelists/issues/6#issuecomment-2025467861, or unsubscribe https://github.com/notifications/unsubscribe-auth/AFS4LF6LXUEVD4SDF5SDHW3Y2QQR3AVCNFSM6AAAAABCVCN44OVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMRVGQ3DOOBWGE . You are receiving this because you were mentioned.Message ID: @.***>

amilan17 commented 6 months ago

Yes, I get the same error when trying to upload

ValueError: Cannot upload to https://ci.codes.wmo.int/wis/global-service-type: 500 : Cannot invoke "com.epimorphics.registry.core.Description.getRoot()" because the return value of "com.epimorphics.registry.store.StoreAPI.getCurrentVersion(String)" is null

josusky commented 6 months ago

I am afraid that the removal of sub-register did not help, I am still getting the internal server error.

On Thu, Mar 28, 2024 at 4:31 PM Anna Milan @.***> wrote:

Yes, I get the same error when trying to upload

ValueError: Cannot upload to https://ci.codes.wmo.int/wis/global-service-type: 500 : Cannot invoke "com.epimorphics.registry.core.Description.getRoot()" because the return value of "com.epimorphics.registry.store.StoreAPI.getCurrentVersion(String)" is null

— Reply to this email directly, view it on GitHub https://github.com/wmo-im/wcmp2-codelists/issues/6#issuecomment-2025506050, or unsubscribe https://github.com/notifications/unsubscribe-auth/AFS4LF76Q5O6JJJOZZDSYE3Y2QSTRAVCNFSM6AAAAABCVCN44OVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMRVGUYDMMBVGA . You are receiving this because you were mentioned.Message ID: @.***>

josusky commented 6 months ago

The error message changed a bit over the weekend, now it reveals less of the internal information, but it is still an "Internal Server Error". I don't think we can fix this from our side. @amilan17 could you please contact the server administrator and request help?

BTW, I have created a register "wis2" (https://ci.codes.wmo.int/wis2) and successfully uploaded all codes to it.

amilan17 commented 6 months ago

@josusky how did you manage to successfully upload all codes? The error didn't prevent that on the wis2 registry?

josusky commented 6 months ago

It seems that the problem somehow related to the wis register.

amilan17 commented 6 months ago

Codelists are now published to https://codes.wmo.int/ with the status of Experimental.

wmo-im / wcmp2-codelists

implement publication workflow to codes.wmo.int #6