Building Codes and Rules API based on Named Graphs

[VladimirAlexiev]

Was Building Codes and Rules: based on GraphQL, or SPARQL and grlc

@beachtom @rickmakkinga @pasi-paasiala

Currently the API talks about access to a whole Regulation. But I think a key question for Building Codes and Rules (BCRL) is how to structure the content and how to fetch what you need. These are captured as RDF. I see two possible ways to query them and get them:

• GraphQL. We can serve JSON of pretty much the same form as the YAML
  • pro: can get exactly the data you need, easy to query, easy to do pagination and basic filters
  • cons: need to make a SOML schema, but that's not hard based on the ontology
• SPARQL and grlc. grcl is open software that exposes a set of SPARQL queries (parameterized using Basil convention) as API. Some description in https://github.com/w3c/sparql-dev/issues/57, and I got a book about it.
  • pro: can write as complex queries as needed
  • cons: harder to return deeply nested data, and in some cases may even need a sequence of queries

[beachtom]

So we could also expose an endpoint to allow a graph QL query - but I think we also need to retain simpler approaches

The API as I defined it so far allowes retrieval of entire documents, and sections/subsection based on REST only.

[VladimirAlexiev]

After the discussion today, @beachtom and I decided to provide a GraphQL API.

• to be generated from https://github.com/Accord-Project/aec3po ontology. Which ontology modules are needed for serving Regulations? 61k in 13 files. Here's a judgement (corrected by Tom):

• aec3po.ttl
• check_method.ttl
• checking_act.ttl
• compliance_verification_report.ttl
• data_requirement.ttl
• document.ttl
• evidence.ttl
• feature_of_interest.ttl
• legal_verifier.ttl
• model.ttl
• rase_statement.ttl
• statement.ttl
• table.ttl

• Which is the relation clause->subclause? Assuming it's hasPart, will add transitive prop hasPartTransitive to allow fetching the complete subtree.

[beachtom]

Hi Vladamir,

The used modules are:

aec3po.ttl check_method.ttl document.ttl rase_statement.ttl statement.ttl table.ttl

The two best examples are:

https://github.com/Accord-Project/bcrl/blob/main/FI-accessibility-experiment/Tom%20Example/FI2.yaml

and

https://github.com/Accord-Project/bcrl/blob/main/EE-Example/EE1-Example.yaml

But I can generate a YAML for any in this folder (but they have no RASE as of yet). let me know if you need more

https://github.com/Accord-Project/bcrl/tree/main/RegulatoryTexts

[VladimirAlexiev]

@nataschake https://ontotext.atlassian.net/issues/ACD-135

Let's use https://github.com/VladimirAlexiev/soml/tree/master/owl2soml (a perl script; there's a similar java module on Platform Workbench, but it might be slightly older)

We need to:

• [x] add a Makefile for all this below
• [x] concat the above 6 of the AEC3PO ontologies, and all vocabularies, and generate SOML (in the makefile)
• [x] add prop hasPartTransitive to allow fetching the complete subtree (GraphQL has no prop paths like (hasPart|hasInlinePart)*): see aec3po-fix.ttl
• [x] make a repo aec3po with subPropertyOf and transitiveOver reasoning (see https://graphdb.ontotext.com/documentation/10.5/rules-optimisations.html#transitiveover and next section) and loada all these ontologies
• [ ] Write some queries to check aec3po and post issues: props without domain, object props without range...
• [ ] inspect SOML it and make corrections (eg unattached props, imprecise cardinalities). Best to do it with a "fix.ttl", see owl2soml/eg/skos-fix.ttl
• [ ] add virtual inverses where bidirectional navigation is needed.

@Gonsco

• You can search on any field (EQ, NE, RE=regex etc)
• But typically you'd just fetch a regulation and all its clauses by ID (URL)
• You can dig deep into the regulation and fetch all clauses and their fields and relations. We'll write nested GraphQL queries, similar to Tom's YAML examples, to dig through the structure.
• It will return JSON, conformant to the GraphQL query shape

Goncal, do you have any other specific retrieval needs? (If we want to sound learned, we'll call this Competency Questions). Eg:

• Getting lists of stored regulations, with what metadata?
• Stats about number of clauses?
• What bidirectional navigation do you need? Eg do you ever need from Clause (child) to Regulation (parent)?

As for storing, let's use the YAML-JSONLD-RDF route.

• https://github.com/Accord-Project/bcrl includes a tool by Thamer to convert YAML-JSON
• If the JSON refers to the respective JSONLD context (which is properly served online), that's all we need: GraphDB can ingest it directly
• Else, I think there's another tool to convert JSON-RDF

[VladimirAlexiev]

@beachtom I looked at how SOML and GraphQL queries would look, and it's not so easy because of complications like dual nature of hasTarget (GraphQL doesn't support that).

We had a call with @Gonsco and agreed on the simplest approach, based on the SPARQL Graph Protocol:

• POST: overwrite a regulation named graph with new regulation content (Goncal only needs this)
• GET: get the content of a regulation named graph

Then we can add some SPARQL queries with grlc:

• List all regulations: with name, countries (forAdministrativeArea), mabve number of subsections
• List regulations for a given country (AdministrativeArea)

[beachtom]

So we also need to consider versions - i..e one document may have multiple versions - so i think a named graph would represent a given version of a given document?

We also need queries to retrieve a given section/subsection of a document/version

[VladimirAlexiev]

@Gonsco , @rlavikka , @k-breitenfelder , @rickmakkinga Do we need versions? Then each regulation needs some metadata as you find in many owl:Ontology:

• its version string,
• link to prev version
• link to latest (unversioned namespace) version

[beachtom]

Generally speaking release date is used as the version string - we then use replaces and replacedBy to form a link list of versions - so I think this covers it.

As we only have one version of any documents I haven't used this - but can easily add it.

[rickmakkinga]

Hi all,

Hopelessly behind with all the ACCORD mails....

But I agree with Tom, that release date should suffice as a version for documents.

But how does this work for individual clauses/rules? Let's say we have:

• Version A of rule document
  • Rule 1
  • Rule 2
  • Rule 3
• Version B of rule document
  • Rule 1 is the same as in version A
  • Rule 2 changed compared to version A
  • Rule 3 is the same as in version A

Do we now get 6 'rule versions' (1A, 2A, 3A, 1B, 2B, 3B) or just four (1A, 2A, 3A, 2B)?

Met vriendelijke groet | Best regards,

Rick Makkinga

Product Owner Clearly Suite

Samen maken wij projecten succesvol

Clearly | .Hub http://www.futureinsight.nl/clearly-hub | .Projects http://www.futureinsight.nl/clearly-projects | .BIM http://www.futureinsight.nl/clearly-bim | .3D-City http://www.futureinsight.nl/clearly-3d-city

Bekijk onze vernieuwde website Ga naar www.futureinsight.nl

FUTURE INSIGHT | www.futureinsight.nl | LinkedIn https://www.linkedin.com/company/future-insight-bv/

Grote Voort 207G | 8041 BK Zwolle | + 31 6 129 866 32

Op wo 3 apr 2024 om 14:41 schreef Tom Beach @.***>:

Generally speaking release date is used as the version string - we then use replaces and replacedBy to form a link list of versions - so I think this covers it.

As we only have one version of any documents I haven't used this - but can easily add it.

— Reply to this email directly, view it on GitHub https://github.com/Accord-Project/API-Development/issues/2#issuecomment-2034501373, or unsubscribe https://github.com/notifications/unsubscribe-auth/AF3H3SDMXB6IZMAYRJT5ERTY3P2H3AVCNFSM6AAAAABFBLVPPGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMZUGUYDCMZXGM . You are receiving this because you were mentioned.Message ID: @.***>

[rickmakkinga]

Hi all,

As long as you can in the end request individual rules/checks, it should all be fine.

And you can always put a REST API in front of the GraphQL (you will loose some flexibility probably) if needed. We do the same for Clearly.BIM. Everything uses GraphQL natively, but we have a REST API in front of practically all functions.

Met vriendelijke groet | Best regards,

Rick Makkinga

Product Owner Clearly Suite

Samen maken wij projecten succesvol

Clearly | .Hub http://www.futureinsight.nl/clearly-hub | .Projects http://www.futureinsight.nl/clearly-projects | .BIM http://www.futureinsight.nl/clearly-bim | .3D-City http://www.futureinsight.nl/clearly-3d-city

Bekijk onze vernieuwde website Ga naar www.futureinsight.nl

FUTURE INSIGHT | www.futureinsight.nl | LinkedIn https://www.linkedin.com/company/future-insight-bv/

Grote Voort 207G | 8041 BK Zwolle | + 31 6 129 866 32

Op wo 27 mrt 2024 om 13:36 schreef Tom Beach @.***>:

Hi Vladamir,

The used modules are:

aec3po.ttl check_method.ttl document.ttl rase_statement.ttl statement.ttl table.ttl

The two best examples are:

https://github.com/Accord-Project/bcrl/blob/main/FI-accessibility-experiment/Tom%20Example/FI2.yaml

and

https://github.com/Accord-Project/bcrl/blob/main/EE-Example/EE1-Example.yaml

But I can generate a YAML for any in this folder (but they have no RASE as of yet). let me know if you need more

https://github.com/Accord-Project/bcrl/tree/main/RegulatoryTexts

— Reply to this email directly, view it on GitHub https://github.com/Accord-Project/API-Development/issues/2#issuecomment-2022659188, or unsubscribe https://github.com/notifications/unsubscribe-auth/AF3H3SBXICPGFBFGP2T2QJ3Y2KVKTAVCNFSM6AAAAABFBLVPPGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMRSGY2TSMJYHA . You are receiving this because you were mentioned.Message ID: @.***>

[beachtom]

So the plan is to put a rest API in front of the graphQL - to allow for premade queries.

For you first point - it is up to this API to deal with issues like this. So if you request clause 2.1 of version 2023 - it does not matter when the clause was first drafted you get that version.

[rickmakkinga]

Ok, clear :)

Met vriendelijke groet | Best regards,

Rick Makkinga

Product Owner Clearly Suite

Samen maken wij projecten succesvol

Clearly | .Hub http://www.futureinsight.nl/clearly-hub | .Projects http://www.futureinsight.nl/clearly-projects | .BIM http://www.futureinsight.nl/clearly-bim | .3D-City http://www.futureinsight.nl/clearly-3d-city

Bekijk onze vernieuwde website Ga naar www.futureinsight.nl

FUTURE INSIGHT | www.futureinsight.nl | LinkedIn https://www.linkedin.com/company/future-insight-bv/

Grote Voort 207G | 8041 BK Zwolle | + 31 6 129 866 32

Op ma 8 apr 2024 om 11:31 schreef Tom Beach @.***>:

So the plan is to put a rest API in front of the graphQL - to allow for premade queries.

For you first point - it is up to this API to deal with issues like this. So if you request clause 2.1 of version 2023 - it does not matter when the clause was first drafted you get that version.

— Reply to this email directly, view it on GitHub https://github.com/Accord-Project/API-Development/issues/2#issuecomment-2042288582, or unsubscribe https://github.com/notifications/unsubscribe-auth/AF3H3SFJNBUOV3IFX6YAGWLY4JPYLAVCNFSM6AAAAABFBLVPPGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDANBSGI4DQNJYGI . You are receiving this because you were mentioned.Message ID: @.***>

[beachtom]

This is now sorted

[Gonsco]

I apologize for being late with the discussion...

On the topic of including the version in the graph name, do we have a definitive position? Does this have a direct relation with the name used as part of the URL when a document is upload to the GraphDB via the REST API? Currently this is indicated in the https://github.com/Accord-Project/API-Development/blob/main/BuildingCodesAndRules/graphdb-webapi.md for the PUT method we use in the RFT to upload the files:

https://graphdb.accordproject.eu/graphdb/repositories/aec3po/statements?context=%3Chttps%3A%2F%2Fgraphdb.accordproject.eu/resource/aec3po/Spain/v2%3E

The name and version issues must be clear in order to know how to upload the file from the RFT and how to control it. Is it assumed that every time the publish button is pressed, the document of the same project is published in a new version? If so, the tool should perform version control, right? But, if the version is controlled by the date, as Thomas suggest, I understand that this should be part of the name in the URI, right?

I raise all these questions so as not to leave any loose ends :)

[beachtom]

@Gonsco it is best you track the issues in the BCRL repo. Together, we are getting close to upload /download from GraphDB - there were can give you a definitive answer