Sveino / Inst4CIM-KG

Instance of CIM Knowledge Graph
Apache License 2.0
5 stars 1 forks source link

representing difference models #53

Open VladimirAlexiev opened 2 months ago

VladimirAlexiev commented 2 months ago

@Sveino (and @griddigit-ci) in https://github.com/Sveino/Inst4CIM-KG/issues/15#issuecomment-2350981370:

We would like to have a JSON-LD way of exchanging difference model (forwardModel and reverseModel). If we also find a way that Turtle and RDF/XML also will support it would be great.

VladimirAlexiev commented 2 months ago

This issue is important since it will impact the test suite: a great way to create invalid models to test each SHACL, is by using a small difference to a base valid model.

The right (only?) way to represent models is with named graphs.

The non-standard RDF/XML addition parseType=statements in fact represents a graph, but it has no name and cannot be referenced from any other place.

Models and differential models can be represented something like this:

# this goes into the default graph

<base-model> a md:Model;
  md:Model.created ... ;
  md:Model.updated ... ;
  # other Header data

<difference-model> a dm:DifferenceModel;
  dm:DifferenceModel.baseModel <base-model>;  #### NEW
  dm:DifferenceModel.preconditions # TODO; # don't know what this is, gotta read the spec
  dm:DifferenceModel.reverseDifferences <differences-to-delete>;
  dm:DifferenceModel.forwardDifferences<differences-to-insert>.

<base-model> {
  <transmissionLine> a TransmissionLine;
    voltage 220
}

<differences-to-delete> {
  <transmissionLine> voltage 220
}

<differences-to-insert> {
  <transmissionLine> voltage 330
}

Currently preconditions, reverseDifferences, forwardDifferences use rdf:Statement (or rdf:Statements) but RDF Reification is not a scalable nor good way to capture a bunch of triples. Named graphs are better.

VladimirAlexiev commented 2 months ago

@Sveino do you agree?

VladimirAlexiev commented 1 month ago

@Sveino by email:

image

VladimirAlexiev commented 1 month ago

@Sveino @griddigit-ci

Sveino commented 1 month ago

You can find the mapping of the header here: https://github.com/3lbits/CIM4NoUtility/issues/277 It might need to be revised to be inline with the latest Metadata document.

VladimirAlexiev commented 1 month ago

@Sveino I've reread https://github.com/3lbits/CIM4NoUtility/issues/277 . Since it's closed, I'll post more stuff here:

VladimirAlexiev commented 1 month ago

This is well described in rdf-improvement sections ## Represent Models as Named Graphs and ### Custom CIM XML Parser.

But I'll leave it open since @Sveino and @griddigit-ci we need to discuss the representation of Models: you cannot use md/dm in XML but dcat/dct in JSON-LD: you have to use the same ontology terms!

griddigit-ci commented 3 weeks ago

Yes, this is a discussion point. @VladimirAlexiev do you know RDF Package (I think this is how it was called) to exchange RDF differences. It was something that was implemented in Jena for that

VladimirAlexiev commented 3 weeks ago

If you search at LOV for "patch" https://lov.linkeddata.es/dataset/lov/terms?q=patch and "diff" you find some stuff.

On the implementation side, there is https://afs.github.io/rdf-delta/rdf-patch.html but it's a line-oriented format to replicate repos, not an RDF format. ("afs" is the maker of Jena)

VladimirAlexiev commented 3 weeks ago

If we decide to use solid: the example in https://github.com/Sveino/Inst4CIM-KG/tree/develop/rdf-improved#custom-cim-xml-parser will become:


<urn:uuid:05edbf91-231f-4386-97c0-d4cb498d0afc> { # model graph

# model metadata
<urn:uuid:05edbf91-231f-4386-97c0-d4cb498d0afc> 
  a dm:DifferenceModel, 
    rdfg:Graph.
    solid:InsertDeletePatch ; <<<<<<<<
  solid:deletes <urn:uri:27c8a164-c656-4712-994a-0ab7cec4fd34> ;  <<<<<<
  solid:inserts <urn:uri:63528ef9-48ff-469b-a58e-ba274f2a10bb> ;  <<<<<<
  solid:patches <urn:uuid:52a409c9-72d8-4b5f-bf72-9a22ec9353f7> ; # base model  <<<<<<
  md:Model.DependentOn <urn:uuid:0cd6ada4-b6dc-4a36-a98c-877a39168cd3> ;
  md:Model.created "2021-11-19T23:16:27Z" .

<urn:uri:27c8a164-c656-4712-994a-0ab7cec4fd34> a rdfg:Graph.
<urn:uri:63528ef9-48ff-469b-a58e-ba274f2a10bb> a rdfg:Graph.
}

<urn:uri:27c8a164-c656-4712-994a-0ab7cec4fd34> { # reverseDifferences
<http://fullgrid.eu/CGMES/3.0#87478acb-cd1f-40a6-b4a7-59ec99f8b063> cim:IdentifiedObject.description "SET_PNT_1" .
<http://fullgrid.eu/CGMES/3.0#fc908c16-468f-4a64-ba74-6f57175e0005> cim:AnalogLimit.value "99" .
}

<urn:uri:63528ef9-48ff-469b-a58e-ba274f2a10bb> { # forwardDifferences
<http://fullgrid.eu/CGMES/3.0#87478acb-cd1f-40a6-b4a7-59ec99f8b063> cim:IdentifiedObject.description "SET_PNT_1 test" .
<http://fullgrid.eu/CGMES/3.0#fc908c16-468f-4a64-ba74-6f57175e0005> cim:AnalogLimit.value "100" .
}
Sveino commented 3 weeks ago

I like the definitions in https://vocab.org/changeset/schema.html: However, it should in principle expand DCAT rather than defining their own term for CreatedDate etc I think that solid: is a very important project and I would like to follow this. However, I am not happy with the terms. It is too syntax based rather than Semantic. I would really like to agree on the CIM JSON-LD serialisation of difference model before we look into trix or N3. We worked with this in this issue: https://github.com/3lbits/CIM4NoUtility/issues/279

Here I listed the following references:

If we start with the set theory: In set theory, the symmetric difference of two sets $A$ and $B$ includes elements that are in either $A$ or $B$, but not in both. This operation is denoted as $A \triangle B$ and is defined formally as:

$$ A \triangle B = (A \setminus B) \cup (B \setminus A) $$

Properties of the Symmetric Difference

  1. Elements Unique to Each Set: The symmetric difference $A \triangle B$ contains only those elements that are in $A$ or $B$, but not shared between them.
  2. Commutative Property: The symmetric difference is commutative, meaning $A \triangle B = B \triangle A$.
  3. Associative Property: This operation is associative, so for any sets $A$, $B$, and $C$, we have $(A \triangle B) \triangle C = A \triangle (B \triangle C)$.
  4. Disjoint Sets: If $A \cap B = \emptyset$, meaning $A$ and $B$ are disjoint, then $A \triangle B = A \cup B$, as there are no shared elements.

Example

Given sets $A = {1, 2, 3, 4}$ and $B = {3, 4, 5, 6}$, the symmetric difference is:

$$ A \triangle B = {1, 2, 5, 6} $$

This result contains elements that are in either $A$ or $B$ but not in both.

In set theory, the difference (or relative complement) between two sets $A$ and $B$ is the set of elements that are in $A$ but not in $B$. This operation is commonly denoted as $A \setminus B$ and is defined as:

$$ A \setminus B = { x \in A \mid x \notin B } $$

Properties of the Difference Set

  1. Uniqueness: $A \setminus B$ only includes elements exclusive to $A$. If $B$ has elements that are also in $A$, those elements are excluded from $A \setminus B$.
  2. Non-Commutative: Unlike union or intersection, the difference operation is not commutative. In general, $A \setminus B \neq B \setminus A$.
  3. Empty Set Case: If $B \subseteq A$, then $A \setminus B$ is the set of elements in $A$ that are not in $B$. If $A = B$, then $A \setminus B = \emptyset$.
  4. Relationship to Complements: In a universal set $U$, the difference $U \setminus A$ is the complement of $A$ relative to $U$.

Example

Given sets $A = {1, 2, 3, 4}$ and $B = {3, 4, 5}$, the difference set is:

$$ A \setminus B = {1, 2} $$

This result contains only the elements in $A$ that are not in $B$.

reverseDifference is

$$ A \setminus B = {1, 2} $$

forwardDifference is

$$ B \setminus A = {5} $$

Sveino commented 3 weeks ago

@VladimirAlexiev you wrote:

I agree with both statement - even it my hand is on it... I think PowerSystemProject must be a separate class as a specialisation of cim:Project. The problem was that we wanted to use DifferenceModel with defining the content. We need to resolve this first.

You wrote:

dm:DifferenceModel.preconditions # TODO; # don't know what this is, gotta read the spec

Precondition statements, comprising statements found in both B1 and B2 and considered to be dependencies of the difference model in an application defined sense.

dm:preconditions is a property of the difference model whose value is the collection of precondition statements.

7.2.4.5 Preconditions and concurrency The precondition statements are a subset of both B1 and B2 and carry no difference information. In simple, sequential model revision scenarios they can be omitted. For a large shared model, sequential revision is not always feasible. Revisions are likely to be constructed concurrently by different participants, without reference to each other. Concurrency issues must be handled, but the conventional database-oriented approach of using locks to detect incompatible concurrent transactions is not feasible on a web-scale. The precondition statements are an alternative to locks. Informally, they represent the information that would have been read-locked in an equivalent database transaction. Software agents that process difference models can check that the preconditions hold and, if not, warn of incompatible model revisions. The choice of statements to include as preconditions is application-specific (as is the choice of which information to lock in a database transaction). Preconditions should include statements that would affect decisions of the agent that produced the model revision.

7.2.4.6 Difference model template

<rdf:RDF xmlns:rdf=”http://www.w3.org/1999/02/22-rdf-syntax-ns#”
    xmlns:cim=”cim-namespace-uri”
    xmlns:md=”cim-model-description_uri”
    xmlns:dm=”difference-model-namespace-uri”
    xml:base="urn:uuid:">
    <dm:DifferenceModel rdf:about=model-uri>
        <!--Content:(literal-property|resource-property|compound-property)* 
        –->
        <dm:preconditions parseType=”Statements”>
            <!-– Content: (definition|description)* –->
        </dm:preconditions>
        <dm:forwardDifferences parseType=”Statements”>
            <!-– Content: (definition|description)* –->
        </dm:forwardDifferences>
        <dm:reverseDifferences parseType=”Statements”>
            <!-– Content: (definition|description)* –->
        </dm:reverseDifferences>
    </dm:DifferenceModel>
</rdf:RDF>

We have to change the md and dm namespace since they are now referring to -552:

@prefix dm: <http://iec.ch/TC57/61970-552/DifferenceModel/1#> 
@prefix md: <http://iec.ch/TC57/61970-552/ModelDescription/1#> .

I do not understand the need for two namespaces. I think one alternative is to extent DCAT. However, in addition we would need to use rdfg:Graph in the serialisation so that we can support the uniqueness of identifier in the same graph.

image

To be compliant with Set theory DifferenceSet should be called SymmetricDifferenceSet - however that we can include in the description. The easiest way to handle the syntax would be to exchange the ForwardDifferenceSet and the ReverseDifferenceSet as separate distribution (instance set) .

I think the dm:preconditions is in general referring to the dataset version. This needs to be know to ably the changes. I do not see any point that ForwardDifferenceSet and ReverseDifferenceSet should have any dataset (header) information as they should be the same as DifferenceSet. I also think that this could be the same as the target DataSet. The following are relevant dataset information:

<urn:uuid:6b7387f2-c7f9-11ed-afa1-0242ac120002> a dct-cim:DifferenceSet ;
    dcterms:accessRights <https://energy.referencedata.eu/Confidentiality/OPDEConfidential> ;
    dcterms:conformsTo <https://ap.cim4.eu/RemedialAction/2.3> ;
    dcterms:description "Statnett Remedial Action dataset"@en ;
    dcterms:identifier "6b7387f2-c7f9-11ed-afa1-0242ac120002" ;
    dcterms:issued "2024-04-10T10:00:00Z"^^xsd:dateTime ;
    dcat:isVersionOf <https://energy.referencedata.eu/Model/Statnett-RA> ;
    dcat:keyword "RA" ;
    dcterms:license <https://creativecommons.org/licenses/by/4.0/> ;
    dcterms:publisher <https://energy.referencedata.eu/EIC/10X1001A1001A38Y> ;
    dcterms:references <urn:uuid:99ae9f41-0a91-4d21-a483-7398c160da96> ;
    dcterms:spatial <https://energy.referencedata.eu/Frame/NO-Power-Transmission-System> ;
    dcat:startDate "2024-04-10T05:00:00Z"^^xsd:dateTime ;
    dcterms:title "20240410_NO-RA" ;
    dcat:version "2.0.0" ;
    adms:versionNotes "This is the second version."@en ;
    prov:wasGeneratedBy <https://energy.referencedata.eu/Action/CGM-RA> ;

    prov:wasRevisionOf <urn:uuid:34f39767-fe2d-45f8-b5f8-aa60375071f1>;
    dct-cim:forwardDifferenceSet <urn:uuid:a4025d93-bf8c-4d38-baeb-ddc2219e7570>;
    dct-cim:reversalDifferenceSet <urn:uuid:e5a0a4ad-f115-4f79-af3f-45a5317398c3>;

dcterms:conformsTo describe what the target dataset should be validated against dcterms:references : cannot be used to define the set that should be updated since the new full model will also have references. prov:wasRevisionOf - This refer to the dataset that is modified through the DifferenceSet

Sveino commented 3 weeks ago

Example - without the header.

{
  "@context": {
        "rdf": "http://www.w3.org/1999/02/22-rdf-syntax-ns#",
        "cim": "https://cim.ucaiug.io/ns#",
        "dcat": "http://www.w3.org/ns/dcat#",
        "dcterms": "http://purl.org/dc/terms/#", 
        "dcat-cim": " https://cim4.eu/nc/dcat-cim#"
  },
  "@graph": [
    {
      "@id": "urn:uuid:a4025d93-bf8c-4d38-baeb-ddc2219e7570",
      "@type": "dcat:Dataset",
      "dcterms:title": "Forward Difference Set",
      "dcterms:identifier": "a4025d93-bf8c-4d38-baeb-ddc2219e7570",
      "dcat-cim:forwardDifferenceSet": [
        {
          "@id": "http://fullgrid.eu/CGMES/3.0#87478acb-cd1f-40a6-b4a7-59ec99f8b063",
          "cim:IdentifiedObject.description": "SET_PNT_1 test"
        },
        {
          "@id": "http://fullgrid.eu/CGMES/3.0#fc908c16-468f-4a64-ba74-6f57175e0005",
          "cim:AnalogLimit.value": "100"
        }
      ]
    },
    {
      "@id": "urn:uuid:e5a0a4ad-f115-4f79-af3f-45a5317398c3",
      "@type": "dcat:Dataset",
      "dcterms:title": "Reverse Difference Set",
      "dcterms:identifier": "a4025d93-bf8c-4d38-baeb-ddc2219e7570",
      "dcat-cim:reverseDifferenceSet": [
        {
          "@id": "http://fullgrid.eu/CGMES/3.0#87478acb-cd1f-40a6-b4a7-59ec99f8b063",
          "cim:IdentifiedObject.description": "SET_PNT_1"
        },
        {
          "@id": "http://fullgrid.eu/CGMES/3.0#fc908c16-468f-4a64-ba74-6f57175e0005",
          "cim:AnalogLimit.value": "99"
        }
      ]
    }
  ]
}
VladimirAlexiev commented 3 weeks ago

@Sveino convert the last example to trig and you'll see that forward, reverse have 2 values each, which are nodes not triple sets. JSONLD nesting structure does not create nested subgraphs. You must use graph URIs explicitly. An example is in rdf-improved/README

Sveino commented 2 weeks ago

Updated Example based on the discussion on 2024-10-31. The example should support it to be resolvable. The namespace and transformation will be address in other issues. I just want to established the goal for the JSON-LD and TRIG description of a difference model.

{
  "@context": {
    "rdf": "http://www.w3.org/1999/02/22-rdf-syntax-ns#",
    "cim": "https://cim.ucaiug.io/ns#",
    "dcat": "http://www.w3.org/ns/dcat#",
    "dcterms": "http://purl.org/dc/terms/#",
    "dcat-cim": "https://cim4.eu/ns/dcat-cim#",
    "xsd": "http://www.w3.org/2001/XMLSchema#",
    "prov": "http://www.w3.org/ns/prov#",
    "base": "https://test.model4powersystem.eu/FullGrid/"
  },
  "@graph": [
    {
      "@graph": [
        {
          "@id": "6b7387f2-c7f9-11ed-afa1-0242ac120002",
          "@type": "dcat-cim:DifferenceSet",
          "dcterms:identifier": "6b7387f2-c7f9-11ed-afa1-0242ac120002",
          "dcterms:issued": {
            "@value": "2021-11-19T23:16:27Z",
            "@type": "xsd:dateTime"
          },
          "dcterms:description": {
            "@value": "CGMES Conformity Assessment Test Configuration. ..."
          },
          "dcat:isVersionOf": {
            "@id": "https://energy.referencedata.eu/Model/FullGrid/EQOP"
          },
          "dcterms:references": {
            "@id": "99ae9f41-0a91-4d21-a483-7398c160da96"
          },
          "prov:wasRevisionOf": {
            "@id": "34f39767-fe2d-45f8-b5f8-aa60375071f1"
          },
          "dcat-cim:forwardDifferenceSet": {
            "@id": "a4025d93-bf8c-4d38-baeb-ddc2219e7570"
          },
          "dcat-cim:reversalDifferenceSet": {
            "@id": "e5a0a4ad-f115-4f79-af3f-45a5317398c3"
          }
        }
      ]
    },
    {
      "@graph": [
        {
          "@id": "a4025d93-bf8c-4d38-baeb-ddc2219e7570",
          "@type": "dcat:Dataset",
          "dcterms:title": "Forward Difference Set",
          "dcterms:identifier": "a4025d93-bf8c-4d38-baeb-ddc2219e7570"
        },
        {
          "@id": "87478acb-cd1f-40a6-b4a7-59ec99f8b063",
          "cim:IdentifiedObject.description": "SET_PNT_1 test"
        },
        {
          "@id": "fc908c16-468f-4a64-ba74-6f57175e0005",
          "cim:AnalogLimit.value": "100"
        }
      ]
    },
    {
      "@graph": [
        {
          "@id": "e5a0a4ad-f115-4f79-af3f-45a5317398c3",
          "@type": "dcat:Dataset",
          "dcterms:identifier": "e5a0a4ad-f115-4f79-af3f-45a5317398c3",
          "dcterms:title": "Reverse Difference Set"
        },
        {
          "@id": "87478acb-cd1f-40a6-b4a7-59ec99f8b063",
          "cim:IdentifiedObject.description": "SET_PNT_1"
        },
        {
          "@id": "fc908c16-468f-4a64-ba74-6f57175e0005",
          "cim:AnalogLimit.value": "99"
        }
      ]
    }
  ]
}

TRIG

@prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .
@prefix cim: <https://cim.ucaiug.io/ns#> .
@prefix dcat: <http://www.w3.org/ns/dcat#> .
@prefix dcterms: <http://purl.org/dc/terms/#> .
@prefix dcat-cim: <https://cim4.eu/ns/dcat-cim#> .
@prefix xsd: <http://www.w3.org/2001/XMLSchema#> .
@prefix prov: <http://www.w3.org/ns/prov#> .
@prefix base: <https://test.model4powersystem.eu/FullGrid/> .

# Graph New full graph
<https://test.model4powersystem.eu/FullGrid/6b7387f2-c7f9-11ed-afa1-0242ac120002> {
  <6b7387f2-c7f9-11ed-afa1-0242ac120002> a dcat-cim:DifferenceSet ;
    dcterms:identifier "6b7387f2-c7f9-11ed-afa1-0242ac120002" ;
    dcterms:issued "2021-11-19T23:16:27Z"^^xsd:dateTime ;
    dcterms:description "CGMES Conformity Assessment Test Configuration. ..." ;
    dcat:isVersionOf <https://energy.referencedata.eu/Model/FullGrid/EQOP> ;
    dcterms:references <99ae9f41-0a91-4d21-a483-7398c160da96> ;
    prov:wasRevisionOf <34f39767-fe2d-45f8-b5f8-aa60375071f1> ;
    dcat-cim:forwardDifferenceSet <a4025d93-bf8c-4d38-baeb-ddc2219e7570> ;
    dcat-cim:reversalDifferenceSet <e5a0a4ad-f115-4f79-af3f-45a5317398c3> .
}

# Forward Graph
<https://test.model4powersystem.eu/FullGrid/a4025d93-bf8c-4d38-baeb-ddc2219e7570> {
  <a4025d93-bf8c-4d38-baeb-ddc2219e7570> a dcat:Dataset ;
    dcterms:title "Forward Difference Set" ;
    dcterms:identifier "a4025d93-bf8c-4d38-baeb-ddc2219e7570" .

  <87478acb-cd1f-40a6-b4a7-59ec99f8b063> cim:IdentifiedObject.description "SET_PNT_1 test" .

  <fc908c16-468f-4a64-ba74-6f57175e0005> cim:AnalogLimit.value "100" .
}

# Reverse Graph
<https://test.model4powersystem.eu/FullGrid/graph3> {
  <e5a0a4ad-f115-4f79-af3f-45a5317398c3> a dcat:Dataset ;
    dcterms:identifier "e5a0a4ad-f115-4f79-af3f-45a5317398c3" ;
    dcterms:title "Reverse Difference Set" .

  <87478acb-cd1f-40a6-b4a7-59ec99f8b063> cim:IdentifiedObject.description "SET_PNT_1" .

  <fc908c16-468f-4a64-ba74-6f57175e0005> cim:AnalogLimit.value "99" .
}
Sveino commented 1 day ago

In the meeting 2024-11-21 we agree to the following UML for the DifferenceSet:

image

VladimirAlexiev commented 4 hours ago

For the record, the above code blocks are copies of

There are various little things to fix in these examples: