search

gs1 / EPCIS

Draft files being shared for EPCIS 2.0 development
Other
22 stars 7 forks source link

capture field dependencies; express them in SHACL #255

Closed VladimirAlexiev closed 3 years ago

VladimirAlexiev commented 3 years ago

The spec includes numerous field dependencies. We should try to express as many of them as possible in a validatable way:

I've worked on EU Customs systems, and they have up to 400 cross-field validations (business rules) for a customs declaration. We're not chasing a number, but I won't be surprised if EPCIS has 70-100 rules scattered throughout the spec.

So @CraigRe @RalphTro @mgh128, I propose to collect all dependencies in a section or appendix called Business Rules.

Please comment on each one below (I'm writing this from memory so may have mistakes); and please add more:

VladimirAlexiev commented 3 years ago

Example of schematron rules:

They come from this tool: https://smartconnected.semantic-treehouse.nl/#/ext/eValidator, which is used to develop data exchange standards.

VladimirAlexiev commented 3 years ago

Mark agrees with 3..7

VladimirAlexiev commented 3 years ago

@CraigRe @RalphTro Please take a look at this issue

RalphTro commented 3 years ago

Dear @VladimirAlexiev, gladly. Just had a look and my take is as follows:

VladimirAlexiev commented 3 years ago

@RalphTro

cc @CraigRe @RalphTro @mgh128

shalikasingh commented 3 years ago

Hi @VladimirAlexiev,

We really believe that having a single source of truth to validate an EPCIS Document semantically will bring great relief to implementers. But we do realize it will require a lot of effort to consolidate such validation constraints in a single place.

One issue we faced initially was tools that can validate events against such rules. For e.g. we couldn't find a 100% reliable tool that can use SHACL.ttl file to validate JSON-LD hence we had to build our own validation framework.

So the question is if we define rules in Schematron, Do we have tools available in popular languages like Python, Java, JavaScript, Ruby, etc. which will make vendor's life easy to implement a plethora of validation rules of EPCIS 2.0?

mgh128 commented 3 years ago

There are some implementations of Schematron available at https://schematron.com/front-page/the-schematron-skeleton-implementation/ - and it is a freely available ISO/IEC standard , ISO / IEC 19757-3

VladimirAlexiev commented 3 years ago
mgh128 commented 3 years ago

Schematron has certainly existed longer than SHACL and is therefore likely to have a greater number of mature implementations, although I note that there are a number of open source implementations at https://github.com/search?q=shacl As far as I'm aware, Schematron is primarily intended for validating XML data. I read ( https://schematron.com/2018/11/schematron-validation-of-json-data/ ) that a Schematron-inspired tool has been developed to validate JSON data. @VladimirAlexiev - do you know if Schematron can validate Linked Data / JSON-LD ?

Yes, we need to fix the quirks in the current SHACL - I'll take a look at (#196) and try to make some edits.

VladimirAlexiev commented 3 years ago

@mgh128 answered in #261 (let's keep this issue for specific constraints/dependencies)

dakbhavesh commented 3 years ago

Dear @RalphTro and @mgh128,

As promised during our call, I gave Schematron a try. Here is the video link to see it in action. It depicts one of the rules which @RalphTro was interested to see the feasibility. More details as follows just in case you wish to try yourself:

Online tool: https://www.liquid-technologies.com/online-schematron-validator Simple Schematron rule defined for any object event in EPCIS document:

<sch:schema xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:fn="http://www.w3.org/2005/xpath-functions" xmlns:sch="http://purl.oclc.org/dsdl/schematron">
<sch:title>Business rules for EPCIS events</sch:title>
<sch:pattern id="default">
  <sch:rule id="EPCIS-SR-00000" fpi="basic assertions" context="ObjectEvent" flag="error">
    <sch:assert test="eventTime">eventTime is mandatory.</sch:assert>
    <sch:assert test="epcList or extension/quantityList/quantityElement">either epcList or quantityList must be present</sch:assert>
  </sch:rule>
</sch:pattern>
</sch:schema>

As @mgh128 suggested we can create a folder to keep a Schematron file with as many important rules as we can build quickly and draft a section about semantic validations in EPCIS.20. We can discuss this briefly in the upcoming call and decide the scope. We will most likely be able to help to come up with a basic rule set.

Looking forward to your thoughts.

cc: @VladimirAlexiev

VladimirAlexiev commented 3 years ago

Thanks!

RalphTro commented 3 years ago

Dear @dakbhavesh , Thanks a lot for putting this together + making the video + point to the online tool! I think in line of what @VladimirAlexiev is pointing out, please note that an ObjectEvent must have an field, even if there are no EPCs in the list (in this case, it would appear as </epcList> to comply with the EPCIS XSD). So, an interesting case is the following one (you could adapt your existing example quite easily for that): Object Event is populated with </epcList> only (i.e. has no EPCs and no quantities). According to the EPCIS XSD, this is valid, though according to the standard, it is NOT as the EPCIS spec says that either epcList or quantityList or both must be non-empty. Could this be uncovered via Schematron?

dakbhavesh commented 3 years ago

Dear @RalphTro, Yes, what you're asking seems possible. i.e. Scehmatron processor is able to detect through XPath that atleast one of epcList or extension/quantityList/quantityElement must be non-empty through below rule definition. I am new to schematron so rule or xpath may not be perfect. Here is new video link showcasing the same.

<sch:schema xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:fn="http://www.w3.org/2005/xpath-functions" xmlns:sch="http://purl.oclc.org/dsdl/schematron">
<sch:title>Business rules for EPCIS events</sch:title>
<sch:pattern id="default">
  <sch:rule id="EPCIS-SR-00000" fpi="eventTime Mandatory field" context="ObjectEvent" flag="error">
    <sch:assert test="(epcList and epcList/epc) or (extension/quantityList/quantityElement and extension/quantityList/quantityElement/epcClass and extension/quantityList/quantityElement/quantity)">either epcList or quantityList must be present</sch:assert>
  </sch:rule>
</sch:pattern>
</sch:schema>

I think we can locate or check existence of element or attribute at any depth through XPath which will make such checks very easy. Let me know if this is what you were wishing to uncover through schematron.

@VladimirAlexiev - you are right single field rule can be very well covered by json schema so it's redundant.

dakbhavesh commented 3 years ago

Dear @RalphTro, In previous example empty value check was missing which can easily be added through epcList and epcList/epc and string-length(epcList/epc) > 0

CraigRe commented 3 years ago

Extract from the spec those cross-validation rules that cannot easily be expressed in schema, and set up a list of pseudocode (i.e., if...then). Start with SHALL statements prior to CommRev; feedback during CommRev (e.g., "you guys overlooked a normative statement that does not carry SHALL/SHALL NOT requirement) can be resolved during CommRev resolution.

CraigRe commented 3 years ago

Following publication of EPCIS/CBV 2.0, the EPCIS 2.0 MSWG will develop an updated conformance requirements document (including all SHALL statements), but NOT an additional artefact as suggested in the issue, at least for the time being.

VladimirAlexiev commented 3 years ago
  1. PersistentDisposition: set and unset must not have any members in common (because it makes no sense to both set and unset the same disposition within the same event)
  2. disposition must not be a member of either PersistentDisposition.set (that would be redundant), nor PeristentDisposition.unset (that would be inconsistent)