ITxPT / DATA4PTTools

Shared space for the development of the DATA4PT Greenlight NeTEx validation tool(s)
MIT License
10 stars 2 forks source link

Validation errors for TypeOfFrameRef #67

Closed skinkie closed 1 year ago

skinkie commented 1 year ago

I think these errors should be 'corrected', and ideally some external entities should be made present. I also wonder how an ensemble of files could be validated, allowing these external entities to match.

   {
    "name": "xsd",
    "valid": false,
    "error_count": 6,
    "errors": [
     {
      "message": "Element '{http://www.netex.org.uk/netex}TypeOfFrameRef': No match found for key-sequence ['epip:EU_PI_LINE_OFFER', 'epip:1.0'] of keyref '{http://www.netex.org.uk/netex}TypeOfFrame_AnyKeyRef'.",
      "line": 21,
      "type": "xsd"
     },
     {
      "message": "Element '{http://www.netex.org.uk/netex}TypeOfFrameRef': No match found for key-sequence ['epip:EU_PI_STOP', 'epip:1.0'] of keyref '{http://www.netex.org.uk/netex}TypeOfFrame_AnyKeyRef'.",
      "line": 46,
      "type": "xsd"
     },
     {
      "message": "Element '{http://www.netex.org.uk/netex}TypeOfFrameRef': No match found for key-sequence ['epip:EU_PI_COMMON', 'epip:1.0'] of keyref '{http://www.netex.org.uk/netex}TypeOfFrame_AnyKeyRef'.",
      "line": 163,
      "type": "xsd"
     },
     {
      "message": "Element '{http://www.netex.org.uk/netex}TypeOfFrameRef': No match found for key-sequence ['epip:EU_PI_NETWORK', 'epip:1.0'] of keyref '{http://www.netex.org.uk/netex}TypeOfFrame_AnyKeyRef'.",
      "line": 191,
      "type": "xsd"
     },
     {
      "message": "Element '{http://www.netex.org.uk/netex}TypeOfFrameRef': No match found for key-sequence ['epip:EU_PI_TIMETABLE', 'epip:1.0'] of keyref '{http://www.netex.org.uk/netex}TypeOfFrame_AnyKeyRef'.",
      "line": 428,
      "type": "xsd"
     },
     {
      "message": "Element '{http://www.netex.org.uk/netex}TypeOfFrameRef': No match found for key-sequence ['epip:EU_PI_CALENDAR', 'epip:1.0'] of keyref '{http://www.netex.org.uk/netex}TypeOfFrame_AnyKeyRef'.",
      "line": 1005,
      "type": "xsd"
     }
    ]
   }
rhorenov commented 1 year ago

Hi guys,

I'm very new to NeTEx / EPIP, so be gentle.

The problem

I was using you validation tool - good work BTW, to check validity of experimentally generated timetables and I assume I get the same issues reported as in the OP. XML file attached bellow.

The issues reported are in the same vein as this one:

     {
      "message": "In violation of key-ref constraint, missing key reference \"TypeOfFrame_AnyKeyRef\" (@ref=\"epip:EU_PI_LINE_OFFER\")",
      "line": 7,
      "type": "consistency"
     },

It refers to this part of the XML:

image

I believe the issues are being reported incorrectly.

Expected behavior

Here is a snapshot from the document: "Public transport - Network and Timetable Exchange (NeTEx)— Part 4: Passenger Information European Profile", chapter 9.7 "Referential integrity".

image

If I understand it correctly, external references, marked with attribute versionRef shall be ignored by the checker.

I believe this issue (#67) is actually an error in the validation tool and not just an enhancement.

I would very much appreciate if it could be fixed. More so that I’ve noticed that there has been some work done recently in the key ref constraint evaluation JS rule.

If I can be of any help, let me know.

Regards, Roman

Support data

Sample file: netex_epip_rhorenov_02.zip

DATA4PTTools version: 57aa7b7, tag:1.0.4, "Merge pull request #87 from ITxPT/fix/keyref", locally built docker image OS: Microsoft Windows [Version 10.0.19045.3086]

Command:

docker run --rm -it -v C:\Work\TTR\2023-NeTEx\DATA4PTTools\mytestdata:/usr/local/greenlight/mytestdata netex validate --output json --schema epip@1.1.1 --input mytestdata

Result:

[
 {
  "name": "mytestdata/netex_epip_rhorenov_02.xml",
  "valid": false,
  "validations": [
   {
    "name": "everyLineIsReferenced",
    "valid": true
   },
   {
    "name": "everyStopPlaceHasAName",
    "valid": true
   },
   {
    "name": "everyStopPlaceHasACorrectStopPlaceType",
    "valid": true
   },
   {
    "name": "stopPlaceQuayDistanceIsReasonable",
    "valid": true
   },
   {
    "name": "frameDefaultsHaveALocaleAndTimeZone",
    "valid": true
   },
   {
    "name": "everyScheduledStopPointHasAName",
    "valid": true
   },
   {
    "name": "everyStopPointHaveArrivalAndDepartureTime",
    "valid": true
   },
   {
    "name": "locationsAreReferencingTheSamePoint",
    "valid": true
   },
   {
    "name": "everyStopPlaceIsReferenced",
    "valid": true
   },
   {
    "name": "passingTimesIsNotDecreasing",
    "valid": true
   },
   {
    "name": "netexUniqueConstraints",
    "valid": true
   },
   {
    "name": "netexKeyRefConstraints",
    "valid": false,
    "error_count": 6,
    "errors": [
     {
      "message": "In violation of key-ref constraint, missing key reference \"TypeOfFrame_AnyKeyRef\" (@ref=\"epip:EU_PI_CALENDAR\")",
      "line": 63,
      "type": "consistency"
     },
     {
      "message": "In violation of key-ref constraint, missing key reference \"TypeOfFrame_AnyKeyRef\" (@ref=\"epip:EU_PI_STOP\")",
      "line": 110,
      "type": "consistency"
     },
     {
      "message": "In violation of key-ref constraint, missing key reference \"TypeOfFrame_AnyKeyRef\" (@ref=\"epip:EU_PI_NETWORK\")",
      "line": 206,
      "type": "consistency"
     },
     {
      "message": "In violation of key-ref constraint, missing key reference \"TypeOfFrame_AnyKeyRef\" (@ref=\"epip:EU_PI_TIMETABLE\")",
      "line": 2242,
      "type": "consistency"
     },
     {
      "message": "In violation of key-ref constraint, missing key reference \"TypeOfFrame_AnyKeyRef\" (@ref=\"epip:EU_PI_LINE_OFFER\")",
      "line": 7,
      "type": "consistency"
     },
     {
      "message": "In violation of key-ref constraint, missing key reference \"TypeOfFrame_AnyKeyRef\" (@ref=\"epip:EU_PI_COMMON\")",
      "line": 29,
      "type": "consistency"
     }
    ]
   },
   {
    "name": "xsd",
    "valid": true
   }
  ]
 }
]
skinkie commented 1 year ago

I think your point make sense, but is there any example where typeOfFrameRef with versionRef="any" is used? Because... it really does not make sense to do so in that way.

xmllint --noout --schema /home/skinkie/Sources/NeTEx/xsd/NeTEx_publication.xsd netex_epip_rhorenov_02.xml 
netex_epip_rhorenov_02.xml validates

Free tips:

<DefaultLocationSystem>WGS84</DefaultLocationSystem>

Change this to EPSG:4326 (or the longer urn)

                <AddressLine1>Na Rovinkách 211, 513 01 Semily, Podmoklice, provozovna Rokytnice</AddressLine1>

There are seperate fields for this.

                <Url>www.busline.cz</Url>

This is not a valid url.

You mention through the document version="1" it is likely we will make this behavior invalid in the future. A small Version element on the composite frame will help validation.

              <keyList>
                <KeyValue>
                  <Key>JdfLineType</Key>
                  <Value>V</Value>
                </KeyValue>
                <KeyValue>
                  <Key>JdfDetourTimetable</Key>
                  <Value>1</Value>
                </KeyValue>
                <KeyValue>
                  <Key>JdfLinExt</Key>
                  <Value>30512;553</Value>
                </KeyValue>
              </keyList>

My guess is you can put this in existing elements like ExternalLineRef, PrivateCode, etc.

            <ScheduledStopPoint id="CZ:cisjr_jdf:ScheduledStopPoint:670553_1_1" version="1">
              <Name>Rovensko p.Tr.,,nám.</Name>
              <Location />
              <tariffZones>
                <TariffZoneRef version="1" ref="CZ:cisjr_jdf:TariffZone:1" />
              </tariffZones>
            </ScheduledStopPoint>

The way how location is left empty is really bad. Omit the element, but better: create a location.

            <ServiceLink id="CZ:cisjr_jdf:ServiceLink:670553_1_2_1_0" version="1">
              <Distance>0</Distance>
              <FromPointRef version="1" ref="CZ:cisjr_jdf:ScheduledStopPoint:670553_1_2" />
              <ToPointRef version="1" ref="CZ:cisjr_jdf:ScheduledStopPoint:670553_1_1" />
            </ServiceLink>

            <ServiceLink id="CZ:cisjr_jdf:ServiceLink:670553_1_10_11_1" version="1">
              <Distance>1000</Distance>
              <FromPointRef version="1" ref="CZ:cisjr_jdf:ScheduledStopPoint:670553_1_10" />
              <ToPointRef version="1" ref="CZ:cisjr_jdf:ScheduledStopPoint:670553_1_11" />
            </ServiceLink>

Typically an error where everything is perfectly rounded.

            <ServiceJourneyPattern id="CZ:cisjr_jdf:ServiceJourneyPattern:2_in" version="1">
              <RouteView id="CZ:cisjr_jdf:RouteView:2_in">
                <LineRef version="1" ref="CZ:cisjr_jdf:Line:670553_1" />
              </RouteView>
              <DirectionRef version="1" ref="CZ:cisjr_jdf:Direction:in" />
              <pointsInSequence>
                <StopPointInJourneyPattern id="CZ:cisjr_jdf:StopPointInJourneyPattern:2_in_20" version="1" order="1">
                  <ScheduledStopPointRef version="1" ref="CZ:cisjr_jdf:ScheduledStopPoint:670553_1_20" />
                  <OnwardServiceLinkRef version="1" ref="CZ:cisjr_jdf:ServiceLink:670553_1_20_19_0" />
                </StopPointInJourneyPattern>
                <StopPointInJourneyPattern id="CZ:cisjr_jdf:StopPointInJourneyPattern:2_in_19" version="1" order="2">
                  <ScheduledStopPointRef version="1" ref="CZ:cisjr_jdf:ScheduledStopPoint:670553_1_19" />
                  <OnwardServiceLinkRef version="1" ref="CZ:cisjr_jdf:ServiceLink:670553_1_19_18_1" />
                  <RequestStop>true</RequestStop>
                </StopPointInJourneyPattern>

Are you sure that all your stops are RequestStops? It is unlikely, but it might be these were the flexible journeys.

rhorenov commented 1 year ago

Hi @skinkie,

and thanks a lot for your feedback regarding the different problematic aspects of the data file in question.

I think your point make sense, but is there any example where typeOfFrameRef with versionRef="any" is used? Because... it really does not make sense to do so in that way.

Do you mean any example of TypeOfFrameRef element with versionRef attribute (disregard the value "any", it's not the issue)?

There are many examples of that in the "Public transport - Network and Timetable Exchange (NeTEx)— Part 4: Passenger Information European Profile (2019)" document.

For example in the chapter "8.4.4 Identifier structure for EPIP version frames" is the following example:

image

Also the sample EPIP XML file from data4pt.org (is it still valid???) Example XML-file for the EPIP profile

They all use typeOfFrameRef with versionRef to identify the type of the frame (e.g. epip:EU_PI_LINE_OFFER).

As I pointed out in my previous post, how I understand it, the presence of the versionRef in the element (as opposite to the version attribute - they are mutually exclusive) means, that the object it refers to, does not live in the same XML file and shall be ignored by the validator - as stated in the chapter 9.7 in the docs.

Would you agree?

Note, that I have the EPIP specification from 2019, so the state of affairs might have changed since...

Regards, Roman

skinkie commented 1 year ago

Do you mean any example of TypeOfFrameRef element with versionRef attribute (disregard the value "any", it's not the issue)?

versionRef is used to refer to an external version, because the version attribute is not present there is no key identity constraint matching going on, and NeTex' special "any" value does not have to be used. So 1.0 it would make sense, any does not.

Also the sample EPIP XML file from data4pt.org (is it still valid???) Example XML-file for the EPIP profile

That example was taken from a national access point, which in effect had many NeTEx issues. My guess is that this example was actually fixed to overcome these issues, it does validate against the full XSD.

They all use typeOfFrameRef with versionRef to identify the type of the frame (e.g. epip:EU_PI_LINE_OFFER).

Yes, but not with any.

As I pointed out in my previous post, how I understand it, the presence of the versionRef in the element (as opposite to the version attribute - they are mutually exclusive) means, that the object it refers to, does not live in the same XML file and shall be ignored by the validator - as stated in the chapter 9.7 in the docs.

Would you agree?

General correct interpretation. I don't know about being mutually exclusive (at least not being enforced in the schema).

Note, that I have the EPIP specification from 2019, so the state of affairs might have changed since...

Documentation could always get better ;-)

rhorenov commented 1 year ago

I get it, you don't like "any" :-) But it really has absolutely nothing to do with this issue.

The issue is that the following element (and other TypeOfFrameRef/@versionRef):

<TypeOfFrameRef ref="epip:EU_PI_LINE_OFFER" versionRef="1.0" />

causes the validation to fail, although the documentation says it shall pass.

Doc snapshot re. external reference:

image

Doc snapshot re. checking (more like not checking) the external references (chapter 5.3.3 Integrity Checking of XML Documents):

image

Updated EPIP XML data, removed "any" from versionRef: netex_epip_rhorenov_03.zip

skinkie commented 1 year ago

I get it, you don't like "any" :-)

I like any, but only on version ;)

But it really has absolutely nothing to do with this issue.

I absolutely agree, if versionRef is checked, it is a bug in the validator.

lekotros commented 1 year ago

I believe this have been fixed in v1.0.6