EU Netex profile support

sjthiessen commented 2 years ago

We're trying to deploy an OTP instance with germany-wide OSM and NeTEx data (EU-Profile) and are having some issues.

Expected behavior

The NeTEx import should finish without NullPointerException and produce a routable graph.

Observed behavior

A Major and several minor compatibility issues which all caused NullPointerExceptions during the import. This raised some questions on how to tackle these issues in the best way with regard to the given dataset and the EU-profile specification.

Note: Whenever I speak of the EU-Profile (EPIP) specification, I refer to the latest draft (2019-06) I found on the internet, since I don’t yet have access to the final document.

StopAssignments without QuayRef:

In our Dataset, the PassengerStopAssignment elements are lacking QuayRef and only provide StopPlaceRef. This seems to comply with the EU-Profile, since QuayRef are not required by the Profile. On the other hand, those Quays are modeled in the StopPlace elements, but never referenced. I think this is an issue with the dataset and hope that the data supplier is able to fix it. But I can imagine that we still might have lines without QuayRef due to less detailed data.

Since the Netex-Import seems to rely heavily on QuayRef being set, would it be feasible to support datasets without those?

RepresentedByGroupRef

Inside the RouteMapper.java, the method findOrCreateAuthority expects the RepresentedByGroupRef field to be set, which our dataset doesn’t provide. I can’t find it in the EU-Profile either, but an AuthorityRef is always set so I assume its safe to use this. Is this correct?

GroupOfStopPlaces without Name and Centroid

OTP expects the Name and Centroid fields to be set for the GroupOfStationsMapper to work, but the EU-Profile doesn’t list them as part of GroupOfStopPlaces at all.

How relevant are these for OTPs functionality?

Minor Issues

I think these are not as relevant and just caused by poor modeling of certain lines but wanted to list them nevertheless.

ServiceCalendarFrame without dayType
TimeTableFrame without vehicleJourney

Version of OTP used (exact commit hash or JAR name)

90e8f897fdfc2fc1e3ad04ec3bbedafdd5b64813 plus some minor changes to locate further NullPointerExceptions I made here: https://github.com/HBTGmbH/OpenTripPlanner/commit/c6e1011e6631c003bec4aa93aed5d738b408f1a8

Data sets in use (links to GTFS and OSM PBF files)

NeTEx-Dataset Note: The ZIP-File seems to be encoded in ISO-8859-1, so I had to specify the encoding in ZipFileDataSource.java

Command line used to start OTP

Intellij Run Configuration with following CLI arguments: --loadStreet --save ....\otp2-data\src\main\resources

Router config and graph build config JSON

build-config:

{
  "dataImportReport": true,
  "extraEdgesStopPlatformLink": true,
  "stationTransfers": true,
  "netex": {
    "groupFilePattern": "(.*)\\/.*\\.xml"
  }
}

Steps to reproduce the problem

Run NeTEx import with the dataset linked above

leonardehrenfried commented 2 years ago

This is very interesting work.

My gut feeling is that most of these problems can be overcome by omitting or interpolating the information. For example, if you don't have a centroid for a group of stops you could calculate it yourself.

t2gran commented 2 years ago

The original idea from our side is to develop a NeTEx import that could support more than just the Nordic profile. The code have evolved over time, and not all of it is at the quality we want, but hopefully we are moving in the right direction. I am happy to see that you like to support the EU profile. Personally I do not know it too well, but I think the effort to support it 95% should be straight forward. You might also have to change the internal model of OTP, and those changes is usually the cases that needs more discussion and thoughts - since we try to make a model witch basically is a work for both GTFS and NeTEx.

I will try to answer you questions to the best of my knowledge, I might not remember everything so let me know if not. I am sure we would find a solution in the end.

The NeTEx import should finish without NullPointerException

Any NPE is considered an programming error. We expect the import to be valid, but our code should be robust and not fail for missing elements. I know a lot of null checks are missing. Our strategy so fare have been to add them when we encounter tham. Here is an example of such fix: #3633

In our Dataset, the PassengerStopAssignment elements are lacking QuayRef and only provide StopPlaceRef.

OTP support using Station(StopPlace) without Stops(Quays), so this should be straight forward to import.

RepresentedByGroupRef . I can’t find it in the EU-Profile either, but an AuthorityRef is always set so I assume its safe to use this. Is this correct?

So, this is a question about what is allowed in the EU profile or not. For the code we are pragmatic, you can start with what you suggest and then we can change it if it is not. The main thin here is that is will not break anything for us, and I expect us to use some time to get everything in place for the EU profile. If you are unsure, write a comment about in the code and do your best. I think it is better to get something working and then improve.

OTP expects the Name and Centroid fields to be set for the GroupOfStationsMapper to work, but the EU-Profile doesn’t list them as part of GroupOfStopPlaces at all.

How relevant are these for OTPs functionality?

You can safely drop GroupOfStations. Are GOS without name/coordinate useful? You may also just set any name/coordinate you what - if it it is important to include them. @leonardehrenfried suggestion would work. The coordinate is only used by clients to place the name on the map at the right location. When searching from a GOS we do a Station/Quay search - we search from ALL Station/quays included in the GOS, not the coordinate. I guess that if you have a GeoCoder that have the name and coordinate, the GOS id is probably enough in OTP.

Minor Issues I am not sure if I understand what the problem with the import is for these. But, if you find a solution that is probably good enough, and if you want some feedback clarify.

Some thoughts

We might run into cases where the Nordic and EU profile is not compatible, for example in validation where fields are required in one profile and not the other. At least our date contain witch profile is used:

<PublicationDelivery xmlns="http://www.netex.org.uk/netex" xmlns:ns2="http://www.opengis.net/gml/3.2" xmlns:ns3="http://www.siri.org.uk/siri" version="1.13:NO-NeTEx-networktimetable:1.3">

so it should be possible to us this while importing the data.

sjthiessen commented 2 years ago

Thank you both for your feedback!

We probably won't have to deal with the missing quays anymore, as they will be provided in the dataset somewhen in november or december

sjthiessen commented 2 years ago

On Meta-Information about the profile being used, this information is not availabale in the PublicationDelivery Element: <PublicationDelivery xmlns="http://www.netex.org.uk/netex" version="ntx:1.1">

But my interpretation of the data and the EU profile is, that this information is available via the Codespace and the TypeOfFrameRef elements:

<TypeOfFrameRef ref="epip:EU_PI_LINE_OFFER" versionRef="1.0"/>
<codespaces>
  <Codespace id="epip_data">
    <Xmlns>epd</Xmlns>
    <XmlnsUrl>http://netex-cen.eu/epip_data</XmlnsUrl>
    <Description>EPIP data</Description>
  </Codespace>
</codespaces>

From the specification

Use of the “epip_data” CODESPACE and identifier structure on frame identifiers is mandatory for the EPIP.

t2gran commented 2 years ago

Regarding the profile, I think we can begin with a build-config parameter setting the profile to use for NeTEx import as well. That should be very simple, but it is up to you.

github-actions[bot] commented 2 years ago

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 30 days

github-actions[bot] commented 2 years ago

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 30 days

github-actions[bot] commented 1 year ago

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 30 days

BredeD commented 1 year ago

I participated in the Data4PT consortium and stakeholder forum yesterday. Data4pt is a CEF-funded project to help member states to produce, validate and share NeTEx and SIRI data. The EU profile will be the minimum profile covered in the MMTIS regulation. Yesterday's discussions covered

GTFS->NeTEx conversion (maybe NOT do it at all)
Giving data producers incentive to produce native NeTEx
Interoperable NeTEx data (do we have too many country profiles already)
validation tool (shall we all use the same).

Next week there will be a validation tool webinar. https://data4pt-project.eu/webinar-netex-validation-tool/ I´m not aware of anyone else that tries to use NeTEx data from different producers.

I´m unsure if any country has a working dataset of the EU profile.

github-actions[bot] commented 1 year ago

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 30 days

github-actions[bot] commented 1 year ago

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 30 days

leonardehrenfried commented 1 month ago

NOI (South Tyrolian regional access point) are funding improvements for EPIP support for their own feeds. Early analysis suggests that their data feeds are already supported adequately. A few issues remain and I am taking a look at them.

cc @rcavaliere

opentripplanner / OpenTripPlanner