[Question/Possible bug] Expected behaviour for Netex? "Netex file ignored" for Slovenian dataset

[adnan-smlatic]

I am trying to import the Netex files for Slovenia from the official government provider at nap.si , they provide the following zip files each containing an .xml with the same name + date eg. NETEX_PI_01_SI_NAP_OPERATOR_20240107.xml

NETEX_PI_01_SI_NAP_FARES.zip NETEX_PI_01_SI_NAP_NETWORK.zip NETEX_PI_01_SI_NAP_OPERATOR.zip NETEX_PI_01_SI_NAP_STOP.zip

I have tried absolutely everything in the documentation and the provided examples, but no matter what I do I get:

16:49:39.863 WARN [main] (NetexDataSourceHierarchy.java:108) Netex file ignored: . ... The file do not match any file patterns in the config.

Doesn't seem to matter what I name and rename the files, or what I set the file pattern to, build config, zip them all together, it's always the same response.

The Slovenian dataset is a lot different to the Norway example provided so I am not really sure.

Is this the expected response, are the provided files in some weird format or is this some sort of bug?

[leonardehrenfried]

Reading in Netex data is a complicated topic and only the Nordic profile has been thoroughly tested. We want to support more though.

I recommend that you come to one of our developer meetings to get an idea of what to do to get it to work: https://github.com/opentripplanner/OpenTripPlanner/blob/dev-2.x/CONTRIBUTING.md#developer-meetings

I guarantee you that you will save time doing this.

Otherwise @t2gran is the Netex expert and might see this issue and respond.

[BredeD]

Looks like Slovenia has made their own profile, https://data4pt.org/w/index.php?title=NeTEX

[adnan-smlatic]

@BredeD thanks for the link. I am very new to netex, but really don't understand the point of a "standard" if it allows for individual profiles that differ in structure and don't seem to work with each other.

[leonardehrenfried]

This is indeed a very unfortunate aspect of NeTex.

[BredeD]

It's a long answer to it, but in short, it is a sad result of the incorrect sequence where the regulation pointing to a profile not existing came first and not a well-organized process. The process is now getting better, but the damage has occurred. Advocate for only two profiles in the future, but it will be hard to get there. Have asked the author of the Slovenian profile for assistance to understand what is different.

[adnan-smlatic]

The transport sector in Slovenia is one giant mess, so I am worried for what you might find out about that hah.

Really hope I will some day be able to contribute to this project, still lack a lot of knowledge on it all, but I am extremely grateful for what you guys have done!

Just in case it might be of interest and since it requires individual approval, here is the Slovenian Netex: https://send.vis.ee/download/b43210b639031b37/#71izPSQlyFJnYJShgol1mg

[BredeD]

Slovenia didn't show up in today's Data4PT meeting, no information and possibility to ask questions. I need to let someone dig deeper.

[github-actions[bot]]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 30 days

[BredeD]

I will participate at IT-Trans May 15 in two sessions, both of them with topics of NeTEx and SIRI in Europe based on the implementation of MMTIS regulation. If you have info of attempt of using NeTEx/SIRI for a country NAPs open dataset, give me a heads-up, and I can add it in my panel discussion.

I have not recieved any info from Slovenia, asked again today.

[adnan-smlatic]

I have had a bit of a correspondence with Neža Drobnič at mzi.ncup@gov.si , so you can try getting in touch there. If nothing comes out of it, I'd be happy to try on your behalf as well.

All I've gotten is this link for the technical specification of the Slovenian profile:

https://www.nap.si/_resources/profiles/NeTEx_SI_Profile_v2.pdf

[BredeD]

Translated the profile to English with DeepL. Not sure if we have one familiar with profiles to look at it. NeTEx_SI_Profile_v2 en.pdf

[github-actions[bot]]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 30 days

[BredeD]

There will be new NAPCORE Mobility Data Days in Turin, Italy, in November. The two-day event will include a few sessions on NeTEx and SIRI from the NAPs throughout Europe.

[urosjarc]

@BredeD We also have this issue #6008 and are interested to provide assistance with the solution implementation... We are currently developing application using GTFS but Slovenian NeTEx profile is much more complete.

@BredeD How much work we are talking about in order to implement custom NeTEx profile? Can we do it ourself? Where can we start? Is there anything we can do to assist in this process?

[BredeD]

Hi, we do not aim to support all national profiles; rather, we state the opposite: We will not support all profiles. To succeed with NeTEx in Europe, we must eliminate most of them. We need to use as few as possible. We must use only one profile to get Apple, Google, Here, OMIO, and other worldwide solutions to go from GTFS to NeTEx. We mean the EU EPIP profile as a minimum for use in trip planning and booking solutions. We must get one maximum profile for all other countries that want to do more than trip planning. This maximum profile consists of the EPIP profile plus add-ons.

In the last two years, I have raised this concern in the work within NAPCORE, CEN standardization, and EU regulation work. It would help to give feedback from the NAP data users back to each country. They need to understand that they must provide NeTEx data that is plug-and-play, and a data user should be able to use data from different countries without unique integration for each country.

Therefore, when you work on the Slovenian NeTEx, you should use the validator from Entur and import the NeTEx dataset as we do, or use the EPIP profile and the same validator as @leonardehrenfried uses when he now implements EPIP with Italian NeTEx.

If none of them are working on the Slovenian NeTEx, we need to know the root cause. Then, we can decide whether to change the Nordic Profile, the EPIP Profile, the dataset from Italy, or the dataset from Slovenia.

NAPCORE Mobility Days will be held in Turin, Italy, from November 5 to 7. It is open to everyone, and I would love to have some NeTEx data users there.

[leonardehrenfried]

The Italian profile conforms to EPIP. I'm not supporting any Italy-specific features.

[BredeD]

My point: If two countries say they are EPIP compliant, and it doesn't play together or in the same "open-source project," maybe none are compliant. The work we are doing now is at the forefront. We need to build the fundament for NeTEx and open-source. The results of these two initiatives are valuable for further work and discussion in NAPCORE.

[urosjarc]

@BredeD, @leonardehrenfried Are there any manuals to validate NeTEx format... Where is this Entur validator that you two are talking about?

Also can you elaborate on NAPCORE Mobility Days?

[BredeD]

I missed talking to my colleague today. I'll come back regards validation tool.

NAPCORE is a huge project funded by EU. Aim is to help EU memeberstates to produce data regards ITS regulations. There are 5 Delegated Regulations, where one is for mobility, MMTIS (multimodal travel information services). The aim for MMTIS is to make available throughout Europe, to provide data for all modes of transport for trip planning and data needed for price information, booking and after market. MMTIS points towards the CEN standards Transmodel, NeTEx and SIRI or machine readable formats that is 100% interoperable with them. The implementation of NeTEx in Europe started out wrongly from the beginning. CEN standards was existing when MMTIS got into action, the profiles was missing. Entur and Norway started to work on a profile 2 years before MMTIS got active, we based our profile on the French one, which was based on a UK profile. No other profiles were existing back then. We collaborated with the Nordic countries and succeeded in a joint Nordic profile, many countries started to work on their own. We got the EU EPIP profile a year or two ago, this one was mostly based on the Nordic/French with some adjustments, so some countries has chosen to use that one.

I have received results from initiatives trying to use NeTEx from various NAPs which has all failed. There are traction on this aspect in CEN, NAPCORE and EU.

The NAPCORE Mobility Days is an annual conference where all NAPs meet. EU, CEN, Ministry, Directorates and others will be there, too.

[urosjarc]

@BredeD Is there anything we can do in the mean time to resolve this issue that we have? What are your suggestions?

[BredeD]

Yes, sorry for the delay. The Entur validator Antu runs in three steps:

• NeTEx standard validation
• Profile validation
• Logical validation The two first are of most interest of now. Antu can be used for it.

Send a PM to @vpaturet with the latest dataset, he can do a quick test of it and can help you setup Antu on your side. Fintraffic has done this setup and uses the two first steps in their NAP validation on the Finnish NeTEx.

[leonardehrenfried]

As a first step you can also use xmllint and the official EPIP schema: https://github.com/NeTEx-CEN/NeTEx-Profile-EPIP/blob/master/NeTEx_publication_EPIP.xsd

[urosjarc]

This is all the data that we can get from www.nap.si

• NETEX_PI_01_SI_NAP_FARES.zip
• NETEX_PI_01_SI_NAP_OPERATOR.zip
• NETEX_PI_01_SI_NAP_STOP.zip
• NETEX_PI_01_SI_NAP_NETWORK-00.zip
• NETEX_PI_01_SI_NAP_NETWORK-01.zip

The NETWORK zip is splited on 25mb parts since github file limit is 25 Mb...

The spliting was done by this suggestion: https://superuser.com/a/336224 I have to rename extension .z01 to .zip because github does not recognize .z01 extension, so to merge those zips you will have to rename NETEX_PI_01_SI_NAP_NETWORK-01.zip into NETEX_PI_01_SI_NAP_NETWORK-00.z01 and then merge them with the linux command zip -s0 NETEX_PI_01_SI_NAP_NETWORK-00.zip --out NETEX_PI_01_SI_NAP_NETWORK.zip.

[urosjarc]

@leonardehrenfried @BredeD Thank you for your suggestion, will take a close look into both validators. We will post results of validators here. :+1:

[vpaturet]

@urosjarc Regarding the original question: "Netex file ignored" This is just a configuration issue, the graph builder expects a certain naming convention for the NeTEx files. I was able to import the dataset in OTP and retrieve the Quays and Lines from the API. I recommend that you post your dataset and configuration in the OTP Gitter chat where we can continue the conversation.

https://app.gitter.im/#/room/#opentripplanner_OpenTripPlanner:gitter.im