Not all journeys are flagged as canceled

[timaschew]

Today my regional train was canceled, it was already announced yesterday and I've realized that my Android app Transportr was only showing the canceled flag for the next two hours/journeys but not on the subsequent journeys even if it's already known that they will be canceled.

So I checked what data the hafas client will give me and it gives me the same output unfortunately:

{ earlierRef:
   '1|OB|MT#11#16447#16447#16459#16459#0#0#165#16431#1#-2147483638#0#1#2|PDH#1f3cc62a0dc25c84b5715f031e292b06|RD#15102019|RT#95131',
  laterRef:
   '1|OF|MT#11#16687#16687#16699#16699#0#0#5#16628#5#-2147483638#0#1#2|PDH#1f3cc62a0dc25c84b5715f031e292b06|RD#15102019|RT#95131',
  journeys:
   [ { type: 'journey',
       legs:
        [ { origin:
             { type: 'stop',
               id: '900000310040',
               name: 'Bad Saarow, Bhf',
               location:
                { type: 'location',
                  id: '900310040',
                  latitude: 52.293605,
                  longitude: 14.064756 },
               products:
                { suburban: false,
                  subway: false,
                  tram: false,
                  bus: true,
                  ferry: false,
                  express: false,
                  regional: true } },
            destination:
             { type: 'stop',
               id: '900000310001',
               name: 'Fürstenwalde, Bhf',
               location:
                { type: 'location',
                  id: '900310001',
                  latitude: 52.367029,
                  longitude: 14.059614 },
               products:
                { suburban: false,
                  subway: false,
                  tram: false,
                  bus: true,
                  ferry: false,
                  express: false,
                  regional: true } },
            departure: null,
            arrival: null,
            reachable: false,
            tripId: '1|29322|4|86|15102019',
            line:
             { type: 'line',
               id: 'rb35',
               fahrtNr: '1150',
               name: 'RB35',
               public: true,
               mode: 'train',
               product: 'regional',
               operator:
                { type: 'operator',
                  id: 'neb-betriebsgesellschaft-mbh',
                  name: 'NEB Betriebsgesellschaft mbH' },
               symbol: 'RB',
               nr: 35,
               metro: false,
               express: false,
               night: false },
            direction: 'Fürstenwalde, Bhf',
            arrivalPlatform: null,
            departurePlatform: null,
            cycle: { min: 3600, max: 3600, nr: 1 },
            cancelled: true,
            arrivalDelay: null,
            scheduledArrival: '2019-10-15T10:19:00+02:00',
            departureDelay: null,
            scheduledDeparture: '2019-10-15T10:07:00+02:00' } ],
       refreshToken:
        '¶HKI¶T$A=1@O=Bad Saarow, Bahnhof@L=900310040@a=128@$A=1@O=Fürstenwalde, Bahnhof@L=900310001@a=128@$201910151007$201910151019$    RB35$$1$',
       cycle: { min: 3600 },
       remarks:
        [ { type: 'status',
            code: 'text.realtime.connection.cancelled',
            text: 'Cancelled' } ] },
     { type: 'journey',
       legs:
        [ { origin:
             { type: 'stop',
               id: '900000310040',
               name: 'Bad Saarow, Bhf',
               location:
                { type: 'location',
                  id: '900310040',
                  latitude: 52.293605,
                  longitude: 14.064756 },
               products:
                { suburban: false,
                  subway: false,
                  tram: false,
                  bus: true,
                  ferry: false,
                  express: false,
                  regional: true } },
            destination:
             { type: 'stop',
               id: '900000310001',
               name: 'Fürstenwalde, Bhf',
               location:
                { type: 'location',
                  id: '900310001',
                  latitude: 52.367029,
                  longitude: 14.059614 },
               products:
                { suburban: false,
                  subway: false,
                  tram: false,
                  bus: true,
                  ferry: false,
                  express: false,
                  regional: true } },
            departure: null,
            arrival: null,
            reachable: false,
            tripId: '1|29322|5|86|15102019',
            line:
             { type: 'line',
               id: 'rb35',
               fahrtNr: '1149',
               name: 'RB35',
               public: true,
               mode: 'train',
               product: 'regional',
               operator:
                { type: 'operator',
                  id: 'neb-betriebsgesellschaft-mbh',
                  name: 'NEB Betriebsgesellschaft mbH' },
               symbol: 'RB',
               nr: 35,
               metro: false,
               express: false,
               night: false },
            direction: 'Fürstenwalde, Bhf',
            arrivalPlatform: null,
            departurePlatform: null,
            cycle: { min: 3600, max: 3600, nr: 2 },
            cancelled: true,
            arrivalDelay: null,
            scheduledArrival: '2019-10-15T11:19:00+02:00',
            departureDelay: null,
            scheduledDeparture: '2019-10-15T11:07:00+02:00' } ],
       refreshToken:
        '¶HKI¶T$A=1@O=Bad Saarow, Bahnhof@L=900310040@a=128@$A=1@O=Fürstenwalde, Bahnhof@L=900310001@a=128@$201910151107$201910151119$    RB35$$1$',
       cycle: { min: 3600 },
       remarks:
        [ { type: 'status',
            code: 'text.realtime.connection.cancelled',
            text: 'Cancelled' } ] },
     { type: 'journey',
       legs:
        [ { origin:
             { type: 'stop',
               id: '900000310040',
               name: 'Bad Saarow, Bhf',
               location:
                { type: 'location',
                  id: '900310040',
                  latitude: 52.293605,
                  longitude: 14.064756 },
               products:
                { suburban: false,
                  subway: false,
                  tram: false,
                  bus: true,
                  ferry: false,
                  express: false,
                  regional: true } },
            destination:
             { type: 'stop',
               id: '900000310001',
               name: 'Fürstenwalde, Bhf',
               location:
                { type: 'location',
                  id: '900310001',
                  latitude: 52.367029,
                  longitude: 14.059614 },
               products:
                { suburban: false,
                  subway: false,
                  tram: false,
                  bus: true,
                  ferry: false,
                  express: false,
                  regional: true } },
            departure: '2019-10-15T12:07:00+02:00',
            arrival: '2019-10-15T12:19:00+02:00',
            reachable: true,
            tripId: '1|29322|6|86|15102019',
            line:
             { type: 'line',
               id: 'rb35',
               fahrtNr: '1148',
               name: 'RB35',
               public: true,
               mode: 'train',
               product: 'regional',
               operator:
                { type: 'operator',
                  id: 'neb-betriebsgesellschaft-mbh',
                  name: 'NEB Betriebsgesellschaft mbH' },
               symbol: 'RB',
               nr: 35,
               metro: false,
               express: false,
               night: false },
            direction: 'Fürstenwalde, Bhf',
            arrivalPlatform: '3',
            departurePlatform: null,
            cycle: { min: 3600, max: 3600, nr: 3 } } ],
       refreshToken:
        '¶HKI¶T$A=1@O=Bad Saarow, Bahnhof@L=900310040@a=128@$A=1@O=Fürstenwalde, Bahnhof@L=900310001@a=128@$201910151207$201910151219$    RB35$$1$',
       cycle: { min: 3600 } },
     { type: 'journey',
       legs:
        [ { origin:
             { type: 'stop',
               id: '900000310040',
               name: 'Bad Saarow, Bhf',
               location:
                { type: 'location',
                  id: '900310040',
                  latitude: 52.293605,
                  longitude: 14.064756 },
               products:
                { suburban: false,
                  subway: false,
                  tram: false,
                  bus: true,
                  ferry: false,
                  express: false,
                  regional: true } },
            destination:
             { type: 'stop',
               id: '900000310001',
               name: 'Fürstenwalde, Bhf',
               location:
                { type: 'location',
                  id: '900310001',
                  latitude: 52.367029,
                  longitude: 14.059614 },
               products:
                { suburban: false,
                  subway: false,
                  tram: false,
                  bus: true,
                  ferry: false,
                  express: false,
                  regional: true } },
            departure: '2019-10-15T13:07:00+02:00',
            arrival: '2019-10-15T13:19:00+02:00',
            reachable: true,
            tripId: '1|29322|7|86|15102019',
            line:
             { type: 'line',
               id: 'rb35',
               fahrtNr: '1147',
               name: 'RB35',
               public: true,
               mode: 'train',
               product: 'regional',
               operator:
                { type: 'operator',
                  id: 'neb-betriebsgesellschaft-mbh',
                  name: 'NEB Betriebsgesellschaft mbH' },
               symbol: 'RB',
               nr: 35,
               metro: false,
               express: false,
               night: false },
            direction: 'Fürstenwalde, Bhf',
            arrivalPlatform: '3',
            departurePlatform: null,
            cycle: { min: 3600, max: 3600, nr: 3 } } ],
       refreshToken:
        '¶HKI¶T$A=1@O=Bad Saarow, Bahnhof@L=900310040@a=128@$A=1@O=Fürstenwalde, Bahnhof@L=900310001@a=128@$201910151307$201910151319$    RB35$$1$',
       cycle: { min: 3600 } },
     { type: 'journey',
       legs:
        [ { origin:
             { type: 'stop',
               id: '900000310040',
               name: 'Bad Saarow, Bhf',
               location:
                { type: 'location',
                  id: '900310040',
                  latitude: 52.293605,
                  longitude: 14.064756 },
               products:
                { suburban: false,
                  subway: false,
                  tram: false,
                  bus: true,
                  ferry: false,
                  express: false,
                  regional: true } },
            destination:
             { type: 'stop',
               id: '900000310001',
               name: 'Fürstenwalde, Bhf',
               location:
                { type: 'location',
                  id: '900310001',
                  latitude: 52.367029,
                  longitude: 14.059614 },
               products:
                { suburban: false,
                  subway: false,
                  tram: false,
                  bus: true,
                  ferry: false,
                  express: false,
                  regional: true } },
            departure: '2019-10-15T14:07:00+02:00',
            arrival: '2019-10-15T14:19:00+02:00',
            reachable: true,
            tripId: '1|29322|8|86|15102019',
            line:
             { type: 'line',
               id: 'rb35',
               fahrtNr: '1146',
               name: 'RB35',
               public: true,
               mode: 'train',
               product: 'regional',
               operator:
                { type: 'operator',
                  id: 'neb-betriebsgesellschaft-mbh',
                  name: 'NEB Betriebsgesellschaft mbH' },
               symbol: 'RB',
               nr: 35,
               metro: false,
               express: false,
               night: false },
            direction: 'Fürstenwalde, Bhf',
            arrivalPlatform: '3',
            departurePlatform: null,
            cycle: { min: 3600, max: 3600, nr: 3 } } ],
       refreshToken:
        '¶HKI¶T$A=1@O=Bad Saarow, Bahnhof@L=900310040@a=128@$A=1@O=Fürstenwalde, Bahnhof@L=900310001@a=128@$201910151407$201910151419$    RB35$$1$',
       cycle: { min: 3600 } } ] }

Then I checked the websites of Deutsche Bahn, BVG and VBB and I've realized that only VBB is showing the 2 hours as canceled, but at least they show warnings for later journeys. But DB and BVG are showing cancelations for all journeys (the planned outage is only until 2pm).

Can we bring the warning somehow into the response of the journey?

Here are some screenshots and the warning message:

[derhuerst]

[...] Transportr was only showing the canceled flag for the next two hours/journeys [...]. So I checked what data the hafas client will give me and it gives me the same output unfortunately:

This is actually a good sign, because it at least it returns the same data.

The fact that the different HAFAS endpoints have different delay data, as well as different sets of human-written warnings, is unfortunately nothing that this project can change. From what I've heard, it's because of bureaucratic and complex technical reasons (which mostly sound unreasonable from an outside perspective) that won't disappear soon, so we can only work around it.

I've started doing that with ~two projects find-db-hafas-leg-in-another-hafas and augment-vbb-hafas that I've forgot to publish (will do that as soon as I have time)~ find-hafas-leg-in-another-hafas, but merging data from two HAFAS (or even non-HAFAS) endpoints is a complex problem. With pan-european-public-transport, I'm trying to take this approach as far as possible.

[...] only VBB is showing the 2 hours as canceled, but at least they show warnings for later journeys. [...] Can we bring the warning somehow into the response of the journey?

Yes. Please find a future connection (ideally >2 days in the future, so we can debug properly) and query with the vbb and/or bvg profiles, with remarks: true. Hopefully it will give you some results.

If it doesn't: Parsing the semi-structured messages is hard, and currently hafas-client parses only some of them (here and here. AFAIK public-transport-enabler and TripKit als only parse some.

I recently talked to @opyh though, and they have a lot of recorded messages or a better parsing logic. See also #5, #90 and #131 for tracking Issues.

[derhuerst]

Closing this because we can't really do something about the low-quality upstream data, just work around it (by putting layers of abstraction on top, like in pan-european-public-transport).