DB: make routing mode configurable, so that cancelled trips can be queried & pagination works

[lamBOOO]

I noticed that some Deutsche Bahn journeys are missing in the response. Especially, canceled trains seem to not show up.

For example, the ICE 595 from Mannheim to Stuttgart today at 12:30 (2023-04-02T12:30:00+02:00) is missing in the response, although it is listed in the reiseauskunft.bahn.de and in the DB Navigator App (see images below).

I try to use the API with the following code:

let {createDbHafas} = await import('db-hafas');  // REPL
let hafas = createDbHafas('test@test.com')
let journeys = await hafas.journeys('8000244', '8000096', {
    results: 2,
    departure: new Date("2023-04-02T12:14:00+02:00")
})
console.log(JSON.stringify(journeys, null, 4))

The result is:

{
    "laterRef": "2|OF|MT#14#164953#164953#164995#164995#0#0#5#164895#2#0#1050#0#0#-2147483648#1#2|PDH#07b4c623b5c8a2cb5cd37e278d01c93a|RD#2042023|RT#121000|US#0",
    "journeys": [
        {
            "type": "journey",
            "legs": [
                {
                    "origin": {
                        "type": "stop",
                        "id": "8000244",
                        "name": "Mannheim Hbf",
                        "location": {
                            "type": "location",
                            "id": "8000244",
                            "latitude": 49.479181,
                            "longitude": 8.469268
                        },
                        "products": {
                            "nationalExpress": true,
                            "national": true,
                            "regionalExpress": true,
                            "regional": true,
                            "suburban": true,
                            "bus": true,
                            "ferry": false,
                            "subway": false,
                            "tram": true,
                            "taxi": false
                        }
                    },
                    "destination": {
                        "type": "stop",
                        "id": "8000096",
                        "name": "Stuttgart Hbf",
                        "location": {
                            "type": "location",
                            "id": "8000096",
                            "latitude": 48.785052,
                            "longitude": 9.182589
                        },
                        "products": {
                            "nationalExpress": true,
                            "national": true,
                            "regionalExpress": true,
                            "regional": true,
                            "suburban": true,
                            "bus": true,
                            "ferry": false,
                            "subway": false,
                            "tram": true,
                            "taxi": false
                        }
                    },
                    "departure": "2023-04-02T12:14:00+02:00",
                    "plannedDeparture": "2023-04-02T12:14:00+02:00",
                    "departureDelay": null,
                    "arrival": "2023-04-02T13:30:00+02:00",
                    "plannedArrival": "2023-04-02T13:30:00+02:00",
                    "arrivalDelay": null,
                    "reachable": true,
                    "tripId": "1|265123|0|80|2042023",
                    "line": {
                        "type": "line",
                        "id": "flx-1241",
                        "fahrtNr": "1241",
                        "name": "FLX 1241",
                        "public": true,
                        "adminCode": "FLX10_",
                        "productName": "FLX",
                        "mode": "train",
                        "product": "regionalExpress",
                        "operator": {
                            "type": "operator",
                            "id": "flixtrain",
                            "name": "FlixTrain"
                        }
                    },
                    "direction": "Stuttgart Hbf",
                    "arrivalPlatform": "5",
                    "plannedArrivalPlatform": "5",
                    "arrivalPrognosisType": "prognosed",
                    "departurePlatform": "8",
                    "plannedDeparturePlatform": "8",
                    "departurePrognosisType": null
                }
            ],
            "refreshToken": "T$A=1@O=Mannheim Hbf@L=8000244@a=128@$A=1@O=Stuttgart Hbf@L=8000096@a=128@$202304021214$202304021330$FLX 1241$$1$$$$$$",
            "remarks": [],
            "price": null
        },
        {
            "type": "journey",
            "legs": [
                {
                    "origin": {
                        "type": "stop",
                        "id": "8000244",
                        "name": "Mannheim Hbf",
                        "location": {
                            "type": "location",
                            "id": "8000244",
                            "latitude": 49.479181,
                            "longitude": 8.469268
                        },
                        "products": {
                            "nationalExpress": true,
                            "national": true,
                            "regionalExpress": true,
                            "regional": true,
                            "suburban": true,
                            "bus": true,
                            "ferry": false,
                            "subway": false,
                            "tram": true,
                            "taxi": false
                        }
                    },
                    "destination": {
                        "type": "stop",
                        "id": "8000096",
                        "name": "Stuttgart Hbf",
                        "location": {
                            "type": "location",
                            "id": "8000096",
                            "latitude": 48.785052,
                            "longitude": 9.182589
                        },
                        "products": {
                            "nationalExpress": true,
                            "national": true,
                            "regionalExpress": true,
                            "regional": true,
                            "suburban": true,
                            "bus": true,
                            "ferry": false,
                            "subway": false,
                            "tram": true,
                            "taxi": false
                        }
                    },
                    "departure": "2023-04-02T12:43:00+02:00",
                    "plannedDeparture": "2023-04-02T12:43:00+02:00",
                    "departureDelay": null,
                    "arrival": "2023-04-02T13:20:00+02:00",
                    "plannedArrival": "2023-04-02T13:20:00+02:00",
                    "arrivalDelay": null,
                    "reachable": true,
                    "tripId": "1|225991|0|80|2042023",
                    "line": {
                        "type": "line",
                        "id": "ice-917",
                        "fahrtNr": "917",
                        "name": "ICE 917",
                        "public": true,
                        "adminCode": "80____",
                        "productName": "ICE",
                        "mode": "train",
                        "product": "nationalExpress",
                        "operator": {
                            "type": "operator",
                            "id": "db-fernverkehr-ag",
                            "name": "DB Fernverkehr AG"
                        }
                    },
                    "direction": "München Hbf",
                    "currentLocation": {
                        "type": "location",
                        "latitude": 48.333856,
                        "longitude": 10.733192
                    },
                    "arrivalPlatform": "16",
                    "plannedArrivalPlatform": "16",
                    "arrivalPrognosisType": "prognosed",
                    "departurePlatform": "5",
                    "plannedDeparturePlatform": "5",
                    "departurePrognosisType": "prognosed",
                    "loadFactor": "very-high"
                }
            ],
            "refreshToken": "T$A=1@O=Mannheim Hbf@L=8000244@a=128@$A=1@O=Stuttgart Hbf@L=8000096@a=128@$202304021243$202304021320$ICE  917$$1$$$$$$",
            "price": null
        }
    ],
    "realtimeDataUpdatedAt": 1680441011
}

In the response, the cancelled train at 12:30 is missing. Am I doing something wrong or miss a parameter to also include cancelled trains?

When researching, I also found the bahn.expert website where the specific train is also missing in https://bahn.expert/routing/8000244/8000096/2023-04-02T10:14:28.904Z/

DB Website:

DB Navigator:

Bahn Expert:

[derhuerst]

Thank you for documenting this. I'm not sure yet if this is something that hafas-client can influence though.

If possible, can you provide another example, ideally with more time in advance? Alternatively, you can run the script you provided above with DEBUG=hafas-client; It will print the raw (JSON) HAFAS request & response bodies; Paste these here, so that we can have a look if the cancelled trip is actually in the data.

Also, we should check if the routing mode, which is set to "do routing on realtime data" by hafas-client, affects this trip. It might be that the DB Navigator app requests with FULL (or some special DB-specific mode).

[lamBOOO]

Hi @derhuerst,

thanks for the tips! I did some further research and looked for canceled trains. Luckily, there are a lot of cancelled S-Bahn connections between Stuttgart University and Stuttgart Mainstation 😂

Using the debug option, I tried the following prompt:

let {createDbHafas} = await import('db-hafas');  // REPL
let hafas = createDbHafas('test@test.com')
let journeys = await hafas.journeys('8006513', '8000096', {
    results: 5,
    departure: new Date("2023-04-05T19:39:00+02:00"),
    remarks: true
})
console.log(JSON.stringify(journeys, null, 4))

Again, the resulting raw response did not include the cancelled 19:39 train (see image). Note that realtime data vanishes like 60-90mins after departure (or arrival). So, reproducing this example might not be possible.

However, I also logged the network request from the iOS DB Navigator app with the Charles app and found out, that it's raw response did include the cancelled trains. Further inspecting the iOS request header showed, that the mobile app is now using the HYBRID routing mode (see image below).

I then changed the routing mode to HYBRID in /p/db/index.js and db-hafas was able to also show the cancelled train. They were also correctly labelled and got the "cancelled": true information in the journey (see image).

[lamBOOO]

So should we include a mechanism to allow changing the rtMode?

[derhuerst]

Thanks for looking into this!

Conceptually, and assuming that the explanation on HAFAS routing modes that someone has posted is accurate, this makes sense:

• HYBRID AND FULL search on plan/schedule data, therefore also find cancelled connections.
• REALTIME does routing on realtime data, so cancelled journeys shouldn't ever show up because they are not feasible ways to get from origin to destination.

I think that when doing a journeys() query ("I want to go from A to B at a specific point in time"), I'd argue most users want the REALTIME behaviour.

But given that, to my knowledge, HAFAS doesn't provide sufficiently powerful APIs to query those cancelled trips/journeys for use cases that do require them (e.g. "Can I get from A to B as usual today?"), I think we should make rtMode configurable.

I would happily merge a PR that adds an option! If you want to tackle this, please check if rtMode works with other endpoints too, of if it is DB-specific.

[derhuerst]

There is another aspect to this: It seem like that, with rtMode: REALTIME, pagination (via earlierRef a.k.a. outCtxScrB & laterRef a.k.a. outCtxScrF) is not possible! So effectively, there is no reliable way to do a 2nd call to get more than the initial set of journeys. This is another reason why we should make rtMode user-configurable IMO.

I would gladly accept a PR that adds an option rtMode to journeys(); It would be sufficient to just 1) add an entry realtimeRouting: true to opt's defaults, and b) check ctx.opt.realtimeRouting in transformReqBody(). Also, documenting this in p/db/readme.md would be helpful.

[lamBOOO]

Thanks for the suggestions 👍 I once had a quick look but didn't figure out the options handling. Now it's clearer.

1. Wouldn't it be better to allow for all the strings of https://pastebin.com/qZ9WS3Cx instead of just a bool-switch?
2. I think that it does not work with all endpoints.. The p/bls/example.js, for example, throws an error if I change the rtMode to HYBRID in the same way that worked for db.
   • Error: hafasMessage: 'HCI Core: Parse fail : Parser error: root.svcReqL.svcReqL.cfg.rtMode(HYBRID)'
   • => Implies that the option is not even recognized.
3. So let's test which endpoints are affected and for the unaffected ones, the opt.realtimeRouting in journeys() then doesn't have any effect. Is that the correct plan?

[derhuerst]

[...] It would be sufficient to just 1) add an entry realtimeRouting: true to opt's defaults [...].

Wouldn't it be better to allow for all the strings of https://pastebin.com/qZ9WS3Cx instead of just a bool-switch?

My idea behind opt.realtimeRouting: true was that it's a more intuitive API and that almost all people probably don't need the flexibility that the HAFAS routing modi provide. But maybe my gut feeling is wrong, and it makes sense to expose the modi as-is (but behind properly named constants, similar to how it's done with DB's traveller age groups). What do you think?

2. I think that [rtMode: HYBRID] does not work with all endpoints.. The p/bls/example.js, for example, throws an error […].

It's likely specific to the DB endpoint, given that they have historically often had some cusomisations done to HAFAS; Or it's specific to some (multiple individual) endpoints, not sure.

3. So let's test which endpoints are affected and for the unaffected ones, the opt.realtimeRouting in journeys() then doesn't have any effect. Is that the correct plan?

My idea would be to only recognise the opt.realtimeRouting/opt.routingMode field in those profiles whose HAFAS endpoints support it, just like how the DB profile currently always sets rtMode: HYBRID:

https://github.com/public-transport/hafas-client/blob/4cb70623025668a14d4493f3137711652e0089cc/p/db/index.js#L27-L34

[lamBOOO]

I would also assume that most people want the realtime, but just thought about what the other option should be if realtimeRouting: false is set.

So should it be FULL, INFO, OFF or the actual data that is shown on the mobile app with HYBRID. So a binary switch for these five options (currently) doesn't sound right 😄 I personally like the idea of exposing everything via the constants (age-group style). It's the same construction as for the loyality cards right?

If we decide to export all modi, then there no need to add something to the default opts, right? Since it's endpoint specific, we can apply the same strategy as for the loyality cards, as explained here. Is that a good idea?

[derhuerst]

[…] via the constants (age-group style). It's the same construction as for the loyality cards right?

Yes, pretty much. I like the use of Symbols (as with the loyalty cards) even more, because this API is harder to use in a wrong way, but Symbols don't work well with serialisation, e.g. in hafas-client-rpc, so I now prefer the age-groups style (plain string constants).

If we decide to export all modi, then there no need to add something to the default opts, right? Since it's endpoint specific, we can apply the same strategy as for the loyality cards, as explained here. Is that a good idea?

Yes. I would just make the constants available via lib/routing-modes.js.

[derhuerst]

I have published the fix (#295) as hafas-client@6.1.0. 🎉