DB endpoint responds with CLIENTVERSION when using user agents from docs

[Issam-Jendoubi]

We faced a CLIENTVERSION error with thehafas-client using the db profile since some days. Basically, all endpoints are failing (journeys, locations ...) The same issue is happening on these 3 different environments:

• hafas-client version 3.4.3 with node 8.
• hafas-client version 5.22.2 with node 16.
• hafas-client version 6.0.2 with node 18.

We tried with other profiles such as vsn or vvv and it works. The issue comes obviously from the URL of DB profile as stated in the logs section below.

We could not know further about its reasons. Any idea about what is going wrong with the DB profile.

Error: CLIENTVERSION
    at Object.request (/private/tmp/hafas-client-test/node_modules/hafas-client/lib/request.js:131:14)
    at Object.locations (/private/tmp/hafas-client-test/node_modules/hafas-client/index.js:412:18)
    at file:///private/tmp/hafas-client-test/index.js:8:30
    at ModuleJob.run (node:internal/modules/esm/module_job:198:25)
    at async Promise.all (index 0)
    at async ESMLoader.import (node:internal/modules/esm/loader:385:24)
    at async loadESM (node:internal/process/esm_loader:88:5)
    at async handleMainPromise (node:internal/modules/run_main:61:12) {
  isHafasError: true,
  request: '{"lang":"en","svcReqL":[{"cfg":{"polyEnc":"GPA","rtMode":"HYBRID"},"meth":"LocMatch","req":{"input":{"loc":{"type":"S","name":"stuttgart?"},"maxLoc":5,"field":"S"}}}],"client":{"type":"AND","id":"DB","v":19040000,"name":"DB Navigator"},"ext":"DB.R20.12.b","ver":"1.34","auth":{"type":"AID","aid":"n91dB8Z77MLdoR0K"}}',
  url: 'https://reiseauskunft.bahn.de/bin/mgate.exe?checksum=00b065a00c13e52bf4148d1a40cc805f',
  statusCode: 200,
  code: 'CLIENTVERSION',
  responseId: null
}

[VanDerLars]

The issue seems to be the following: When you follow the documentation and instantiate the hafas-client with this: const client = createClient(dbProfile, 'my-awesome-program') then the request are failing.

We've never changed the string my-awesome-program in one project, while we changed it in another. While the first project stopped working days ago, the second still works.

My assumption is, that the DB or someone else started to restrict calls from clients called my-awesome-program. Maybe because they want to ban hobby projects or what ever reason.

As soon as we changed this my-awesome-program string to another, the calls started to work again.

@derhuerst can you confirm this behavior? Should you make a note in the documentation or use another placeholder-client-name? What reasons else could be there that this specific string my-awesome-program stopped working?

[VanDerLars]

It is EXACTLY this string my-awesome-program which is restricted.

• my-awesome-Program works (capital P)
• my-awesomeprogram works
• and so on

[maxpietzsch]

I can confirm the observations of @VanDerLars but for us it was about the strings hafas-client-example and hafas-client.

Just changing these to hafas-client-examplE or hafas-clienT did already help.

[VanDerLars]

So, it looks like that the DB starts to restrict the use of the API, doesn't it? That's a critical development.

[derhuerst]

We've never changed the string my-awesome-program in one project, while we changed it in another. While the first project stopped working days ago, the second still works.

My assumption is, that the DB or someone else started to restrict calls from clients called my-awesome-program. Maybe because they want to ban hobby projects or what ever reason.

From my experience, Deutsche Bahn has too many sub-companies and teams to have a homogenous view on this topic, but mostly, IMHO, it can be summarised that even the least open-access-to-their-APIs-friendly teams consider some access okay (not saying anything about legal aspects here, IANAL!), and – given that their official APIs lack a lot of features – seem to accept reasonable amounts of queries.

However, this requires that everyone plays nice and doesn't send too many requests! Also, I'm pretty sure they would like to have a vague impression of who's actually causing how much traffic on their API, which is why I have added a mandatory User-agent-like parameter to hafas-client, so that people pick a sensible string that explains who they are or at least their project's URL or name.

I'm a bit disappointed that you folks apparently just copied the string my-example-program. 😕 – I think you can imagine that, over time (strangers sending requests as my-example-program over and over again) this really erodes trust on their side that hafas-client users make an effort to be transparent about their usage of the API.

[…] for us it was about the strings hafas-client-example and hafas-client. Just changing these to hafas-client-examplE or hafas-clienT did already help.

While, technically, this solves the immediate problem for you, this is not what you're supposed to do. – Please provide a meaningful way to either a) get in touch with you about the traffic, or b) a way to find the project (e.g. a repo URL) it causes.

So, it looks like that the DB starts to restrict the use of the API, doesn't it? That's a critical development.

As far as I can tell, I'm not sure though, their apparent policy/behaviour described above has been in place for several years already. It could of course be that they have recently decided to be more strict about access to their API. Another possible cause is that yet another person sent traffic as my-awesome-program or hafas-client-example, possibly a lot, and that they intervened.

Given that hafas-client is just "out there" for people to use in order access their (basically unprotected) API, we as its users need to find a way to make sure we're not causing too much trouble.

[derhuerst]

As a drastic step to get people to specify their own user agent, I'm thinking about changing hafas-client to throw if my-example-program is being passed in. Not sure about this yet though.

[VanDerLars]

I'm a bit disappointed that you folks apparently just copied the string my-example-program.

For our case, we're using a very descriptive client name in our main calls. We only forgot to change the string in a smaller sub-system (which then stopped working), so I am pretty sure that the block from DB side wasn't because of us - we use this sub-system very rarely and the main system definitely more often.

But also, you might at least want to update the documentation that changing the string is crucial. Sometimes test-projects grow into production and then such a carelessness stays there forever.

As a drastic step to get people to specify their own user agent, I'm thinking about changing hafas-client to throw if my-example-program is being passed in. Not sure about this yet though.

IMHO, you should do this. As we now see, it is mandatory for the whole project to maintain the good will of the DB. And this comes with a fair use and and a fair display of your actions. When you missuse the endpoints, then you should't be able to affect others.

I know this might be a breaking change, but it'll break even more projects when the DB continues to block generic client names.

[derhuerst]

For our case, we're using a very descriptive client name in our main calls. We only forgot to change the string in a smaller sub-system (which then stopped working), so I am pretty sure that the block from DB side wasn't because of us - we use this sub-system very rarely and the main system definitely more often.

Good to hear, and thanks for taking care!

But also, you might at least want to update the documentation that changing the string is crucial. Sometimes test-projects grow into production and then such a carelessness stays there forever.

Will definitely change it, albeit slightly. 👍 Don't want to put a lengthy explainer in every place where createClient is mentioned (~50), but e.g. const userAgent = 'link-to-my-awesome-project'; // change this!.

[maxpietzsch]

While, technically, this solves the immediate problem for you, this is of course not what you're supposed to do.

I think by only writing such a short comment this created a misunderstanding. The phrasing "did already help" was supposed to relate to and confirm the above mentioned observations when testing, that requests are answered again when only changing one letter to a captial one.

As from this point on it was apparent to us that this string was and is included in the requests against the server (as you confirmed further with the explanation "User-agent-like parameter") to identify the requesting instance, we already changed this to a more meaningful string, for the endpoint to be able to categorize our traffic.

Please provide a meaningful way to either a) get in touch with you about the traffic, or b) a way to find the project (e.g. a repo URL) it causes.

With regards to your suggestion we now have appended contact info to the string.

So I second both the before mentioned documentation update and the throw when default strings are used, to force us to use a custom string for identification. In the long run this will definetly be the better way for all of us that are relying on the generous openess of this API.

[derhuerst]

While, technically, this solves the immediate problem for you, this is of course not what you're supposed to do.

I think by only writing such a short comment this created a misunderstanding.

I missed that this apparently was a misunderstanding, and not a salty comment, sorry. I have adapted my previous post to sound less aggressive.

[maxpietzsch]

All good, no worries.

[derhuerst]

As a drastic step to get people to specify their own user agent, I'm thinking about changing hafas-client to throw if my-example-program is being passed in. Not sure about this yet though.

IMHO, you should do this. As we now see, it is mandatory for the whole project to maintain the good will of the DB. And this comes with a fair use and and a fair display of your actions. When you missuse the endpoints, then you should't be able to affect others.

I know this might be a breaking change, but it'll break even more projects when the DB continues to block generic client names.

I have done this in https://github.com/public-transport/hafas-client/commit/e7602e6c848f95a1ac22a0764eccad79c3a4c37c, published as hafas-client@6.0.3. :shipit: So be careful when upgrading to it! 💥

[derhuerst]

Also published as hafas-client@5.26.3.