get the public endpoint running again (use BVG endpoint for now)

berlincount commented 6 years ago

$ curl -v 'https://2.vbb.transport.rest/stations/9007102/departures?when=1529416634&duration=30'
* Connected to 2.vbb.transport.rest (54.37.75.136) port 443 (#0)
> GET /stations/9007102/departures?when=1529416634&duration=30 HTTP/1.1
> Host: 2.vbb.transport.rest
> User-Agent: curl/7.47.0
> Accept: */*
>
< HTTP/1.1 504 Gateway Time-out
< Server: nginx/1.10.3 (Ubuntu)
< Date: Tue, 19 Jun 2018 13:53:00 GMT
< Content-Type: text/html
< Content-Length: 192
< Connection: keep-alive
<
<html>
<head><title>504 Gateway Time-out</title></head>
<body bgcolor="white">
<center><h1>504 Gateway Time-out</h1></center>
<hr><center>nginx/1.10.3 (Ubuntu)</center>
</body>
</html>

thanks for having a look!

joecorcoran commented 6 years ago

Yeah the V2 departures endpoint has been down since sometime yesterday. Looks like an upstream connection is gone? Not clear whether it's this project or the actual underlying API that has the problem. Anything we can do to help?

https://status.transport.rest/779961406

joecorcoran commented 6 years ago

I can run this server locally and the departures endpoint seems fine, so that makes me think the problem is not upstream. Maybe just needs restarting?

curl http://localhost:3000/stations/900000120004/departures | jq .[0].journeyId

"1|31739|0|86|20062018"

derhuerst commented 6 years ago

The upstream endpoint has often been down for very brief periods of time in the past days.

I can run this server locally and the departures endpoint seems fine, so that makes me think the problem is not upstream.

I can confirm this. Seems like they have blocked the IP address of my server or something. Will look into it later.

rejas commented 6 years ago

Any news on this?

derhuerst commented 6 years ago

I requested a new IPv4 address for my VPS. Would like to avoid to moving the API over to another VPS.

If someone wants to host the endpoint or a fallback, let me know!

derhuerst commented 6 years ago

Still waiting for an IP address and currently don't have the time to migrate to a different VPS/VPS provider.

Again: If anyone is interested in hosting it, let me know! A really small VPS with nginx and Node.js installed is sufficient. @juliuste are you interested?

juliuste commented 6 years ago

I can do it.

juliuste commented 6 years ago

Endpoint should be up again.

derhuerst commented 6 years ago

Closing for now. We should find a long-term solution though for what I think is the cause: The VBB can block the IP address of the API server.

joecorcoran commented 6 years ago

Thanks for solving this. I'm happy to contribute some € to the running costs. Let me know if that's possible.

derhuerst commented 6 years ago

@juliuste Do you have a PayPal/Patreon/OpenCollective/bank/crypto currency account?

You can support me on my Patreon page or via SEPA if you want to skip the fees. Keep in mind though:

Recently, I've started to work at Deutsche Bahn. I'm aware that I'm in a privileged position. Nevertheless, the sector of public transport is a place where open source work like mine is often not valued, let alone seen as necessary. Especially connecting public transport networks, for the benefit of everyone, is something that almost no company pursues. Independent support will allow me to keep working on what I – and assume you as well – think is truly important for passenger-friendly, accessible public transport services.

derhuerst commented 6 years ago

BTW @joecorcoran what are you building with the API?

juliuste commented 6 years ago

Thanks for solving this. I'm happy to contribute some € to the running costs. Let me know if that's possible.

@juliuste Do you have a PayPal/Patreon/OpenCollective/bank/crypto currency account?

Thank you very much, but I'm alright, just support @derhuerst if you want to. 🙂

rejas commented 6 years ago

@derhuerst how much traffic would a VPS generate if I'd rent something to be a fallback?

derhuerst commented 6 years ago

@rejas Not a lot. For a while I've hosted it on a 3€ Scaleway VPS. It should be around 50k-100k requests per day, but vbb-rest is very light on the CPU, it basically just parses JSON.

derhuerst commented 6 years ago

Not sure how one would do a fallback for this though without adding too much tooling. Maybe a load balancer that supports an /health HTTP route.

derhuerst commented 6 years ago

I will also investigate if vbb-rest runs out of the box on Now. Then we'd have one fallback endpoint for free.

joecorcoran commented 6 years ago

BTW @joecorcoran what are you building with the API?

I'm building a departure board app, designed for leaving open on a screen. Taking it slowly as I've never used React before. :)

derhuerst commented 6 years ago

Have deployed a fallback endpoint at https://vbb-rest-ribczgxrzz.now.sh. We still need to figure out how to do the load balancing though.

juliuste commented 6 years ago

Seems like the new endpoint has been blocked as well. @derhuerst we could use a load balancer + different now instances, for example…

derhuerst commented 6 years ago

@joecorcoran @rejas @berlincount can you deploy vbb-rest via Now? With a (free) account, run now in the vbb-rest repo and post me the URL it returns.

derhuerst commented 6 years ago

https://vbb-rest-usfolnrykj.now.sh
https://vbb-rest-ribczgxrzz.now.sh

derhuerst commented 6 years ago

Didn't get it to work using nginx, trying haproxy now. If someone knows any other simple and efficient load balancing solution, let me know.

joecorcoran commented 6 years ago

https://vbb-rest-mudlmnzhtp.now.sh

deg0nz commented 6 years ago

@derhuerst Don't you have a chance to talk to the VBB guys about this? Is there any communication channel? And do you know why they blocked the IPs? Too many request over a period of time or something?

And what was the problem with nginx as load balancer? I never configured it as a load balancer, but the docs about this seemed pretty easy to do it.

derhuerst commented 6 years ago

Don't you have a chance to talk to the VBB guys about this?

Maybe, I will try.

Is there any communication channel?

They have a Twitter account, but they haven't even responded to our request to use HTTPS, which seems more relevant to all users.

And do you know why they blocked the IPs? Too many request over a period of time or something?

Nope, haven't investigated when specifically and how many requests I had sent before. But I didn't see quota-based temporary blockings before, so I assume that the current blocking (of the server IP address) is permanent.

This is what I get on my server:

$ curl -4 -v 'https://fahrinfo.vbb.de/'
*   Trying 83.246.89.20...
* connect to 83.246.89.20 port 443 failed: Connection timed out
* Failed to connect to fahrinfo.vbb.de port 443: Connection timed out
* Closing connection 0
curl: (7) Failed to connect to fahrinfo.vbb.de port 443: Connection timed out

And what was the problem with nginx as load balancer? I never configured it as a load balancer, but the docs about this seemed pretty easy to do it.

Indeed, the markup is pretty simple. There are minor caveats:

The free nginx variant only allows for passive monitoring based on returned status codes. Active HEAD /health checks are only available in the paid variant.
nginx seems to allow either only HTTPS upstream URLS or only HTTP. This is not really an issue though.
I don't want to mark an upstream vbb-rest node as down if it responded with HAFAS errors caused by invalid station IDs, but only if node itself is down or its connection to HAFAS is blocked.

I have also tried haproxy, but had these issues:

Apparently I can only use a static Host header for health checks. This doesn't work with any reverse proxy setup like nginx with multiple domains or with Now.

Otherwise haproxy looks very promising!

Thinking more about the problem, I came to the conclusion that I can solve other problems I've wanted to solve for a while as well. I've started to write a WebSockets-based load balancing mechanism. You might think: Why use a proprietary load balancing mechanism instead of off-the-shelf software?

I can think of the following advantages:

The load-balanced interface is exactly the same as that of hafas-client. [edit:] This allows using it in other places than vbb-rest/db-rest, e.g. hafas-monitor-departures or hafas-discover-stations.
Because of the above, reduces the number of components in the chain that I/we have to maintain. The vbb-rest -> hafas-client-rpc -> hafas-client x N depends less on the vbb-rest interface.
WebSockets are more efficient. The load balancer will maintain one permanent TLS connection to each node. Calls only have to be serialised to and parsed from JSON.
Some clients might prefer a WebSockets-based interface over an HTTP based.

I see these disadvantages:

It uses custom load balancing & connection handling instead of battle-tested tools like nginx or haproxy.
If people want to contribute a node to the pool and have their own API, they have to run two services.

Even though building this is fun, I should probably have a look at Træfik as well.

derhuerst commented 6 years ago

Will keep you updated about my efforts to get 2.vbb.transport.rest up again.

derhuerst commented 6 years ago

Everyone can help me by deploying vbb-hafas-rpc-server on their server or using Now: Install Now CLI, run now inside the repo.

The nodes will be used to load-balance requests to 2.vbb.transport.rest via WebSockets, as explained above. This setup is not yet live at 2.vbb.transport.rest, but will be soon.

Comment with the URL of the deployment. I will update the list below:

wss://vbb-hafas-rpc-server-wfjpndyabe.now.sh
wss://vbb-hafas-rpc-server-tmswhngblr.now.sh
wss://vbb-hafas-rpc-server-vuzjveicht.now.sh
wss://vbb-hafas-rpc-server-iwpbawrcys.now.sh

@juliuste @joecorcoran @deg0nz @berlincount

Let's see if this works as a mitigation. Otherwise people will have to run their own servers locally/privately.

ferby09 commented 6 years ago

vbb-hafas-rpc-server-tmswhngblr.now.sh

I downloaded your repo and ran now-win.exe inside the folder. I hope that was the right way to do it. How will it be updated?

k-nut commented 6 years ago

wss://vbb-hafas-rpc-server-vuzjveicht.now.sh

derhuerst commented 6 years ago

@deg0nz I've deployed the load-balanced API with the 4 nodes mentioned above. Let's see if this is sufficient as a long-term solution.

deg0nz commented 6 years ago

Awesome! Thank you very much for your effort! Do you need an instance more? If yes, I would deploy one as well.

derhuerst commented 6 years ago

@deg0nz Yes, the more the better!

deg0nz commented 6 years ago

Here's my instance: wss://vbb-hafas-rpc-server-bmbdjpmewd.now.sh

joecorcoran commented 6 years ago

Here's another https://vbb-hafas-rpc-server-iwpbawrcys.now.sh

derhuerst commented 6 years ago

As you might have noticed, the endpoint is down again because VBB has blocked the new nodes as well. I won't run the public endpoint at 2.vbb.transport.rest for the time being. Maybe VBB and I will find a compromise when we meet next week. Read above for more details.

If you want to use consume VBB data using JavaScript, use vbb-hafas directly, otherwise host your own vbb-rest instance for now.

To quote deg0nz/MMM-PublicTransportBerlin#32:

Hm. Maybe this should be a sidecar to deploy locally, i.e. everyone should be able to run the VBB REST translation service locally instead of depending on an external third party ...

This is indeed a good idea if you depend on it for critical things! (The upstream VBB API is not too reliable either, though.) Or use the underlying lib vbb-hafas directly if you use JS.

For all other magic mirrors and quick hack projects, this is not feasible. The goal is that, eventually, public transport providers will host a proper API themselves. Until then, we need proxies like 2.vbb.transport.rest.

rejas commented 5 years ago

Fingers crossed for a succesfull meeting with VBB @derhuerst At least they speak to you, had feared they wouldnt react at all.

berlincount commented 5 years ago

Is there any update? :)

derhuerst commented 5 years ago

Cross-post from the transport.rest RSS feed:

@juliuste and I have met recently with VBB. We were able to communicate:

our desire to keep this API running,

that the 2.vbb.transport.rest is no responsible for the amount of traffic they've received with a hafas-client User-Agent,

that an open GTFS RT feed would solve many issues for both parties.

I will elaborate more on that tonight.

derhuerst commented 5 years ago

Good news everyone!

I have set up a new API: 1.bvg.transport.rest provides data from the BVG endpoint. It returns almost exactly the same data as 2.vbb.transport.rest, but in the output format of hafas-client@3.0.0-alpha.20, so you will have to make minor adjustments. Check the docs at derhuerst/bvg-rest!

Let's see for how long this API will stay up.

timaschew commented 5 years ago

If you want to use consume VBB data using JavaScript, use vbb-hafas directly, otherwise host your own vbb-rest instance for now.

Tried to do that (instlaling bvg-rest, which is using hafas-client@3.0.0-alpha.20) but get Bad Gateway back. I tried to find out which credentials are used and found these lines: https://github.com/public-transport/hafas-client/blob/0b81a59c71d07bd31e2ae809b4d089eca11d8f88/p/vbb/index.js#L22 https://github.com/public-transport/hafas-client/blob/0b81a59c71d07bd31e2ae809b4d089eca11d8f88/p/vbb/index.js#L158

I wonder if they were also blocked. I asked now VBB for my personal API key, I hope that will work if I replace my personal client ID? But I guess hafas-client needs to support an environment variable, because I don't see how to pass this down even from hafas-client/p/bvg

derhuerst commented 5 years ago

Tried to do that (instlaling bvg-rest, which is using hafas-client@3.0.0-alpha.20) but get Bad Gateway back.

Keep in mind that the lines you posted are for VBB, but not for BVG.

I can reproduce it with bvg-rest though. Seems like they have blocked all User-Agents that end in bvg-rest/hafas-client, even if a randomly generated ID is in front of it. Will work on a fix.

I tried to find out which credentials are used and found these lines: https://github.com/public-transport/hafas-client/blob/0b81a59c71d07bd31e2ae809b4d089eca11d8f88/p/vbb/index.js#L22 https://github.com/public-transport/hafas-client/blob/0b81a59c71d07bd31e2ae809b4d089eca11d8f88/p/vbb/index.js#L158

This would really surprise me. If they blocked these credentials, they'd kill all the old installed VBB apps out there.

timaschew commented 5 years ago

Keep in mind that the lines you posted are for VBB, but not for BVG.

Ahh, okay so which credentials are then used for VBB?

Indeed, it's working for me if I change one line to this:

const hafas = createHafas(bvgProfile, 'chrome')

It's working fine within Berlin, but if I try to use a station in Brandenburg, actually Bad Saarow (ID: 650030840706) then I get back 502 again. This is my query:

require('vbb-client')({endpoint: "http://localhost:3000"}).journeys('650030840706', '900000024101', {results: 1}).then(console.log).catch(console.error)

// edit

Ops, was using a wrong ID, it was for bus I guess. The ID for the railstation is 900000310042 and works fine :)

derhuerst commented 5 years ago

Keep in mind that the lines you posted are for VBB, but not for BVG. Ahh, okay so which credentials are then used for VBB?

The credentials you linked to (these and these) are indeed for VBB. These are being use for BVG: https://github.com/public-transport/hafas-client/blob/ae31807eb7fbea8e6d75fd64ccbce1f25939817f/p/bvg/index.js#L16-L19

Indeed, it's working for me if I change one line to this:
const hafas = createHafas(bvgProfile, 'chrome')

The createHafas signature with the custom User-Agent is from hafas-client@next (which is hafas-client@3.0.0-alpha.21 currently). Please use this version and provide a meaningful value.

If you use hafas-client@latest (hafas-client@2.10.3 currently) instead, there is no custom User-Agent:

const hafas = createHafas(dbProfile)

It's working fine within Berlin, but if I try to use a station in Brandenburg, actually Bad Saarow (ID: 650030840706) then I get back 502 again. This is my query:
require('vbb-client')({endpoint: "http://localhost:3000"}).journeys('650030840706', '900000024101', {results: 1}).then(console.log).catch(console.error)

You're using vbb-client here, which queries 2.vbb.transport.rest, which is down. Use vbb-hafas/hafas-client directly.

For any additional issues (not related to the 2.vbb.transport.rest API), please create an issue at the respective repo.

timaschew commented 5 years ago

Alright, I'm using hafas-client directly with version 3.0.0-alpha.21

This is my code

const createClient = require('hafas-client')
const profile = require('hafas-client/p/vbb')

const client = createClient(profile, '')

client.journeys('900000310042', '900000024101', {results: 1})
.then(result => {
    console.log(result[0].legs)
})
.catch(console.error)

It works with railway station 900000310042 but not with the bus station 650030840706 (HCI Service: location missing or invalid), but that's okay for me ;)

Thanks!

derhuerst commented 5 years ago

It works with railway station 900000310042 but not with the bus station 650030840706 (HCI Service: location missing or invalid), but that's okay for me ;)

What's the name of this 650030840706 station? Where did you get the ID from?

timaschew commented 5 years ago

From the stops.txt of VBB

$ cat stops.txt | grep "Bad Saarow" | grep "Klinikum"
900000310109,"","Bad Saarow, HELIOS Klinikum",,"52.285765000000","14.062177000000",1,
650030840701,"","Bad Saarow, HELIOS Klinikum",,"52.285765000000","14.062177000000",0,900000310109
650030840706,"","Bad Saarow, HELIOS Klinikum",,"52.285765000000","14.062177000000",0,900000310109
900000310042,"","Bad Saarow, Klinikum Bhf",,"52.284505000000","14.065870000000",1,
450009310042,"","Bad Saarow, Klinikum Bhf",,"52.284505000000","14.065870000000",0,900000310042

Seems that only 9xx are working, the others are duplicates anyway.

So 900000310042 is the railway station and 900000310109 is the bus station which are both working.

Problem solved ✅

derhuerst commented 5 years ago

From the stops.txt of VBB

$ cat stops.txt | grep "Bad Saarow" | grep "Klinikum"
900000310109,"","Bad Saarow, HELIOS Klinikum",,"52.285765000000","14.062177000000",1,
650030840701,"","Bad Saarow, HELIOS Klinikum",,"52.285765000000","14.062177000000",0,900000310109
650030840706,"","Bad Saarow, HELIOS Klinikum",,"52.285765000000","14.062177000000",0,900000310109
900000310042,"","Bad Saarow, Klinikum Bhf",,"52.284505000000","14.065870000000",1,
450009310042,"","Bad Saarow, Klinikum Bhf",,"52.284505000000","14.065870000000",0,900000310042

The 6* stops are stops of the 9* stations. The API doesn't seem to know about this hierarchy (as you said, it doesn't know about them at all).

derhuerst commented 5 years ago

To re-post from above:

Good news everyone!

I have set up a new API: 1.bvg.transport.rest provides data from the BVG endpoint. It returns almost exactly the same data as 2.vbb.transport.rest, but in the output format of hafas-client@3.0.0-alpha.20, so you will have to make minor adjustments. Check the docs at derhuerst/bvg-rest!

Let's see for how long this API will stay up.

michaelcheers commented 5 years ago

Seems like the vbb endpoint is working now.

derhuerst / vbb-rest

get the public endpoint running again (use BVG endpoint for now) #29