search

derhuerst / vbb-rest

An HTTP API for Berlin & Brandenburg public transport.
https://v6.vbb.transport.rest/
ISC License
127 stars 12 forks source link

get the public endpoint running again (use BVG endpoint for now) #29

Closed berlincount closed 4 years ago

berlincount commented 6 years ago 
$ curl -v 'https://2.vbb.transport.rest/stations/9007102/departures?when=1529416634&duration=30'
* Connected to 2.vbb.transport.rest (54.37.75.136) port 443 (#0)
> GET /stations/9007102/departures?when=1529416634&duration=30 HTTP/1.1
> Host: 2.vbb.transport.rest
> User-Agent: curl/7.47.0
> Accept: */*
>
< HTTP/1.1 504 Gateway Time-out
< Server: nginx/1.10.3 (Ubuntu)
< Date: Tue, 19 Jun 2018 13:53:00 GMT
< Content-Type: text/html
< Content-Length: 192
< Connection: keep-alive
<
<html>
<head><title>504 Gateway Time-out</title></head>
<body bgcolor="white">
<center><h1>504 Gateway Time-out</h1></center>
<hr><center>nginx/1.10.3 (Ubuntu)</center>
</body>
</html>

thanks for having a look!

derhuerst commented 5 years ago

Seems like the vbb endpoint is working now.

Can't reproduce. 😕

curl 'https://2.vbb.transport.rest/stations/900000013102/departures'
michaelcheers commented 5 years ago

@derhuerst, try https://2.vbb.transport.rest/stations?query=oranienburg

derhuerst commented 5 years ago

Hm, it seems like half of the features of the API are working again.

deg0nz commented 5 years ago

I just discovered dfb6c4aed45b328ac5df176db27cc6859946a004. Does this mean, we will have a new working endpoint for VBB data? Are the rate-problems with VBB solved?

derhuerst commented 5 years ago

As mentioned above, I introduce breaking changes in hafas-client or any of its wrapper libraries (such as vbb-hafas or db-hafas) every now an then. When doing this, I increase the major version number by 1.

I've recently released hafas-client@4 and – building on top of that – vbb-hafas@6, db-hafas@4, bvg-hafas@2, etc. For the wrapping REST APIs, I've created a branch each for the old version (2 in vbb-rest, 2 for db-rest and 1 for bvg-rest) and adapted master to the aforementioned latest *-hafas versions.

Long story short: 3.vbb.transport.rest/3.db.transport.rest/2.bvg.transport.rest represent the latest data format based on hafas-client@4. I haven't announced these APIs publicly, so expect sudden changes.

derhuerst commented 5 years ago

Does this mean, we will have a new working endpoint for VBB data?

With the new 3.vbb.transport.rest API the fundamental problem is not solved: The VBB endpoint may still block the servers' IP address(es) if there are too many requests.

We can reduce the chance by load-balancing over multiple servers though, so please deploy the latest REST APIs on your servers or cloud services (such as DigitalOcean, Scaleway, AWS, Now or Heroku):

  1. Check out vbb-rest#3/db-rest#3/bvg-rest#2.
  2. Deploy it and check its health with the /health HTTP route (e.g. curl 'https://my-vbb-rest-3-instance.example.org/health).
  3. Post its URL here.
deg0nz commented 5 years ago

Alright, thank you for clarifying.

I guess I was a little confused by the 3 in 3.vbb.transport.rest I thought that you maybe had a talk with the VBB people... But yeah, your explanation makes sense :)

jonathan-reisdorf commented 5 years ago 
3. Post its URL here.

Hi, I can offer these for the lb:

https://db.fishlevel.com
https://vbb.fishlevel.com
https://bvg.fishlevel.com

The instances run on a bare-metal server located in a data center in Berlin. I could also add the version prefix in the domain if you prefer. Do you know how much traffic is generated on average per month?

derhuerst commented 5 years ago

@jonathan-reisdorf

Thank you! 💚

I could also add the version prefix in the domain if you prefer.

Yes please, it helps me not to misconfigure something by accident. You could adopt the naming scheme of @juliuste and me (Julius will run b.*, I run a.*):

Do you know how much traffic is generated on average per month?

All 6 endpoints together get receive ~50-500k requests per day. On my (really cheap) VPS, they usually contribute to a load of about 0.15.

jonathan-reisdorf commented 5 years ago

@derhuerst ok, I have set these up and also c.3.db.fishlevel.com. The load is no problem, just the traffic might be, bc the data center where my server is housed has a fair-use policy regarding traffic (-> if it's "a couple of hundred GB" per month I might have to pay an additional fee), so let's see how much traffic is generated after a month :)

derhuerst commented 5 years ago

We can also define your nodes as fallback, so that they won't receive any requests if a.* or b.* are up.

jonathan-reisdorf commented 5 years ago

:+1: sounds great!

derhuerst commented 5 years ago

@joecorcoran @deg0nz @ferby09 @k-nut @berlincount @timaschew Do you happen to have the knowledge and time to set up mirror/load balancing nodes as well? I would appreciate it.

k-nut commented 5 years ago

Sorry, I don't have capacities for this right now...

derhuerst commented 4 years ago

I've set up 3.vbb.transport.rest a while ago (forgot to let everyone know about this), and so far it has been running reasonably well, so please use this endpoint or 2.bvg.transport.rest. Unfortunately both have a slightly updated output format, so make sure to check their docs.

I'm going to close this issue now, as the VBB admins haven't blocked 3.vbb.transport.rest so far. Thanks for you patience!

derhuerst commented 4 years ago

I have opened a thread in the transport.rest meta-repo to discuss hosting further: https://github.com/public-transport/transport.rest/issues/2

derhuerst commented 4 years ago

I have set up v5.vbb.transport.rest, the successor of 3.vbb.transport.rest. As usual, unfortunately the response format has changed slightly (to the format of hafas-client@5, make sure to check its migration guide).

I will keep 3.vbb.transport.rest running for a while, and announce its shutdown via RSS before.