Bertware
commented
6 years ago Hi!
Thanks for getting in touch. The IP address was blocked yesterday, as you were making a huge number of requests which were returning an error 503. At the moment you were blocked, we already received 6000 invalid requests, We expect you to use this status code as a sign to lower the number of requests, as returning that 503 error still uses server resources which we'd like to keep available to other users.
As you can see from this excerpt, you were polling the same vehicle over and over, multiple times per second.These are only the blocked requests, there were still 3 requests per second passing through.
Please update your code to do the following:
- Send no more than 3 requests per second
- Don't request the same page twice within 15 seconds, as we're caching it server-side, you won't get any new data. The request will be fulfilled (as long as your not over 3 req/s), but it won't contain new data.
Please change your user agent so we can contact you if should this happen again in the future. Your user agent is empty now, so we hadn't any lead on who was causing these requests.
If you need to retrieve information on stations:
- Please spread your requests over 3 or 5 minutes. Don't make requests between 2 and 4 am when there are no trains.
- Please do lookups based on id! Converting the station name to an ID requires extra processing time on our side. Please convert the names to ids before making your requests. There is a library available here https://github.com/iRail/stations for looking up stations.
If you need to retrieve information on trains, I suggest you do the following strategy:
- Between 3 and 4 am, retrieve every vehicle. Spread your requests over this period of time, this might be possible with even less than 3 requests per seconds.
- Use this information to know when a train will drive, in order to create some sort of index.
- Only retrieve a train starting a half hour before departing, until half an hour after arriving. Poll once a minute while the train is driving, and no more than once every 30 seconds while the train is driving.
Your IP address has been unblocked for a second at 10:38 today, and your server started a neverending flow of requests responses, so I blocked it again. Please fix your code, as this behaviour will always result in a block.
In comparison, a typical client throws less than hundred 503 requests on a day. You just caused 4500 blocked requests in the first one minute. You can resolve this by implementing throttling on your side, or aborting requests when you receive a 503. Storing failed requests in a queue to then run when the server comes back online needs throttling or you'll run into this issue.
Feel free to ask for more specific advice if you need help optimizing your queries. Please comment in this issue when you have resolved this behaviour to get the IP unblocked.
See also https://docs.irail.be/#header-best-practices-when-using-the-api
We know the rate limiting might be annoying, but we're trying to keep the data available for everyone. We're working on a new system which will allow you to get all this data ever few seconds with simple requests, instead of having to search every train. However, we're still ironing the kinks out and it might take a few months before it's stable and 100% correct. Follow us on twitter, our blog, or gitter to stay updated and get notified when the new system is available.
You can read more here: https://hello.irail.be/2018/03/16/one-million-daily-requests-where-do-they-come-from-and-how-well-cope-with-them/
PolMrt
Hi, I have an app which collects data from your api to create analytics from delay. Since yesterday at 2 pm, it doesn't work anymore. My app calls all the departure from each station every 30min. Have I been blocked?