alfonsrv / impf-botpy

Impf Bot.py 🐍⚡ – Automatisierung für den Corona ImpfterminService Bot
242 stars 46 forks source link

Comprehensive Guide to Getting Appointments for Dummies #57

Closed alfonsrv closed 3 years ago

alfonsrv commented 3 years ago

Getting an Appointment 101

The experience collected is only for BaWü, but should apply anywhere else too.

GitHub stars

Overview

Getting appointments is just a matter of the amount of Centers you're checking. As of writing this on June 8th, I've been getting 1-3 appointment offers a day, though I have gone without getting any appointments for 2-3 days too – it's just part of the process.

If you're only checking one center over and over again, your chances of getting an appointment are significantly lower. Center capacities vary from their size and overall load – e.g. getting an appointment at Robert Bosch Krankenhaus is like catching an extremely rare Pokémon; I've only seen it give out appointments twice over the period of 4 weeks.

When new appointments are added by a center, you're playing Roulette against everyone else currently checking the center for appointments,– so it's just a matter of improving your chances.

Improving Your Chances

aka Q: I have been searching for 2-85430 days 24/7, but didn't find anything yet! oh my god pls does this even work

If you're only searching one center your chances of getting an appointment are obviously lower. It will definitely take some time due to the reasons mentioned initially.

Improving your chances means turning on parallel searching. Setting CONCURRENT_ENABLED = True enables you to search multiple centers at the same time, thus increasing your chances immensely. Ideally, search with six instances / for six centers concurrently. But why six?

As mentioned in the README.md centers are assigned to specific servers, indicated by the number at the start of the URL. It's best to spread out and choose ONE center on each server to search on. Checking more on the same server doesn't yield any benefit afaik

Just choose 4-6 centers on different servers, get Vermittlungscodes (see below), enter them in your settings.py and start searching. Depending on how many centers you have entered you should set your CONCURRENT_WORKERS to the same values as the centers you added – you might have to decrease it depending on how potent your PC is. And then just be patient; appointments will come.

If your PC cannot handle multiple browsers at the same time at least do not jump around centers while searching. It's usually desirable to keep on checking the same center over and over again.

This is the way I run my setup – it's just an old laptop that I turn on in the morning, and let run throughout the day – imo the smallest Vaccination Center in Stuttgart.

Getting Vermittlungscodes

Getting Vermittlungscodes via the Browser should be avoided at all costs.

aka Q: I'm not getting any Vermittlungscodes! The instant Code Generation is broken! I tried so many times, never works

As long as the Instant Vermittlungscode bullet point is not crossed out on the main page (README.md), it still works. Simply try again in an hour, two, ten, or tomorrow. Just. Keep. On. Trying.

It's unclear whether or not your cookies need a certain age before the server will accept your request; just something to keep in mind. That's why the --surf switch was added, but should also be a-ok to just let the bot run for 2-4 hours before.

And while you're at it, as soon as you can generate one, generate a bunch! One for each of the six centers. One doesn't work? Doesn't matter. Just move on to the next one and try again later.

Disclaimer

Please note: All of this is based on my own experiences – your mileage may vary. A lot of things are unknown about the platform and simply assumptions based on trial-and-error and my previous technical experience. If anything is wrong, please feel free to discuss it, but understand that I cannot get back to each and every one individually. Especially non-elaborated questions, clearly showing a lack of having read this thoroughly or lack of self-initiative, will be ignored.

EricDuminil commented 3 years ago

Excellent tutorial, thanks.

When setting CONCURRENT_ENABLED = True and CONCURRENT_WORKERS: int = 5, I have 5 open Chrome windows on my laptop, which is expected.

Those windows are regularly closed, though, and when they reopen, they pop-up in front of my current application, even if I started the script in another workspace.

I tried to add self.driver.minimize_window() inside __post_init__(self):. The Chrome windows get minimized, but they still pop-up for a few milliseconds, and sometimes manage to steal a few keystrokes from what I'm currently typing. :-/

Is there any way to not close the browser windows regularly when CONCURRENT_ENABLED is set to True? I wouldn't like to start the browser in headless mode, if I ever need to manually enter the appointment info.

alfonsrv commented 3 years ago

@EricDuminil – browser windows should not close and reopen if you have a Vermittlungscode for a center and RESCAN_APPOINTMENT set to True.

The only other time windows open and close is when the bot is searching for a Vermittlungscode and CONCURRENT_ENABLED = True, which I consider to be a corner case, due to Vermittlungscodes being a commodity.

Another way could be to start multiple instances of Impf Bot.py manually having CONCURRENT_ENABLED = False. There shouldn't be any noticeable performance penalty doing so, as that's what the bot is basically doing if concurrency is enabled.

Lastly if none of this is satisfying or there should be any use-case I'm missing – I'm sure you've seen this, but still gonna mention it again for completeness. Somebody pointed out a way of using headless browser with a small modification and accessing the headless instance via the browser later.

EricDuminil commented 3 years ago

@alfonsrv : Thanks, RESCAN_APPOINTMENT was indeed set to False, I don't remember why. The browsers still randomly close and reopen, but much less often. It if bothers me too much, I'll try the headless + port alternative.

EricDuminil commented 3 years ago

Your guide works fine. I got 2 appointments at Robert Bosch Krankenhaus today. Gotta Catch 'Em All!

klaus-vb commented 3 years ago

Works fine if one reads the instructions in the settings file carefully. This guide is even better. Just noticed it after I got my appointment today.

EricDuminil commented 3 years ago

Yesterday, I got 3 appointments, but only in theory. I tried to send "appt:1", but it failed with error 429. I try again 30s later, from the web-interface, but it tells me the appointment is gone, even though they are supposedly booked for 10 minutes. Do you have any suggestion for how to book the appointment once it's available? Not too fast, so it doesn't return 429, not too slow, so it doesn't get taken by someone else?

alfonsrv commented 3 years ago

Remote booking is no guarantee. A number of things might go wrong; e.g. there's always a slight chance you're thrown into the Waiting Room when "reloading" the page. Same is for 429 – when the server is experiencing too many requests in general, it will just lock up and block requests randomly. That's why I added a booking fallback option via the GUI, in case the API should fail, but I assume you patched that out?

Unfortunately when appointments are offered to the user, it seems like they're not only offered exclusively but to multiple people at the same time. The page's 10 minute reservation is simply misleading. Nothing we can really do about it – simply the way it is.

EricDuminil commented 3 years ago

@alfonsrv Thanks for the quick answer. By "booking fallback option via the GUI", do you mean the new browser window which is open with a possibility to enter data manually? Yes, I tried it, but the appointments were already gone. As you said, probably something that we don't have any influence over.

alfonsrv commented 3 years ago

https://github.com/alfonsrv/impf-botpy/blob/7178239bf2bef081ec8a4f53e18027f0e45090fa/impf/browser.py#L318

Ionic commented 3 years ago

I've got a few questions that weren't answered (directly) in any of the provided documentation:

1.) Do codes have a TTL? I.e., are they automatically invalidated after a specific time period? 2.) Can codes be invalided by ITS (bot detection/prevention seems to exist to some degree?) and if yes, is there any way to know whether a specific code has been invalidated?

Then, there's this whole shadow ban business. I'm aware that no one (other than ITS developers, of course) probably knows all details about it, but it doesn't hurt to ask.

HTTP status code 429 is treated as a shadow ban by the bot, which generally makes sense, but I'm skeptical that it always is a shadow ban. I started using the bot last Friday in normal mode and figured that it would fetch codes automatically, with a list containing four centers on three different servers, all in BW, with most other settings unchanged from the defaults (other than personal information). The bot cycled through all centers and most often returned "No vacancy right now" until Saturday, when it began seeing 429 status codes. At that point, I decided to use the instant code generation feature and first fetch the codes manually.

Interestingly, fetching/generating a code always failed with a 429 status code the first time around, which made the shadow ban evasion kick in. After multiple rounds of waiting totaling 58 minutes waiting time, it was always able to fetch a code from ITS. That is, the same pattern emerged for all three servers: 429 at first, wait, code generated.

Due to this pattern, I wonder if ITS is actively enforcing waiting times by always rejecting the first attempt.

Next, I ran the bot normally, with concurrency disabled, so it only checked one specific location.

It wasn't able to find any available slot... up until Sunday at about 12 pm, which contradicts your notion that slots are not made available on Sunday. Sadly, the notification didn't work for me (not due to software, but hardware issues - my speakers were turned off, so I just missed it). Stopped the process at about 4 pm.

Later on Monday at about 9 pm, I turned on concurrent mode and had the bot check all three servers concurrently, thinking that it would speed up the process (and that seems to generally be advised in this issue report). I was immediately greeted with 429 status codes. Stopped it, restarted at about 12 pm, but the situation didn't change.

Even more interestingly, I fully stopped the bot at about 4 pm again and restarted it at 10 am today, only to get another round of 429 status codes.

That's purely anecdotal and empirically determined, but I wonder if 429 status codes are also triggered by backend issues (e.g., general throttling due to load issues on their side) that have nothing to do with direct client shadow bans.

On a related note, I wonder if using the concurrent mode actually makes sense. Especially if the ITS backend is already overloaded, it'll only make the situation worse and it potentially also makes client-based shadow bans more likely. Should this really be recommended?

alfonsrv commented 3 years ago

Hey @Ionic, yes they do have a TTL and it's unknown whether or not Vermittlungscodes are invalidated (maybe even actually Shadow Banned), if they are associated with being a bot – but it doesn't seem to be the case.

429s probably have nothing to do with actually being Shadow Banned, but with the server outright simply rejecting requests, if the overall demand is too high. It's also unknown whether or not cookies need to have a certain age before the server accepts your requests in the first place. The new code generation in the latest version works quite reliably though.

Sunday appointments can appear, but not by centers adding appointments, but by someone simply cancelling one. So new slots do in-fact appear, but the chance is too slim imo to bother.

Simply wait out the 429s. I'm not sure why people seem to pay an excessive amount of attention to them – just wait it out and let it run. As soon as you have a Vermittlungscode and are searching appointments 429 throttling doesn't apply anymore.

Ionic commented 3 years ago

Simply wait out the 429s. I'm not sure why people seem to pay an excessive amount of attention to them – just wait it out and let it run. As soon as you have a Vermittlungscode and are searching appointments 429 throttling doesn't apply anymore.

That doesn't seem to be the case for me. I've been running the application in concurrent mode with three different servers for 11 hours today and it didn't make it to the appointment list even once - instead, it has been consistently getting 429 errors.

Disabling concurrent mode and only checking one center likewise always returned a 429 error at first, but fetching the appointment list worked after the first waiting round of 14 minutes.

Due to this pattern, my personal suspicion is that ITS implemented some sort of temporary banning in a greylisting fashion. That is, they always seem to reject the first attempt with a 429 error, remember the client and only allow requests after a specific amount of time (5 minutes? 10 minutes?) has passed without additional requests. If the client sends another requesting during that period, the banning time is probably extended.

That would explain what I'm seeing, since in concurrent mode, the application sends three requests within 90 seconds (each to a different server, but ITS still seems to somewhat synchronize the banning/greylisting state).

I'll try raising WAIT_CONCURRENT from the default value of 30 seconds to something like 20 or 30 minutes (or potentially even higher) instead. If that works for me, i.e., if that lets me get to the appointment lists and scan multiple centers concurrently, I'll handle that as a confirmation of my suspicion.

alfonsrv commented 3 years ago

Just tested – not getting a single 429 – getting Vermittlungscodes and entering booking pages concurrently worked first try. Likely a Layer 8 issue.