Random user agent for each API call

[predictablyrandom]

Is your feature request related to a problem? Please describe. Currently the API calls returns 403 a lot of times.

Describe the solution you'd like Perhaps using a random user agent like in this python package https://pypi.org/project/fake-useragent/ should fix it?.

[SRvSaha]

@predictablyrandom Yeah, need to do that equivalent in .NET

[predictablyrandom]

Perhaps just have a list of the agents & choose one of the elements from the list at random? and use that one in the API call? @SRvSaha https://github.com/rummolprod999/RandomUserAgent

[SRvSaha]

@predictablyrandom Yes, can be done. I think Chrome, Mozilla would be fine. What say?

[predictablyrandom]

https://github.com/rummolprod999/RandomUserAgent/blob/master/RandomUserAgent/RandomUA.cs this file has the list & the relevant code snippet @SRvSaha. More the merrier I think, possibility of same agent being used for subsequent calls become very less.

[SRvSaha]

https://github.com/rummolprod999/RandomUserAgent/blob/master/RandomUserAgent/RandomUA.cs this file has the list & the relevant code snippet @SRvSaha. More the merrier I think, possibility of same agent being used for subsequent calls become very less.

@predictablyrandom Understand that more will be better. Just was thinking for restricting to some small set, as we don't know which all are supported by CoWIN. Can you help in writing automatic testing of one endpoint using all possible UAs and see if all are supported by CoWIN?

[predictablyrandom]

@SRvSaha I have tried the python package which I had shared before that works without any issue. Let me check

[SRvSaha]

@SRvSaha I have tried the python package which I had shared before that works without any issue. Let me check

Yeah, please just check the whether the UAs are same in both

[predictablyrandom]

checked entries for Firefox, Chrome, Opera & Safari from https://github.com/rummolprod999/RandomUserAgent/blob/master/RandomUserAgent/RandomUA.cs they seem to work fine

[SRvSaha]

Awesome! Shall add them then! @predictablyrandom

[predictablyrandom]

@SRvSaha In case its not already done, perhaps pass Cache-Control: no-cache, no-store, max-age=0, must-revalidate in headers too?

[SRvSaha]

@predictablyrandom That’s a good suggestion, shall do that

[SRvSaha]

@predictablyrandom Observed today that after some time, IP was getting throttled. Any ideas if they have added IP Throttling? Any workaround?

[Rajat-dey]

Is that about IP got blocked?? @SRvSaha

[SRvSaha]

@Rajat-dey Yeah, seem like they had added some throttling.

Check this #53

[Tnu02]

Have you found any solution @SRvSaha for this IP throttling ??

[SRvSaha]

@Tnu02 The Rate Limit as I feel by observation is 100 Hits from IP within 5 minutes. So, if you are searching for 1 PIN Code then you'll have to keep the SleepInterval as 3000, similarly if you have 2 PIN Codes, it will be 6000 and so on and so forth so that IP is within the rate limit. Another way out is to use two different network IPs and then keep switching between the once you see the error. You can copy the Bearer Token you get after entering OTP, and then on subsequent run from different IP. you have use that in the config file so that re-OTP is not required. This the only work around as of now. Also, try varying the User Agent, might help.

[Tnu02]

@Tnu02 The Rate Limit as I feel by observation is 100 Hits from IP within 5 minutes. So, if you are searching for 1 PIN Code then you'll have to keep the SleepInterval as 3000, similarly if you have 2 PIN Codes, it will be 6000 and so on and so forth so that IP is within the rate limit. Another way out is to use two different network IPs and then keep switching between the once you see the error. You can copy the Bearer Token you get after entering OTP, and then on subsequent run from different IP. you have use that in the config file so that re-OTP is not required. This the only work around as of now. Also, try varying the User Agent, might help.

I am using 10000 for 2 pin code still after few time start getting the game error.. 😀

[xxxAng]

@SRvSaha what about if we search 1 district instead of pincodes ? Tried with 1 district 7 seconds, still getting rate limited :(

[SRvSaha]

@Tnu02 If possible, use Hotspot from different different ISPs for every 100 Hits/5 minutes.

[SRvSaha]

@SRvSaha what about if we search 1 district instead of pincodes ? Tried with 1 district 7 seconds, still getting rate limited :(

Yes this is a bit strange, CoWIN is not handling IP Throttling properly, similar issue is being faced by others as well: Check here: https://github.com/bombardier-gif/covid-vaccine-booking/issues/270

[predictablyrandom]

@predictablyrandom Observed today that after some time, IP was getting throttled. Any ideas if they have added IP Throttling? Any workaround?

@SRvSaha yes seems like they have done some sort of IP throttling, the only way I am aware of is to route requests randomly through different proxies (https://www.scrapehero.com/how-to-rotate-proxies-and-ip-addresses-using-python-3/, https://towardsdatascience.com/5-strategies-to-write-unblock-able-web-scrapers-in-python-5e40c147bdaf, https://stackoverflow.com/a/49064965), this could also be tricky because requests could be blocked if they are coming from outside of India, and not all proxies can be trusted.

[SRvSaha]

Hi @predictablyrandom, your suggestions are integrated in v4.2.0 is up now. Check it out and share your feedback. This release tries to address some of the issue of IP Throttling.

[SRvSaha]

@Rajat-dey Now the Throttling Interval and Throttling Time is there as a part of Config File to modify along with the changes in Sleep Interval. Give it a try in v4.2.0 is up now. Check it out and share your feedback. This release tries to address some of the issue of IP Throttling. See if it helps

[SRvSaha]

Have you found any solution @SRvSaha for this IP throttling ??

v4.2.0 is up now. Check it out and share your feedback. This release tries to address some of the issue of IP Throttling.