search

cowinapi / developer.cowin

This group is created to facilitate technical and integration discussions related to cowin platform. API related contents can be obtained at API setu portal https://apisetu.gov.in/public/marketplace/api/cowin
115 stars 30 forks source link

A real-time scalable vaccine notification service #189

Open abizerlokhandwala opened 3 years ago

abizerlokhandwala commented 3 years ago

Hey All!

A few friends and I have built a vaccine notification subscription portal: https://vaccinepost.co.in to send notifications to users whenever vaccination slots are available in your district.

We currently send emails based on districts and your preferences, and we're working on a few features (centers within your radius, Whatsapp/push notifications etc) too.

The whole stack is built on AWS, so we are able to scale out easily with increase in new users and districts. We are currently polling 194 unique districts and can support a lot more.

The code is all open-source, if you would like to contribute, you can go through the repositories below: Backend: https://github.com/abizerlokhandwala/Cowin-Notification-Service Frontend: https://github.com/Guzzler/Cowin-Notification-System

swingcake commented 3 years ago

This is like an open-ended hackathon, but our lives depend on it. Great, more competition to beat!

pranab-witbybit commented 3 years ago

Hey how are you able to poll so many districts, as cowin has an api limit of 100 requests per 5 mins?

abizerlokhandwala commented 3 years ago

Hey how are you able to poll so many districts, as cowin has an api limit of 100 requests per 5 mins?

Hey! As we are using AWS cloud services, each polling operation is taking place on an AWS Lambda. A Lambda is a serverless function (similar to a container), that AWS hosts on their servers. These Lambdas are hosted on a large number of different servers that AWS has, which helps us as they have all have different IPs. This way we are able to circumvent the rate limit issues as all are requests go through different IPs.

pranab-witbybit commented 3 years ago

Got it, Thanks !

abhijeetsingh26 commented 3 years ago

What I believe is instead of the polling mechanism the cowin devs need to implement push type notifications. Their public API is taking un-necessary load due to people findings ways to circumvent the rate limits basically implementing the same thing again and again.

We should minimize the clients which are doing the same thing that under45.in does. Just my 0.02$ on this.

FWIW there is a improvement listed https://github.com/cowinapi/developer.cowin/issues/174.

rudrasura commented 3 years ago

Even if you are using AWS Lambda, what is the time gap between initiating an email and actually a user recieving it? I have seen cases where slots when opened in Mumbai/Bengaluru/Thane/Pune gets filled within 1-2 minutes. What I have observed Integration with any available lightweight messaging service like Twilio Whatsapp or Telegram is faster as compared to notification based on email.

abizerlokhandwala commented 3 years ago

Hey @rudrasura ! From our experiments, our emails were faster than messages we received on a few Telegram groups. We use AWS SES to send emails to our users, and we send emails immediately instead of queueing them to be processed asynchronously.

rudrasura commented 3 years ago

Hey @rudrasura ! From our experiments, our emails were faster than messages we received on a few Telegram groups. We use AWS SES to send emails to our users, and we send emails immediately instead of queueing them to be processed asynchronously.

I experimented with Spring Boot based Google email integration and found that the notification were actually received with a gap of 4-5 minutes even with Async based approach. Probably AWS SES is better than that. AWS SES also has SMS based notification. Were you guys able to try that? I guess it has cost associated and not sure whether messaging in India via AWS SES is allowed.

abizerlokhandwala commented 3 years ago

Hey @rudrasura ! From our experiments, our emails were faster than messages we received on a few Telegram groups. We use AWS SES to send emails to our users, and we send emails immediately instead of queueing them to be processed asynchronously.

I experimented with Spring Boot based Google email integration and found that the notification were actually received with a gap of 4-5 minutes even with Async based approach. Probably AWS SES is better than that. AWS SES also has SMS based notification. Were you guys able to try that? I guess it has cost associated and not sure whether messaging in India via AWS SES is allowed.

I have not worked with Google's email services, not really sure about that. For SMS, I believe you're talking about AWS SNS. We did look into it, but the costs turned out to be quite high for our usecase and userbase.

sriharry16 commented 3 years ago

@abizerlokhandwala How are you able to make sure that data returned from Public API's are real time one? They say that the data might be 30 mins old as they send back the cached data back to the users.

shoumikg commented 3 years ago

@abizerlokhandwala hey man thanks a ton for this service! I was able to book a slot, thanks to the email alert which came about 15 seconds before the telegram U45 alerts for my location. I also had to stay logged in all day (re-login with new OTP every 12-15 minutes) on Aarogya Setu app. I used the app instead of CoWIN website because it requires no captcha.

Also, been going through the issues in this repo and all the comments, and it's clear that the captcha only covers the CoWIN website. So it's probably possible to book slots entirely using bots. I'm not sure how much the @cowinapi team knows about this!

Once again, thanks a ton for building a sweetly optimised notification service. Kudos!

abizerlokhandwala commented 3 years ago

@abizerlokhandwala How are you able to make sure that data returned from Public API's are real time one? They say that the data might be 30 mins old as they send back the cached data back to the users.

I've noticed that the calendarByDistrict API does cache data, but the findByDistrict API either has a really low cache expiry, or doesn't cache at all, as most of our notifications are quite consistent with Cowin's data.

abizerlokhandwala commented 3 years ago

@abizerlokhandwala hey man thanks a ton for this service! I was able to book a slot, thanks to the email alert which came about 15 seconds before the telegram U45 alerts for my location. I also had to stay logged in all day (re-login with new OTP every 12-15 minutes) on Aarogya Setu app. I used the app instead of CoWIN website because it requires no captcha.

Also, been going through the issues in this repo and all the comments, and it's clear that the captcha only covers the CoWIN website. So it's probably possible to book slots entirely using bots. I'm not sure how much the @cowinapi team knows about this!

Once again, thanks a ton for building a sweetly optimised notification service. Kudos!

Really glad we were able to help you out @shoumikg :) I did not know that the mobile application does not require captcha, this is a good find. We have more features coming up that would optimize notifications making them more personalized, do head over to the repository and contribute if you're interested! :)

lyveng commented 3 years ago

@abizerlokhandwala How are you able to make sure that data returned from Public API's are real time one? They say that the data might be 30 mins old as they send back the cached data back to the users.

I've noticed that the calendarByDistrict API does cache data, but the findByDistrict API either has a really low cache expiry, or doesn't cache at all, as most of our notifications are quite consistent with Cowin's data.

@abizerlokhandwala First of all great work. Since the calendarByDistrict API is indeed cached, don't you think it is a problem for users since the cowin website seems to be making call to this api as per the network tab. Wouldn't the users see cached data even when they are logged in making it more difficult to schedule appointment? There have been some discussions where people suggested that cache could be disabled when the Authorization header is passed. But from my experiments, even the authorized calls seem to return cached data for this api.

I'd definitely try out your suggestion though. Thanks again.

abizerlokhandwala commented 3 years ago

Hey @lyveng! The cowin website uses the protected APIs when making calls when a user is logged in (https://apisetu.gov.in/public/marketplace/api/cowin/cowin-protected-v2). I highly doubt the data is cached with the protected APIs. Also, they have no rate limits. Have you observed cached data with authorized calls too? Try using the findByDistrict API if that is the case.

lyveng commented 3 years ago

@abizerlokhandwala Yes. From what I've seen and logged, the cowin protected apis seem to be returning stale data when compared to telegram. I tried using findByDistrict and this seems to be giving fresh data like you suggested. Thanks a lot buddy.

sahilt75 commented 3 years ago

@abizerlokhandwala I wanted to understand how the rate limiting is bypassed using lambda. Once the lambda is invoked, there will be a single IP that your request will go through for a single invocation right? And since you have 194 unique districts, do you hit 194 API calls during this single lambda invocation? I have also created a similar service and hosted the same on AWS. So I am just trying to understand your design!

abizerlokhandwala commented 3 years ago

Hey @sahilt75 ! As each lambda has its own IP, we spawn concurrent lambdas so that each lambda handles only 1 district (Lambda + SQS Triggers). We are currently polling every district every 10 seconds, so in 5 min we were making 30 API calls for each district.

Though, we started to use the findByDistrict API and had to query it multiple times (for multiple dates), which lead to around 120 API calls for each district. Which meant forcing SQS + Lambda to spawn more concurrent lambdas, which became more expensive.

Later, we found a way to force Lambdas to respawn (by updating any Lambda configuration). This lead to us having cold starts on most of our calls, though that wasn't an issue for us. If you want to know more about our architecture, you can ping me personally, I'd be happy to talk about it! Also, if you're interested in such solutions, would be glad to have you contribute to our project :D

sahilt75 commented 3 years ago

@abizerlokhandwala Ohh now I get it! Sure would love to jump in and work with you guys. Will definitely ping you when I will be able to put in some hours for this. Caught up with another project rn :)

CodeWithBishal commented 3 years ago

Did anyone get protected API access? I have built a PHP website (like https://www.cowin.gov.in/home) that will pull the data from "calendarByPin" Public API (when someone searches for their Pincode) but the problem is the limit of 100 calls per 5 min and in my project, it will be exhausted in a minute or less

RotonEvan commented 3 years ago

Hello everyone, I know I'm late to the party but I got a question. I saw that one difference in the request headers of public and protected APIs of CoWin is the cache-control. It is max-age=0 in public API and no-cache in protected API. If we somehow use no-cache for public API as well then will we get real-time data?