search

CSSEGISandData / COVID-19

Novel Coronavirus (COVID-19) Cases, provided by JHU CSSE
https://systems.jhu.edu/research/public-health/ncov/
29.16k stars 18.47k forks source link

Created a JSON-based API. #8

Closed ExpDev07 closed 4 years ago

ExpDev07 commented 4 years ago

Hello and thank you so much for maintaining this! I have created an API that reads your data and returns it in a way that's more friendly to use in programs. It also supports history. It's still a W.I.P and I would love contributions! It is open-sourced here: https://github.com/ExpDev07/coronavirus-tracker-api. Feel free to use it in your projects!

It is very fast due to caching.

The current endpoints are (more will be added): https://coronavirus-tracker-api.herokuapp.com/confirmed https://coronavirus-tracker-api.herokuapp.com/deaths https://coronavirus-tracker-api.herokuapp.com/recovered

For all of them combined: https://coronavirus-tracker-api.herokuapp.com/all

brent-field commented 4 years ago

This is super helpful and works swimmingly. Thanks!

altezza04 commented 4 years ago

Thank you! I start to use it on my site.

ExpDev07 commented 4 years ago

Glad to hear you're using it! @altezza04 . Feel free to share your site :)! I'd love to see it.

bjarkimg commented 4 years ago

WIP - Power BI http://bit.ly/38qeMbz

bjarkimg commented 4 years ago

@ExpDev07 possible to return the date fields unpivoted ? Can do this in power bi just a bit of hassle.

grate job btw :)

3verse commented 4 years ago

WIP - Power BI http://bit.ly/38qeMbz hi @bjarkimg - nice viz! I created my own on Tableau public (https://tabsoft.co/2OKTU72) but this was before the dataset was moved to github, I'll have work on the datasource again to update this now

bjarkimg commented 4 years ago

@3verse nice viz :), yea did the google sheet first.. hope this will be stable.

altezza04 commented 4 years ago

Glad to hear you're using it! @altezza04 . Feel free to share your site :)! I'd love to see it.

http://chinavirus.live/ I use it on the world ranking part only, it seem the data is not updated often , but anyway, thanks for the effort!

bjarkimg commented 4 years ago

FYI added Deaths and Recovered, with Measure select option. http://bit.ly/38qeMbz

bjarkimg commented 4 years ago

@ExpDev07 possible to return the date fields unpivoted ? Can do this in power bi just a bit of hassle.

grate job btw :)

@ExpDev07 Ignore this request, solved in Power BI, auto update should work now. thanks for the API!

ralyodio commented 4 years ago

Looks to be broken now.

ExpDev07 commented 4 years ago

@chovy it broke when they re-formatted the repo, I've fixed it to point to the correct paths now and it's working again.

ralyodio commented 4 years ago

awesome. thank you sir.

bjarkimg commented 4 years ago

FYI Added Forecast on Page 3 http://bit.ly/38qeMbz

image

wobsoriano commented 4 years ago

I made a website too with latest tweets, timeline and charts regarding the virus https://the2019ncov.com

Screen Shot 2020-02-18 at 7 51 40 AM

leopoul commented 4 years ago

Screenshot 2020-02-24 at 21 25 59

I parse your API endpoint, massage it and throw it in an influxDB. At the same time I build a unique list of locations to feed the Grafana WorldMap plugin served via Flask. Here is a first draft of the Grafana dashboard. I plan to work a bit more around my code and share dashboard, parser and Flask server code.

iceweasel1 commented 4 years ago

@leopoul Grafana and InfluxDB, I love it. Nice work!

smartcuc commented 4 years ago

Cool, how did you get the data into influxDB?

ralyodio commented 4 years ago

I noticed a map that has quanrtined data. Can we add that?

ExpDev07 commented 4 years ago

@chovy ill see what I can do!:)

leopoul commented 4 years ago

Code is here: https://github.com/leopoul/covid19_grafana . It's a quick and dirty release so feel free to contribute. I transform the data to get it into Influx and extract the unique locations to make it work with the Grafana plugin. I wanted to have something quickly out so there is lots of room for improvement. grafana_1 grafana_2

elmortero commented 4 years ago

Great work! Is there a possibility to filter by country in the request?

ExpDev07 commented 4 years ago

Please head over to https://github.com/ExpDev07/coronavirus-tracker-api for any questions/discussions regarding the API. :)!

Laeyoung commented 4 years ago

Oh nice work!

I did similar thing : )! https://github.com/Laeyoung/COVID-19-API

mauro-gobbi commented 4 years ago

Hello, this API is never up to date and not in sync with the daily reports

ExpDev07 commented 4 years ago

Hello @mauro-gobbi. It caches the daily reports for 1 hourr. So at most, it takes 1 hour for it to show the changes after an update. This is to allow for faster results and avoid having to make calls to GitHub per request. If you fork the API, you can change the cache duration to 30 minutes, or completely remove it if you wish. It's as simply as just changing a variable. Feel also free to open an issue or pull request if you believe it should be lowered. I haven't had any complaints besides yours, so most seem OK with the delay.

mauro-gobbi commented 4 years ago

Hello @mauro-gobbi. It caches the daily reports for 6 hours. So at most, it takes 6 hours for it to show the changes after an update. This is to allow for faster results and avoid having to make calls to GitHub per request. If you fork the API, you can change the cache duration to an hour, 30 minutes, or completely remove it if you wish. It's as simply as just changing a variable. Feel also free to open an issue or pull request if you believe it should be lowered. I haven't had any complaints besides yours, so most seem OK with the delay. EDIT: It’s cached for 1 hour.

Hi, thanks a lot for your reply and thanks for making this API available. 😊 Sorry but I do not understand, even if you refresh the data every hour, and regardless caching results, isn't the data source (the .csv in "csse_covid_19_time_series") updated only once a day from J. Hopkins? It's probably my fault but I do not actually understand how to get fresh data more frequently than one time per day.

ExpDev07 commented 4 years ago

isn't the data source (the .csv in "csse_covid_19_time_series") updated only once a day from J. Hopkins

Yes, you are correct in that. So whenever JSU updates their time series, it takes a maximum of 1 hour and a minimum of 0 seconds for it to reflect in the API (which is why the data may differ). If you're talking about live updates, that's not really a thing. Most sources will give you daily updates.

mauro-gobbi commented 4 years ago

isn't the data source (the .csv in "csse_covid_19_time_series") updated only once a day from J. Hopkins

Yes, you are correct in that. So whenever JSU updates their time series, it takes a maximum of 1 hour and a minimum of 0 seconds for it to reflect in the API (which is why the data may differ). If you're talking about live updates, that's not really a thing. Most sources will give you daily updates.

Ok great, fair enough. Live updates are not possible for sure, I hope they will start to upload a new .csv report at least every 12h. Thanks again 👍

@ExpDevv07 also, it's quite weird, that this branch https://github.com/CSSEGISandData/COVID-19/tree/web-data mentions a crawler to worldometers.info and bnonews.com.

I thought those two websites used to get data from the JHU dashboard, not the opposite (they are fully synchronized indeed).

Omaroid commented 4 years ago

Hello everyone,

I maid an android app calling @ExpDev07 Rest API, check the repo. https://github.com/Omaroid/Covid-19-Android

MohammedAlrozzi commented 4 years ago

@ExpDev07 thanks for the great work. could you please explain on how to get confirmed cases for example for one single country. how the GET url will look like. Thanks a lot.

Jeff-Lewis commented 4 years ago

Bjarki @bjarkimg Can you share your Power BI pbix file please? Or is it possible to make the PBI clonable/save as on app.powerbi.com? I'm wanting to restore County level data and add a Logistics curve chart https://github.com/CSSEGISandData/COVID-19/issues/637

ralyodio commented 4 years ago

api missing data again.

roenw commented 4 years ago

@ExpDev07 Your API works great. I have one hosted locally since I’m developing my own coronavirus dashboard.

Sorry if off topic but typically how often do the numbers presented update? Thank you.

DevakiTrishani commented 4 years ago

@bjarkimg : I'm new to Power BI, and would like to use the data in this Repo. The problem is I don't know how to connect said data in Power BI. Please can you help. Sorry for asking on this thread, I have no other means to ask you. btw, your Dashboards look impressive. Thank you.

ExpDev07 commented 4 years ago

@roenw Updates whenever this repo updates (with a delay of max an hour for cache to reload) :).

roenw commented 4 years ago

Thanks @ExpDev07!

Aurel-PBI commented 4 years ago

Hi, bjarkimg

Could you share your pbix file. I would like to learn how you do this amazing dashboards Thanks again

Aurel-PBI commented 4 years ago

@ExpDev07 possible to return the date fields unpivoted ? Can do this in power bi just a bit of hassle. grate job btw :)

@ExpDev07 Ignore this request, solved in Power BI, auto update should work now. thanks for the API!

Hi bjarkimg How did u unpivoted the date with PowerBi with data Json AND with an auto update. I'm realy interested to know. Thanks again

DevakiTrishani commented 4 years ago

@ExpDev07 Thank You!

itsgorain commented 4 years ago

Are individual US states data no longer coming through in recoveries and deaths?

ExpDev07 commented 4 years ago

@itsgorain JHU has dropped tracking of recoveries completely, for confirmed and deaths, they have decided to do U.S.A as a whole instead of individual states. These changes obviously are reflected in the API.

ajmishra1989 commented 4 years ago

I'm a beginer here and wanted to store how we can store this data in Google cloud storage first (Using the API). Would be helpful if anyone can help me with the exact steps.

Also, any resources for data related to user sentiments like shopping behavior, spending etc.

Much thanks in advance.

ExpDev07 commented 4 years ago

@ajmishra1989 I'm guessing you'd have to write a script to periodically retrieve the data you want and push it to Google Cloud Storage.

ajmishra1989 commented 4 years ago

@ajmishra1989 I'm guessing you'd have to write a script to periodically retrieve the data you want and push it to Google Cloud Storage.

OK, Thanks. Any possible workaround my second question? I'm looking for place to fetch data for user sentiments around Covid-19.

Thanks.

ExpDev07 commented 4 years ago

I believe there’s some scrapers out there that offers a news feed etc, google around “Covid19 news scraper github” @ajmishra1989

wobsoriano commented 4 years ago

Good bye recovered data.

Omaroid commented 4 years ago

Good bye recovered data.

They added the recovered data !

mahdighazali commented 4 years ago

@ExpDev07 possible to return the date fields unpivoted ? Can do this in power bi just a bit of hassle. grate job btw :)

@ExpDev07 Ignore this request, solved in Power BI, auto update should work now. thanks for the API!

Hi bjarkimg How did u unpivoted the date with PowerBi with data Json AND with an auto update. I'm realy interested to know. Thanks again

Hi, Basically: Move the data columns (before expand) to be the last one, Create a variable to get all the column names, ( Cols = Table.ColumnNames(#"Reordered Columns"),)

Create a variable all the columns (by excluding the one who are not part of the unpivot) ColsUnPivot = List.Skip(Cols,9),

Unpivot the columns: Unpivot = Table.Unpivot( #"Reordered Columns",ColsUnPivot,"Attribute","Value"),

Source = Json.Document(Web.Contents("https://coronavirus-tracker-api.herokuapp.com/all")),

"Converted to Table" = Record.ToTable(Source),

#"Expanded Value" = Table.ExpandRecordColumn(#"Converted to Table", "Value", {"last_updated", "latest", "locations", "source"}, {"Value.last_updated", "Value.latest", "Value.locations", "Value.source"}),
#"Expanded Value.locations" = Table.ExpandListColumn(#"Expanded Value", "Value.locations"),
#"Expanded Value.locations1" = Table.ExpandRecordColumn(#"Expanded Value.locations", "Value.locations", {"coordinates", "country", "country_code", "history", "latest", "province"}, {"Value.locations.coordinates", "Value.locations.country", "Value.locations.country_code", "Value.locations.history", "Value.locations.latest", "Value.locations.province"}),
#"Expanded Value.locations.coordinates" = Table.ExpandRecordColumn(#"Expanded Value.locations1", "Value.locations.coordinates", {"lat", "long"}, {"Value.locations.coordinates.lat", "Value.locations.coordinates.long"}),
#"Expanded Value.locations.history" = Table.ExpandRecordColumn(#"Expanded Value.locations.coordinates", "Value.locations.history", {"1/22/20", "1/23/20", "1/24/20", "1/25/20", "1/26/20", "1/27/20", "1/28/20", "1/29/20", "1/30/20", "1/31/20", "2/1/20", "2/10/20", "2/11/20", "2/12/20", "2/13/20", "2/14/20", "2/15/20", "2/16/20", "2/17/20", "2/18/20", "2/19/20", "2/2/20", "2/20/20", "2/21/20", "2/22/20", "2/23/20", "2/24/20", "2/25/20", "2/26/20", "2/27/20", "2/28/20", "2/29/20", "2/3/20", "2/4/20", "2/5/20", "2/6/20", "2/7/20", "2/8/20", "2/9/20", "3/1/20", "3/10/20", "3/11/20", "3/12/20", "3/13/20", "3/14/20", "3/15/20", "3/16/20", "3/17/20", "3/18/20", "3/19/20", "3/2/20", "3/20/20", "3/21/20", "3/22/20", "3/23/20", "3/24/20", "3/25/20", "3/26/20", "3/27/20", "3/3/20", "3/4/20", "3/5/20", "3/6/20", "3/7/20", "3/8/20", "3/9/20"}, {"Value.locations.history.1/22/20", "Value.locations.history.1/23/20", "Value.locations.history.1/24/20", "Value.locations.history.1/25/20", "Value.locations.history.1/26/20", "Value.locations.history.1/27/20", "Value.locations.history.1/28/20", "Value.locations.history.1/29/20", "Value.locations.history.1/30/20", "Value.locations.history.1/31/20", "Value.locations.history.2/1/20", "Value.locations.history.2/10/20", "Value.locations.history.2/11/20", "Value.locations.history.2/12/20", "Value.locations.history.2/13/20", "Value.locations.history.2/14/20", "Value.locations.history.2/15/20", "Value.locations.history.2/16/20", "Value.locations.history.2/17/20", "Value.locations.history.2/18/20", "Value.locations.history.2/19/20", "Value.locations.history.2/2/20", "Value.locations.history.2/20/20", "Value.locations.history.2/21/20", "Value.locations.history.2/22/20", "Value.locations.history.2/23/20", "Value.locations.history.2/24/20", "Value.locations.history.2/25/20", "Value.locations.history.2/26/20", "Value.locations.history.2/27/20", "Value.locations.history.2/28/20", "Value.locations.history.2/29/20", "Value.locations.history.2/3/20", "Value.locations.history.2/4/20", "Value.locations.history.2/5/20", "Value.locations.history.2/6/20", "Value.locations.history.2/7/20", "Value.locations.history.2/8/20", "Value.locations.history.2/9/20", "Value.locations.history.3/1/20", "Value.locations.history.3/10/20", "Value.locations.history.3/11/20", "Value.locations.history.3/12/20", "Value.locations.history.3/13/20", "Value.locations.history.3/14/20", "Value.locations.history.3/15/20", "Value.locations.history.3/16/20", "Value.locations.history.3/17/20", "Value.locations.history.3/18/20", "Value.locations.history.3/19/20", "Value.locations.history.3/2/20", "Value.locations.history.3/20/20", "Value.locations.history.3/21/20", "Value.locations.history.3/22/20", "Value.locations.history.3/23/20", "Value.locations.history.3/24/20", "Value.locations.history.3/25/20", "Value.locations.history.3/26/20", "Value.locations.history.3/27/20", "Value.locations.history.3/3/20", "Value.locations.history.3/4/20", "Value.locations.history.3/5/20", "Value.locations.history.3/6/20", "Value.locations.history.3/7/20", "Value.locations.history.3/8/20", "Value.locations.history.3/9/20"}),
#"Removed Columns" = Table.RemoveColumns(#"Expanded Value.locations.history",{"Value.source"}),
#"Reordered Columns" = Table.ReorderColumns(#"Removed Columns",{"Name", "Value.last_updated", "Value.latest", "Value.locations.latest", "Value.locations.province", "Value.locations.coordinates.lat", "Value.locations.coordinates.long", "Value.locations.country", "Value.locations.country_code", "Value.locations.history.1/22/20", "Value.locations.history.1/23/20", "Value.locations.history.1/24/20", "Value.locations.history.1/25/20", "Value.locations.history.1/26/20", "Value.locations.history.1/27/20", "Value.locations.history.1/28/20", "Value.locations.history.1/29/20", "Value.locations.history.1/30/20", "Value.locations.history.1/31/20", "Value.locations.history.2/1/20", "Value.locations.history.2/10/20", "Value.locations.history.2/11/20", "Value.locations.history.2/12/20", "Value.locations.history.2/13/20", "Value.locations.history.2/14/20", "Value.locations.history.2/15/20", "Value.locations.history.2/16/20", "Value.locations.history.2/17/20", "Value.locations.history.2/18/20", "Value.locations.history.2/19/20", "Value.locations.history.2/2/20", "Value.locations.history.2/20/20", "Value.locations.history.2/21/20", "Value.locations.history.2/22/20", "Value.locations.history.2/23/20", "Value.locations.history.2/24/20", "Value.locations.history.2/25/20", "Value.locations.history.2/26/20", "Value.locations.history.2/27/20", "Value.locations.history.2/28/20", "Value.locations.history.2/29/20", "Value.locations.history.2/3/20", "Value.locations.history.2/4/20", "Value.locations.history.2/5/20", "Value.locations.history.2/6/20", "Value.locations.history.2/7/20", "Value.locations.history.2/8/20", "Value.locations.history.2/9/20", "Value.locations.history.3/1/20", "Value.locations.history.3/10/20", "Value.locations.history.3/11/20", "Value.locations.history.3/12/20", "Value.locations.history.3/13/20", "Value.locations.history.3/14/20", "Value.locations.history.3/15/20", "Value.locations.history.3/16/20", "Value.locations.history.3/17/20", "Value.locations.history.3/18/20", "Value.locations.history.3/19/20", "Value.locations.history.3/2/20", "Value.locations.history.3/20/20", "Value.locations.history.3/21/20", "Value.locations.history.3/22/20", "Value.locations.history.3/23/20", "Value.locations.history.3/24/20", "Value.locations.history.3/25/20", "Value.locations.history.3/26/20", "Value.locations.history.3/27/20", "Value.locations.history.3/3/20", "Value.locations.history.3/4/20", "Value.locations.history.3/5/20", "Value.locations.history.3/6/20", "Value.locations.history.3/7/20", "Value.locations.history.3/8/20", "Value.locations.history.3/9/20"}),
 Cols = Table.ColumnNames(#"Reordered Columns"),
ColsUnPivot = List.Skip(Cols,9),
Unpivot = Table.Unpivot( #"Reordered Columns",ColsUnPivot,"Attribute","Value"),
mahdighazali commented 4 years ago

Hello and thank you so much for maintaining this! I have created an API that reads your data and returns it in a way that's more friendly to use in programs. It also supports history. It's still a W.I.P and I would love contributions! It is open-sourced here: https://github.com/ExpDev07/coronavirus-tracker-api. Feel free to use it in your projects!

It is very fast due to caching.

The current endpoints are (more will be added): https://coronavirus-tracker-api.herokuapp.com/confirmed https://coronavirus-tracker-api.herokuapp.com/deaths https://coronavirus-tracker-api.herokuapp.com/recovered

For all of them combined: https://coronavirus-tracker-api.herokuapp.com/all

Hi @ExpDev07 , Thank you for the API. It's really helpfull...Please if you can provide me when you refresh the data...It's 2020-03-29 and the last refresh of the data is 2020-03-27 :) Thank you very much for your help :)