US data is missing from daily data file April 1st. SOLVED!

[JuanPufleau]

US data is missing from daily data file April 1st.

https://github.com/CSSEGISandData/COVID-19/blob/master/csse_covid_19_data/csse_covid_19_daily_reports/04-01-2020.csv

[amalic]

Can confirm!

[joshpritch]

US data is missing from the app as well

[ryanwoconnor]

Why is this data missing? People rely on this.

[rmoyers58]

Yes the US data is missing

[icarlitoss]

Thanks for posting this. A quick alternative (until we wait for this fix) at this source: http://dash-coronavirus-2020.herokuapp.com/

[judecalvillo]

Can confirm that US data is missing and affecting the dashboard...

[mashranga]

US data missing !! But why !! Updating the dashboard?

[oltdaniel]

I think they are currently entering the data, as e.g. deaths is not entered for now on the dashboard. Still amazing work you do here collecting the data. To the developers, yes it could be more reliable.

[sailorsails]

Looks like it is back up

[amalic]

Instead of the state/province the only US line in the dataset says "Recovered" despite having a Recovered column.

[oltdaniel]

@sailorsails Well CDN sucks of arcgics, I still have 712 with cache disabled. 😆 @amalic Something we need to live with. It is still the most "official" data source we could get. @CSSEGISandData does still a good job about collecting, and medium at publishing.

[judecalvillo]

Looks like it is back up

Oh, interesting. When I click 'US' on the left, to subset the other visuals, the dash appears to revert. Probably just a cache issue.

[KrishnaAgarwal05]

Data is still missing from the daily report files for 04/1. Would appreciate if someone can look into it.

[amalic]

@amalic Something we need to live with. It is still the most "official" data source we could get.

I know. GitHub issues are to me just feedback. No offense intended.

[oltdaniel]

@KrishnaAgarwal05 I find it more unusual that the automated update for the timeseries isn't out yet. Normally it comes 5mins later than the daily report automated update, so still waiting for it at 2am in the morning... 🤣

Anyone interested, I convert the data each day into a "good" format (though states get lost in conversion) and have a small visualization on https://oltdaniel.at/2020/covid-19-analysis.html.

[octopicorn]

Both US and Canada data are missing from Daily report file 2020-04-01: https://github.com/CSSEGISandData/COVID-19/blob/master/csse_covid_19_data/csse_covid_19_daily_reports/04-01-2020.csv#L60

Every previous file had a row for each US county. Where did this go?

A breaking change here is that the word "Recovered" appears as the Province_State value. This will throw off any code looking to this column for matching with states in US and Canada. That is why "Recovered" appears mysteriously in dashboard.

[amalic]

Both US and Canada data are missing from Daily report file 2020-04-01:

Canada data is in the file

[octopicorn]

@amalic line 59, it has "Recovered" as a Province, with values 0 for Confirmed and Deaths. You are correct that province rows are filled in above.

[amalic]

@CSSEGISandData

[oltdaniel]

@octopicorn @amalic @k7nixen Each commit here needs to be reviewd by each developer to verify their needs to their application. I personally stay awake during night and verify the commits to finally convert them to my data format at https://c.oltdaniel.at . They (@CSSEGISandData) do an aazing work here, as collecting the data is extremly hard, due to countries not reporting enough or not having an reliable data source to collect the data from. For us as developers, using this datasource here, we have the task of putting it into a finaly beauty of data format.

[amalic]

@CSSEGISandData @hongru94 @enshengdong @arthurzhang434 You might want to take a look at this. This is not an April fools joke!

[octopicorn]

@oltdaniel nice work with your data vis. appreciate your focus on the commits. i can' timagine having to manage this work, there are 1000+ issues and 192 PRs currently!

yes, i'm just here to help report the issue. very grateful for the excellent reference work of collecting this info. i just turned off my data feed for 2020-04-01 until the US data comes back.

this file is created by an automatic process, i think, so some breaking change must have happened upstream

[joshpritch]

Is there any data source that tracks recoveries by state on a daily basis?

[amalic]

I personally stay awake during night and verify the commits to finally convert them to my data format at https://c.oltdaniel.at . They (@CSSEGISandData) do an aazing work here

I am also staying awake since a couple of days to make sure, it gets updated properly. Switched today even to a different strategy using Google Apps Script.

I totally agree about the work they do, as long as the data format does not change. I don't mind having additional columns added, but renaming or replacing columns is not ideal since the file we are using is to us an API.

[amalic]

Is there any data source that tracks recoveries by state on a daily basis?

This one does for a couple of countries, e.g. USA, Canada, China when it works properly.

[DinsdalePiranha85]

No Time-Series update (Global OR USA) either.

[joshpritch]

Is there any data source that tracks recoveries by state on a daily basis?

This one does for a couple of countries, e.g. USA, Canada, China when it works properly.

Maybe I'm blind but this report shows 0 recoveries for all US rows. I must be missing something obvious.

[octopicorn]

Just as reference point, the Daily Report file from 2020-03-31 had 2436 rows Today's has only 257 rows. All the county rows are gone.

[amalic]

Is there any data source that tracks recoveries by state on a daily basis?

This one does for a couple of countries, e.g. USA, Canada, China when it works properly.

Maybe I'm blind but this report shows 0 recoveries for all US rows. I must be missing something obvious.

It does. Recovered is now a US state/province

[amalic]

Just as reference point, the Daily Report file from 2020-03-31 had 2436 rows Today's has only 257 rows. All the county rows are gone.

That's not correct. see https://docs.google.com/spreadsheets/d/15OKNhZosR9xb0gShWtKw0P2toEN5AabZfp6kfnsVEKs/edit#gid=0

Only US is missing

[joshpritch]

Is there any data source that tracks recoveries by state on a daily basis?

This one does for a couple of countries, e.g. USA, Canada, China when it works properly.

Maybe I'm blind but this report shows 0 recoveries for all US rows. I must be missing something obvious.

It does. Recovered is now a US state/province

Yes I see that. I just meant is there a data source that breaks the Recovered number down by county or state?

[amalic]

Is there any data source that tracks recoveries by state on a daily basis?

This one does for a couple of countries, e.g. USA, Canada, China when it works properly.

Maybe I'm blind but this report shows 0 recoveries for all US rows. I must be missing something obvious.

It does. Recovered is now a US state/province

Yes I see that. I just meant is there a data source that breaks the Recovered number down by county or state?

There are other data sets, but you need to register to access them. This is the only openly available that I am aware of.

[andyquinterom]

My best advice, for now, is to limit your data to March 31st. I have set my data visualization web app to only go up to March 31st. You can download the data in a more "friendly" format through this link: https://users.indexmic.com/corona/ I'll inform you if anything changes.

[oltdaniel]

@amalic I currently work issue by issue until I will fell asleep to reduce the issue work for @CSSEGISandData. Willing to help?

[tienthanhdhcn]

If you only care about the state level, please see my backup here https://www.dropbox.com/s/zzbmmelywsxnh28/04-01-2020.csv

[ebwinters]

how is a top public research institution releasing this data? If you don't have the data, that's fine, but you realize when you have thousands of users, you can't just introduce breaking changes with no warning.

[rkrinjeck]

Is a file/data update coming or in the works? The JHU Dashboard is working, so I'm assuming that you have the correct data. We just need an update.

[oltdaniel]

@ebwinters Put yourself into this position. No country prepared for such an event in this time with this tech. The reports happen in an unregular basis and different dates. It is extremly difficult to put al into one format. @CSSEGISandData does an amazing and difficult job.

[ebwinters]

@oltdaniel I'm simply pointing out, at some point once you have a lot of users relying on your data, there is an understanding that breaking changes are not made. I completely understand technical difficulties (especially at a time like this), which would lead to data being delayed, but the types of changes JHU has been making can have a huge impact on developers and the people using their projects.

[d-huck]

I have an automated script to update my data visualization and I have had to come back and check why my script breaks almost every single day for the last week. I shouldn't have to tweak my code every few days because the data keeps getting dropped or changed. I understand this is released free of charge, but there are people relying on this data.

[oltdaniel]

@ebwinters I agree, but this source is the best one we have. We need the ope source people that belive in small scripts and throwing time in this to upgade this data format. Yes, @CSSEGISandData can do a better job, but we need to split the time we put into this. So we need different layers with different reliabilty of data formats. I personally convert the data each day to https://c.oltdaniel.at to use it for my application.

[KrishnaAgarwal05]

Data has been updated

[amalic]

Works!

[octopicorn]

@KrishnaAgarwal05 excellent, thank you so much

[JuanPufleau]

I'll leave the Issue open for a while longer, so people who experienced problems can now see it's been resolved.

I'll close it later today.

[cjtexas]

Harris County Texas' FIPS column is still blank!