Create database and celery task or management command to load it with data

[aptiko]

Once in a day or half a day or an hour, we will be deleting all the data from the database and importing fresh data. I think that for such a simple thing a management command run with cron is better than a celery task run with celerybeat (but if you're comfortable with Celery and you somehow you think it's better suited, go ahead and use celery).

The data should be loaded from https://github.com/CSSEGISandData/COVID-19/tree/master/csse_covid_19_data/csse_covid_19_time_series (unless you know of another source).

[abdullahkady]

I'm wondering - even though this is not in my interest :sweat_smile: -, couldn't we just use a frontend app and that's it?

Basically the "backend" will be a simple script that runs periodically, fetches the data and parses it into the appropriate JSON format, and then the frontend will be served with the JSON file bundled, and the rendering/configuration will all be frontend based.

[aptiko]

It's not a bad idea. Thanks for proposing it.

Here's the thing: I might not be able to find a front-end programmer soon. But I'm confident you (and probably I as well) could do the full stack alone, even if less perfectly. This is the fallback plan.

So, do you feel capable of doing the entire thing in the way you proposed? If yes, please go ahead and create that script that converts the information to JSON.

In fact, if we succeed in advertising it and it gets many users, this idea is way better—static files only.

BTW, are you aware of any list of countries with their populations?

[abdullahkady]

There are some datasets on kaggle as well, but I think Johns Hopkins are doing an amazing job. You can check their [dashboard(https://www.arcgis.com/apps/opsdashboard/index.html#/bda7594740fd40299423467b48e9ecf6), which I think has all you're looking for at the time?

Also I just found out that it's already available in JSON format on datahub The script can simply take that JSON, and parse it for faster access (for instance, group countries data points, and nest provinces too).

As for country-population data, I can look into it, we'll surely find something even if it's not updated frequently enough, the magnitude/ratio should still be sufficient.

[aptiko]

Thanks! I've seen this dashboard. Can it show deaths ("new", not cumulative) as a percentage of the population? Can it show many countries on the chart at the same time? I've been unable to find these things. If you can find charts that show these things, we may cancel the project. Thanks for continuing to try!

If the JSON already exists, so much the better! The CSVs on that site are definitely much better than the ones I had found.

[aptiko]

But if you can't find such charts, and given we already have the JSON, you could start working on the front-end!

[abdullahkady]

Perfect! Will start by searching a bit for existing dashboards, and implementing a basic frontend & transformation script.

As for the frontend, do you have any preferences when it comes to technologies? I'm suggesting a framework rather than vanilla JavaScript, I have worked with all the main three (Angular, Vue, and React). So any suggestion?

[aptiko]

Is using a framework going to be about as fast as plain JavaScript? If yes, go ahead and use a framework.

Now, don't tell anyone, but I'm a Toptal-certified React programmer—which only serves to prove that certifications don't mean anything :-) I'm more attracted to Vue, for two reasons. The first is that I worked with some time with a colleague who was a Vue fan, and he had an influence on me.

The second reason is that when I learned React, I used a book that had been published five months earlier, and it was starting to be outdated. A year later, well, don't mention it. These people don't know what "release notes" means. Vue is much better in this respect, or that's what this colleague told me. He also assured me Vue is easier, although I'm not certain about that—I remember I was overwhelmed when I looked into it. But the comparison isn't fair, since I learned React in two months, whereas I barely worked one week with Vue.

The thing is, whichever framework you use, I will barely be able to understand what you do, so you could just as well use Angular. It's up to you. My order of preference is Vue, React, Angular.

By the way, what about a CSS framework? As you know, I've been using Bootstrap 3, although lately I've taken a liking with Bulma (again, influenced by the same guy).

[abdullahkady]

Is using a framework going to be about as fast as plain JavaScript? If yes, go ahead and use a framework.

As for CSS, I have used both before, but I think I can leave that up to you. I'm not good when it comes to styling anyway, so whichever we end up choosing, I'll most probably end up googling some stackoverflow questions on "how to center this thing" :grin: I guess we'll also look into some components which could save us a bit of styling (just like the charts themselves).

[aptiko]

OK, let's use Bulma. BTW, if you use Vue, take a look at https://buefy.org/.

Also I think that before deciding on JS and CSS framework, the most important is to decide the graphic tool. Then we must use frameworks that already support it well.

[abdullahkady]

Alright. I have spent ~30m looking at other people's visualizations, and I gotta say that there is some awesome output out there, and the best thing is, almost everything is open source (thank you awesome community!). So I'm going to share some of the ones that I actually liked or found insightful compared to the rest which are just showing basic line graph of total cases/deaths.

This collection contains a number of different visualization that are open source, and most probably have their daily updates via scripts setup already. This Includes cases per 1Million person, which as far as I understood is something you're looking for.

And I think this one is pretty cool to visualize the trend too, I believe it had a trending YouTube video that I saw like a week ago. And then there is this basic chart that allows for visualizing the deltas (aka the change).

Finally I think this library is very powerful! Thought you might wanna check it out.

I have found that a lot of them are using the following JSON parser, which is what I had in mind. It is updated daily using github actions, so a simple crontab can be setup to pickup the latest file from the parsed repo.

Let me know what you think.

[aptiko]

This Includes cases per 1Million person, which as far as I understood is something you're looking for.

Essentially what I'm looking for is this: https://wattsupwiththat.com/daily-coronavirus-covid-19-data-graph-page/. The first chart you'll meet as you scroll down.

The difference is that I want "new" deaths (per 10 million), not cumulative.

If you can find the one I want, that's fine. Otherwise please create it, either in a new app or by adding it to whatever open source coronasoftware you like.