Data source

[konradkalemba]

Hi all!

Currently the data is updated manually from MZ Twitter account. However, in the long run this approach is not effective.

I found the official website - https://www.gov.pl/web/koronawirus/wykaz-zarazen-koronawirusem-sars-cov-2 where we can get the data from automatically. Their data are a bit inconsistent though - there are cities specified in some cases, in others there are "powiaty".

There is one more source we scrap the data from - https://docs.google.com/spreadsheets/d/1ierEhD6gcq51HAm433knjnVwey4ZE5DCnu1bW7PRG3E/htmlview?usp=sharing&sle=true

Any thoughts?

[mhajder]

For me, the idea of scraping official data is a good one. But is it sense to scrape them to create the same map? I would stay with the official account of the MZ twitter.

I like the data with cities more. They are more readable.

You would also need to prepare the same map as it is on gov.pl, or use coordinate location by place name.

JSON data from gov.pl

```json [ { "Województwo":"Cała Polska", "Powiat/Miasto":"Łącznie", "Liczba":"156", "Liczba zgonów":"3", "Id":"t0000" }, { "Województwo":"łódzkie", "Powiat/Miasto":"bełchatowski", "Liczba":"1", "Liczba zgonów":"", "Id":"t1001" }, { "Województwo":"śląskie", "Powiat/Miasto":"będziński", "Liczba":"2", "Liczba zgonów":"", "Id":"t2401" }, { "Województwo":"lubelskie", "Powiat/Miasto":"biłgorajski", "Liczba":"1", "Liczba zgonów":"", "Id":"t0602" }, { "Województwo":"dolnośląskie", "Powiat/Miasto":"bolesławiecki", "Liczba":"1", "Liczba zgonów":"", "Id":"t0201" }, { "Województwo":"śląskie", "Powiat/Miasto":"Chorzów", "Liczba":"1", "Liczba zgonów":"", "Id":"t2463" }, { "Województwo":"śląskie", "Powiat/Miasto":"cieszyński", "Liczba":"4", "Liczba zgonów":"", "Id":"t2403" }, { "Województwo":"pomorskie", "Powiat/Miasto":"Gdańsk", "Liczba":"2", "Liczba zgonów":"", "Id":"t2261" }, { "Województwo":"lubuskie", "Powiat/Miasto":"Gorzów Wielkopolski", "Liczba":"1", "Liczba zgonów":"", "Id":"t0861" }, { "Województwo":"lubelskie", "Powiat/Miasto":"janowski", "Liczba":"5", "Liczba zgonów":"", "Id":"t0605" }, { "Województwo":"opolskie", "Powiat/Miasto":"kędzierzyńsko-kozielski", "Liczba":"1", "Liczba zgonów":"", "Id":"t1603" }, { "Województwo":"świętokrzyskie", "Powiat/Miasto":"Kielce", "Liczba":"1", "Liczba zgonów":"", "Id":"t2661" }, { "Województwo":"małopolskie", "Powiat/Miasto":"Kraków", "Liczba":"2", "Liczba zgonów":"", "Id":"t1261" }, { "Województwo":"opolskie", "Powiat/Miasto":"krapkowicki", "Liczba":"1", "Liczba zgonów":"", "Id":"t1605" }, { "Województwo":"dolnośląskie", "Powiat/Miasto":"Legnica", "Liczba":"1", "Liczba zgonów":"", "Id":"t0262" }, { "Województwo":"dolnośląskie", "Powiat/Miasto":"legnicki", "Liczba":"1", "Liczba zgonów":"", "Id":"t0209" }, { "Województwo":"podkarpackie", "Powiat/Miasto":"leżajski", "Liczba":"7", "Liczba zgonów":"", "Id":"t1808" }, { "Województwo":"lubelskie", "Powiat/Miasto":"lubartowski", "Liczba":"1", "Liczba zgonów":"", "Id":"t0608" }, { "Województwo":"lubelskie", "Powiat/Miasto":"Lublin", "Liczba":"9", "Liczba zgonów":"1", "Id":"t0663" }, { "Województwo":"łódzkie", "Powiat/Miasto":"łódzki wschodni", "Liczba":"1", "Liczba zgonów":"", "Id":"t1006" }, { "Województwo":"łódzkie", "Powiat/Miasto":"Łódź", "Liczba":"14", "Liczba zgonów":"", "Id":"t1061" }, { "Województwo":"dolnośląskie", "Powiat/Miasto":"oleśnicki", "Liczba":"1", "Liczba zgonów":"", "Id":"t0214" }, { "Województwo":"warmińsko-mazurskie", "Powiat/Miasto":"Olsztyn", "Liczba":"3", "Liczba zgonów":"", "Id":"t2862" }, { "Województwo":"świętokrzyskie", "Powiat/Miasto":"ostrowiecki", "Liczba":"3", "Liczba zgonów":"", "Id":"t2607" }, { "Województwo":"warmińsko-mazurskie", "Powiat/Miasto":"ostródzki", "Liczba":"1", "Liczba zgonów":"", "Id":"t2815" }, { "Województwo":"łódzkie", "Powiat/Miasto":"pabianicki", "Liczba":"1", "Liczba zgonów":"", "Id":"t1008" }, { "Województwo":"mazowieckie", "Powiat/Miasto":"piaseczyński", "Liczba":"3", "Liczba zgonów":"", "Id":"t1418" }, { "Województwo":"łódzkie", "Powiat/Miasto":"poddębicki", "Liczba":"1", "Liczba zgonów":"", "Id":"t1011" }, { "Województwo":"zachodniopomorskie", "Powiat/Miasto":"policki", "Liczba":"2", "Liczba zgonów":"", "Id":"t3211" }, { "Województwo":"wielkopolskie", "Powiat/Miasto":"Poznań", "Liczba":"7", "Liczba zgonów":"1", "Id":"t3064" }, { "Województwo":"wielkopolskie", "Powiat/Miasto":"poznański", "Liczba":"3", "Liczba zgonów":"", "Id":"t3021" }, { "Województwo":"mazowieckie", "Powiat/Miasto":"pruszkowski", "Liczba":"3", "Liczba zgonów":"", "Id":"t1421" }, { "Województwo":"mazowieckie", "Powiat/Miasto":"Radom", "Liczba":"1", "Liczba zgonów":"", "Id":"t1463" }, { "Województwo":"śląskie", "Powiat/Miasto":"Rybnik", "Liczba":"5", "Liczba zgonów":"", "Id":"t2473" }, { "Województwo":"podkarpackie", "Powiat/Miasto":"Rzeszów", "Liczba":"1", "Liczba zgonów":"", "Id":"t1863" }, { "Województwo":"lubuskie", "Powiat/Miasto":"słubicki", "Liczba":"1", "Liczba zgonów":"", "Id":"t0805" }, { "Województwo":"śląskie", "Powiat/Miasto":"Sosnowiec", "Liczba":"1", "Liczba zgonów":"", "Id":"t2475" }, { "Województwo":"zachodniopomorskie", "Powiat/Miasto":"stargardzki", "Liczba":"3", "Liczba zgonów":"", "Id":"t3214" }, { "Województwo":"opolskie", "Powiat/Miasto":"strzelecki", "Liczba":"4", "Liczba zgonów":"", "Id":"t1611" }, { "Województwo":"zachodniopomorskie", "Powiat/Miasto":"Szczecin", "Liczba":"1", "Liczba zgonów":"", "Id":"t3262" }, { "Województwo":"warmińsko-mazurskie", "Powiat/Miasto":"szczycieński", "Liczba":"1", "Liczba zgonów":"", "Id":"t2817" }, { "Województwo":"śląskie", "Powiat/Miasto":"tarnogórski", "Liczba":"1", "Liczba zgonów":"", "Id":"t2413" }, { "Województwo":"lubelskie", "Powiat/Miasto":"tomaszowski", "Liczba":"1", "Liczba zgonów":"", "Id":"t0618" }, { "Województwo":"łódzkie", "Powiat/Miasto":"tomaszowski", "Liczba":"1", "Liczba zgonów":"", "Id":"t1016" }, { "Województwo":"śląskie", "Powiat/Miasto":"Tychy", "Liczba":"1", "Liczba zgonów":"", "Id":"t2477" }, { "Województwo":"dolnośląskie", "Powiat/Miasto":"wałbrzyski", "Liczba":"1", "Liczba zgonów":"", "Id":"t0221" }, { "Województwo":"mazowieckie", "Powiat/Miasto":"Warszawa", "Liczba":"24", "Liczba zgonów":"", "Id":"t1465" }, { "Województwo":"dolnośląskie", "Powiat/Miasto":"Wrocław", "Liczba":"18", "Liczba zgonów":"1", "Id":"t0264" }, { "Województwo":"śląskie", "Powiat/Miasto":"zawierciański", "Liczba":"3", "Liczba zgonów":"", "Id":"t2416" }, { "Województwo":"łódzkie", "Powiat/Miasto":"zgierski", "Liczba":"3", "Liczba zgonów":"", "Id":"t1020" } ] ```

[mulawamichal]

there are cities specified in some cases, in others there are "powiaty".

sometimes powiat is named same as city. theese are "miasta na parawach powiatu". see https://pl.wikipedia.org/wiki/Lista_powiat%C3%B3w_w_Polsce or TERYT database (http://eteryt.stat.gov.pl/eTeryt/rejestr_teryt/udostepnianie_danych/baza_teryt/uzytkownicy_indywidualni/wyszukiwanie/wyszukiwanie.aspx?contrast=default)

[konradkalemba]

@mhajder There is also an option to stay with manually updated data, but we would have to create a team responsible for data updating, because right now I'm not available all the time. With more people, delays in data update would be lower.

@mulawamichal I see - that's good, we wouldn't have to deal with showing "powiat" on map.

In both approaches there are some trade-offs:

• In manual data update with more people someone would have to be online, which isn't the problem with big team. But the problem is to assemble such team.

• In automatic scraping we would lose source of individual cases and of course there is some time to implement such scraper.

[mhajder]

@konradkalemba I can help with adding data.

Implementing such a scraper is very simple. It's best to do a simple python script that will be run in cron. Also, page scraping is not very ethical. It generates a lot of traffic.

[konradkalemba]

@mhajder I know that isn't the best way ethical-wise, but it wouldn't generate a lot of traffic though. Running the script every 5 minutes wouldn't hurt the server very much.

An another problem with this data source is that I'm not sure if it's updated regularly

[konradkalemba]

Okay, for the time being we are staying with the data updated manually. Official MZ's website was outdated for at least 1 hour after the latest confirmation.

[mhajder]

Okay, for the time being we are staying with the data updated manually. Official MZ's website was outdated for at least 1 hour after the latest confirmation.

Probably the change are only during working hour 😆

[konradkalemba]

There is an another problem - MZ's Twitter doesn't specify cities where new cases are, only voivodeships now...

[mulawamichal]

geoportal now has "koronawirus" layer: https://mapy.geoportal.gov.pl/imap/Imgp_2.html?locale=en&gui=new&sessionID=4955220

[mhajder]

geoportal now has "koronawirus" layer: https://mapy.geoportal.gov.pl/imap/Imgp_2.html?locale=en&gui=new&sessionID=4955220

But it is extremely difficult to scrape it.

For gov.pl, all you need is:

import json

import requests
from bs4 import BeautifulSoup

response = requests.get('https://www.gov.pl/web/koronawirus/wykaz-zarazen-koronawirusem-sars-cov-2')

soup = BeautifulSoup(response.text, 'html.parser')

json_data = json.loads((soup.find(id='registerData')).text)
print(json_data['parsedData'])

[konradkalemba]

@mulawamichal @mhajder Guys, we have a big problem - their official twitter account lists only voivodeships now as I wrote above, but I thought maybe their website will be more precise... but they limited both map and table to voivodeships.

[mhajder]

@konradkalemba Gov.pl also now only provides voivodships.

[konradkalemba]

@mhajder Yes... I think we also have to do so. Because trying to find where every new case is will be very time-consuming.

[mhajder]

Also correct the case that the patient recovered, as it is written on Twitter this is the first patient. So it is from Zielona Góra.