search

thegraphnetwork / epigraphhub_py

Epigraphhub Python package
GNU General Public License v3.0
2 stars 9 forks source link

BREAKING CHANGE(refactor SINAN data collection): Move SINAN data collection to ETL format #194

Closed luabida closed 1 year ago

luabida commented 1 year ago

I've made some changes so Airflow can best perform while fetching data from SINAN. I also used the my branch on PySUS SINAN because of issue https://github.com/AlertaDengue/PySUS/issues/99

My plan in to make download() method asynchronous, so it can speed up the download when requested every year from a specific disease

Usage:

from epigraphhub.data.brasil.sinan import extract, loading, viz

#extract data for zika:
zika = extract.download('Zika')

#upload data into psql:
loading.upload(zika)

#visualize data:
viz.parquet(zika[0]) #has to be the path
viz.table('Zika', 2018)
luabida commented 1 year ago

I've made a lot of changes on PySUS package, now the ETL codes here are very simple here bc is almost everything done by PySUS.SINAN.

Still, upserting is very slow. @fccoelho I remeber you mentioned a better way of inserting into sql, can you link it again here please?

luabida commented 1 year ago

@xmnlab Thank you so much for your review, I'm going to look into that now

luabida commented 1 year ago

PR waiting PySUS rebuild in Pypi

fccoelho commented 1 year ago

@luabida any progress here?

luabida commented 1 year ago

@fccoelho I've encountered some errors I'm not so sure how to solve it. It is supposedly done, but for some reason it is still failing locally

fccoelho commented 1 year ago

@fccoelho I've encountered some errors I'm not so sure how to solve it. It is supposedly done, but for some reason it is still failing locally can you point to what is happening?

luabida commented 1 year ago

The tasks are apparently failing to the airflow, but they are doing what they are supposed to do. image

image

fccoelho commented 1 year ago

According to this Airflow will mark a task successful only if it hasn't raised and exception. Can you check for that?

luabida commented 1 year ago

I think it is working, this PR depends on epigraphhub_py and PySUS

fccoelho commented 1 year ago

Shall we merge? @luabida

luabida commented 1 year ago

I think so, thank you flávio. Any bugs that may show up will be solved

github-actions[bot] commented 1 year ago

:tada: This PR is included in version 2.0.0 :tada:

The release is available on:

Your semantic-release bot :package::rocket: