Clean Up ETL Functionality

paul-tqh-nguyen / arxiv_as_a_newspaper

arxiv.org portrayed as if it were a news paper.

0 stars 0 forks source link

Clean Up ETL Functionality #4

Closed paul-tqh-nguyen closed 5 years ago

paul-tqh-nguyen commented 5 years ago

The goal of task #2 is just to get the ETL working.

This task is to enumerate a lot of the flaws and clean them up and to further robustify that functionality.

Some initial goals include:

[x] Update access privileges
[x] Make sure everything is used
[ ] Make sure everything has types declared

paul-tqh-nguyen commented 5 years ago

Progress Patch: https://github.com/paul-tqh-nguyen/arxiv_as_a_newspaper/commit/e9df9aa7fee2df9b8bdc3748152d26d0e91285a9

We were missing exception handling for a case relating to bad authentication credentials.

paul-tqh-nguyen commented 5 years ago

Progress Patch: https://github.com/paul-tqh-nguyen/arxiv_as_a_newspaper/commit/4c5879f251300a5998525e6e697092d361fa3735

After this patch, everything is used.

paul-tqh-nguyen commented 5 years ago

The result of almost everything in our ETL is an iterator, and the precision to which the types of those can be checked is very limited; thus, we're going to forgo the efforts to proliferate type declarations within our ETL at this time.