Add BigQuery usage, data syncing (iss. #20, #21)

tl-its-umich-edu / mpr-research-data

3 stars 4 forks source link

Add BigQuery usage, data syncing (iss. #20, #21) #22

Closed takposha closed 2 years ago

takposha commented 2 years ago

This PR should add BigQuery storage, and also check if course data has been updated, and only then push it to the table. This should allow for predictions to be made only on fresh data to prevent the pipeline from processing the same data repeatedly.

pep8speaks commented 2 years ago

Hello @takposha! Thanks for updating this PR. We checked the lines you've touched for PEP 8 issues, and found:

In the file mpr-research-data.py:

Line 173:39: W503 line break before binary operator

Comment last updated at 2022-06-13 15:44:47 UTC

takposha commented 2 years ago

I now see that this commit traces back to Lance's first commits that were not signed. I thought that was resolved, but seemingly not.