#121 Replace main.sh with main.py

JackGilmore commented 1 year ago

Description

Replacing main.sh with main.py as an entrypoint to the scraping scripts for the pipeline
Adding loguru as a logging library
Moves the static web scrapers out of the web-scrapers directory and into the root

Motivation and Context

Inconsistencies between main.sh and live pipeline causing issues. This would solve it if we subbed out all the *.py file calls to just call main.py

How Has This Been Tested?

Ran script and compared files to last sync. Not much change other than stale datasets.

Screenshots (if appropriate):

N/A

Types of changes

[ ] Bug fix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[X] Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

[X] My code follows the code style of this project.
[X] My change requires a change to the documentation.
[ ] I have updated the documentation accordingly.
[X] I have read the CONTRIBUTING document.
[ ] I have added tests to cover my changes.
[ ] All new and existing tests passed.

KarenJewell commented 1 year ago

lol this is a big one to check. I would remove /data and log files from the PR though as they are time sensitive (i.e. by the time we merge the files will be out of date) the next pipeline run will handle it.

KarenJewell commented 1 year ago

@JackGilmore could you update docs also for this change please? https://docs.opendata.scot/contribute/How-to-run-backend-locally/

JackGilmore commented 1 year ago

@JackGilmore could you update docs also for this change please? https://docs.opendata.scot/contribute/How-to-run-backend-locally/

Have opened OpenDataScotland/opendata.scot_docs#7 to address this. Ready to be merged in once this is merged in.

KarenJewell commented 1 year ago

lgtm.

OpenDataScotland / the_od_bods