Feature: Data Download

[maxmalynowsky]

Adds two modules for assessing COD-AB quality:

• metadata: generates a spreadsheet at data/metadata.csv listing information that can be accessed from the ITOS ArcGIS server without downloading layers themselves.
• download: using the metadata sheet, iterates through the list to download every available layer.

Other changes:

• Remove black / isort in favour of ruff for linting and formatting.
• Add docker container able to run all commands through docker compose up.

[vshlemon]

Trying to test out the httpx.download option using just poetry & no docker. Got the following error, I believe it originates in the way the Path is stated here - cod-ab-data-quality/src/download/__init__.py

[maxmalynowsky]

Trying to test out the httpx.download option using just poetry & no docker. Got the following error, I believe it originates in the way the Path is stated here - cod-ab-data-quality/src/download/__init__.py

Not likely, otherwise src.metadata would have failed since it sets Path in the exact same way. What I think might be happening is that when you ran the command with Docker, there's a mkdir that created the download directory if it doesn't exist. It looks like there's a permission error accessing that folder from when it was created with Docker under WSL and then accessed by Windows. I restructured the data directory to add the folder boundaries are downloaded to into GitHub which should prevent that error.

[vshlemon]

Thanks Max - it seems ready to merge, but it takes a while as it downloads everything each time. I imagine sometimes this might not be desired so an option at the CLI for overwrite & output locations would be handy. My comments for the CLI features are below - I can either close this branch (merge) & open an issue to implement them or if you'd prefer to I can leave the branch open if you wanted to implement them? I am happy either way, would be good for me to implement to get more familiar but cool if you want to.

...ps. the cloud one will be unachievable for now and is just a comment for future

[maxmalynowsky]

Thanks Max - it seems ready to merge, but it takes a while as it downloads everything each time. I imagine sometimes this might not be desired so an option at the CLI for overwrite & output locations would be handy. My comments for the CLI features are below - I can either close this branch (merge) & open an issue to implement them or if you'd prefer to I can leave the branch open if you wanted to implement them? I am happy either way, would be good for me to implement to get more familiar but cool if you want to.

Thanks a lot for battle testing this, you've given a lot of super useful feedback, downloading is the hardest part since it deals with things outside of our control. I think we should merge now, otherwise this runs the risk of turning into a forever branch.

On the things you brought up:

• I don't think you can pass arguments to a make rule, for example make run --ISO3=MDG?
• This could be an option, but I wouldn't have this as the default behaviour. There's no way of knowing if the data you've already downloaded has a newer version available from the ArcGIS server without downloading it. My thinking was that you would use the ISO3 environment variable to run the pipeline for just a subset of locations. I was also thinking that downloading and running checks could be separate, users could add their own data to data/boundaries and skip make metadata / make download.
• For this, do you mean allow users to point to a different URL to get data from, and uploading to cloud storage? There has been interest from the Centre for Humanitarian Data in creating a new HDX dataset with all 525 current CODs in one easy to download zip file, something this tool could support.

[vshlemon]

Ok am merging now - will open up the download options as issues as I think it would be useful to include.

I meant adding arguments to the module execution via. an argparser. I wasn't envisioning make as the main point of entry for everyone, but it could keep the default args for module execution.

Yep, not as default, just for flexibility.

And I meant more for where you download to in case you didn't want to save locally. But that could come later if there's a need. And adding an argparser would enable for different sources to download from as well (or methods) so that would be good.

This is really great stuff, Thanks for your patience with me testing it out as well :)