SPC toolkit python package

[sgreenbury]

Closes #55.

Remaining tasks:

• [x] Refactor to single class for reading protobuf or parquet
• [x] Complete support for pandas backend:
  • [x] Reader
  • [x] Merging
• [x] Refine merging methods/implement dataframe builder
• [x] Update dependencies and create dev and examples groups
• [x] Update notebook examples
• [x] Add build and running tests to actions
• [x] Update README with different install options including pip
• [x] Check serialized population for flat output matches single protobuf structure
• [x] Test examples
• [x] Update test data
• [x] Remove #[derive(Serialize) on types no longer requiring serialize
• [x] Add test for determinism in parquet and JSON
• [x] Fix unnest exception when column exists in dataframe after unnesting (e.g. nssec8)
• [x] Implement reading from remote gzip
• [x] Re-export Reader and Builder classes
• [x] Update CLI with --output-formats Vec<OutputFormat>
• [x] Update pages docs

[review-notebook-app[bot]]

Check out this pull request on 

See visual diffs & provide feedback on Jupyter Notebooks.

---

Powered by ReviewNB

[dabreegster]

Hi Sam, @Hussein-Mahfouz would like to use this; need a beta tester?

[sgreenbury]

That would be great, thanks! @Hussein-Mahfouz the python/README.md is the best place to start but happy to discuss if unclear at all. Just to flag the API will change while this PR is in draft and the remaining features at the top are added.

Any comments on the usability/missing features would be very helpful!

[Hussein-Mahfouz]

Hi @sgreenbury ! I'm trying the SPC toolkit python package now, but having a small issue (I haven't used poetry before).

• I've gone through all the requirements in the README.
• I've also run poetry install but I get the following error

Poetry could not find a pyproject.toml file in /home/hussein/Documents/GitHub/uatk-spc or its parents

• I've searched the repo and there isn't a pyproject.toml. I see from this issue that there should be one https://github.com/alan-turing-institute/uatk-spc/issues/55. Am I missing something?
• If there is no pyproject.toml, do I currently need to create my own virtual environment to run the python code?

[sgreenbury]

Hi @Hussein-Mahfouz, thanks for trying this out! Currently the instructions in the README assume you are on the branch associated with this PR (55-output-formats-python) and are in the the python/ directory (where the README and pyproject.toml are), apologies that wasn't clear. From the repo root if you try:

git checkout 55-output-formats-python
cd python

then poetry install should work as you will now have the pyproject.toml and package.

[sgreenbury]

Removed upper limit on required python version (ad1d826, see https://github.com/Urban-Analytics-Technology-Platform/acbm/issues/10).

[sgreenbury]

Thanks @dabreegster!