search

mysociety / theyworkforyou

Keeping tabs on the UK's parliaments and assemblies
http://www.theyworkforyou.com/
Other
224 stars 50 forks source link

Division title importer/exporter #1765

Closed ajparsons closed 3 months ago

ajparsons commented 4 months ago

This PR adds division_io.py - a click CLI to handle importing and exporting division data.

image

This handles bulk importing updates to division titles (e.g. from Parliament's commons votes api), and exporting parquet tables of divisions as a basic api to feed twfy-votes.

There is an adjustment to the division table schema - adding a title_priority field. This is to keep track of the origin of the current title, and stop manual updates to titles being overridden by automated ones. Existing titles with a yes_text are retrospectively updated in priority.

For the moment, this uses a requirements.txt file for the needed python packages. When the server is updated, I'll do a new PR to tidy up the python-tooling in general. Unsure for the moment if this needs the morningupdate commands to run in a venv, or if system packages can just be updated in the short term.

ajparsons commented 4 months ago

I've added a venv creation process to deploy.bash.

Added some packages to the packages file for future docker use but these are all present on the server anyway so should be fine.

ajparsons commented 3 months ago

Issues above resolved, I've also just quickly updated the packages now we're not stuck on an old pandas version because of 3.7 - https://github.com/mysociety/theyworkforyou/pull/1765/commits/49053830b6d01c0baa9156abddfc7bf693fa82a4