HowTheyVote / howtheyvote

HowTheyVote.eu allows anyone to easily find out how Members of the European Parliament vote in plenary.
https://howtheyvote.eu
GNU Affero General Public License v3.0
31 stars 2 forks source link

Add CSV export #965

Open tillprochaska opened 1 week ago

tillprochaska commented 1 week ago

This implements a htv export command that exports the database to CSV tables. All relevant database tables are exported as (mostly) normalized CSV tables (e.g. members.csv, votes.csv, member_votes.csv).

The command also creates a metadata file that describes the table schema following the CSV on the Web standard and a markdown Readme that describes the table schema (example). The implementation is similar to the implementation of the OpenAPI specs: Rows of each table are modeled using a Python TypedDict, for example:

class MemberRow(TypedDict):
    """Each row represents a Member of the European Parliament (MEP)."""

    id: int
    """Member ID as used by the [MEP Directory](https://www.europarl.europa.eu/meps/en/home)."""

    first_name: str
    """First name"""

    last_name: str
    """Last name"""

    country_code: str
    """3-letter ISO-3166-1 code"""

    # ...

The type annotations and docstrings are used to auto-generate metadata and documentation.

Publishing

The htv export command creates multiple CSV and metadata files and stores them on the local file system. I’ve configured Caddy to serve these files at https://howtheyvote.eu/export.

My idea was to then create a separate public GitHub repository with a scheduled Actions workflow that downloads the files and commits them to the repository or publishes them as a release.

This way, we don’t have to (manually) handle credentials for downloading the export or committing it to the repository.

Data coverage

The export is currently still missing some data:

Todos