search

mysociety / theyworkforyou

Keeping tabs on the UK's parliaments and assemblies
http://www.theyworkforyou.com/
Other
224 stars 50 forks source link

Fetch policy/alignment data from new location. #1752

Closed dracos closed 5 months ago

dracos commented 6 months ago

For https://github.com/mysociety/theyworkforyou/issues/1743 (I ignored the comparison stuff in the JSON for the moment, assume I'll try and understand that at some later point; hope I've picked the right fields). This stops fetching dream MP XML and policy JSON from PW, instead fetching JSON only from twfy-votes that contains the policy votes and alignment information. The import script no longer imports divisions, assuming they've already been imported elsewhere, only policies (if missing), policy/division links, and policy alignments.

Also fixes #1747.

dracos commented 6 months ago

This creates a local JSON file of the policies from the list, rather than change the DB at present. It keeps 'summary' being hard-coded at present.

ajparsons commented 6 months ago

Putting my notes back in the PR for history. Following review comment explains new commits.

All policy information is now loaded through the json2db.pl - running this on new instanance seems to work fine.

Reference to twfy-votes

URL is not hardcoded - but added as an CONF setting 'TWFY_VOTES_URL'.

When we have a semi-permenemnt home we should set a default on this - although still useful to have as a config setting for testing.

Meta infomration

There is a new file /scrapedjson/policies.json that stores the polices data.

This is a reprocessed version of the /policies/commons/active/all.json in twfy-votes.

This specifically stores the descriptions of sets - and the assignment of specific polices to sets. This is referenced when the Policies object is constructed, and which then can be used as the old manual assigments were to display information in views.

Note: it would be useful for testing if twfy-votes let you reference both active+candidate in this URL - now https://github.com/mysociety/twfy-votes/issues/24

Loading policy information

Assume that no new division information is being returned, and so no longer do update checks based on that.

Person policy information is set in the 'personinfo' table as keys - continuing the dreammp setup currently used.

For each policy_id - fetching the twfy-compatitible popolo and reading the info in.

Note: we are no longer storing this json locally as part of parlparse.

Process motions - stash in database the policydivisions information (connection between policy and division) - either adding or update existing.

Note: maybe here the function should be process_policydivision for clarity? This is following the existing 'motionadd, motionupdate' quieries - both of which refer to policydivisions table and could also be renamed. Addressed in follow up commit.

Process alignments - for each person per vote - create or update distance, both_voted, and absent num numbers in the person info table.

Meta note: We're loading policies one at at a time from twfy-votes, mirroing pw API - in principle this could be one big policies file. I don't see a massive reason to do this at this point but could.

Meta note: Still pulling in attendance/rebellion info from Public Whip - which I don't think we're displaying anywhere.

ajparsons commented 6 months ago

And one more - copied how TESTING was used elsewhere.