mysociety / yournextrepresentative

A website for crowd-sourcing structured election candidate data
https://candidates.democracyclub.org.uk/
GNU Affero General Public License v3.0
56 stars 21 forks source link

UK: inconsistent data: memberships associated with an election and a post, where the post is not associated with that election #972

Closed mhl closed 7 years ago

mhl commented 7 years ago

The error emails we've been getting from candidates.dc are an exciting new type of data inconsistency :( A MembershipExtra has a election, but at the moment that's not constrained to be an election that the post is linked with, via the PostExtraElection join model:

In [1]: for me in MembershipExtra.objects.select_related('base__post__extra', 'election', 'base__person'):
    if not PostExtraElection.objects.filter(postextra=me.base.post.extra, election=me.election).exists():
        print "There was a membership for", me.base.person, "with post", me.base.post, "and election", me.election.slug, "but there's no PostExtraElection linking them"
   ...:         
There was a membership for Ed Buxton with post Churchdown and election local.gloucestershire.churchdown.2016-05-05 but there's no PostExtraElection linking them
There was a membership for Graham John Bocking with post Churchdown and election local.gloucestershire.churchdown.2016-05-05 but there's no PostExtraElection linking them
There was a membership for Jack Lawrence Williams with post Churchdown and election local.gloucestershire.churchdown.2016-05-05 but there's no PostExtraElection linking them
There was a membership for Joshua Baker with post Amesbury East and election local.wiltshire.amesbury-east.2016-05-05 but there's no PostExtraElection linking them
There was a membership for Jamie Paul Capp with post Amesbury East and election local.wiltshire.amesbury-east.2016-05-05 but there's no PostExtraElection linking them
There was a membership for Andy Derry with post Amesbury East and election local.wiltshire.amesbury-east.2016-05-05 but there's no PostExtraElection linking them
There was a membership for Steve McAuliffe with post Amesbury East and election local.wiltshire.amesbury-east.2016-05-05 but there's no PostExtraElection linking them
There was a membership for Les Webster with post Amesbury East and election local.wiltshire.amesbury-east.2016-05-05 but there's no PostExtraElection linking them
There was a membership for Robert Alexander Yuill with post Amesbury East and election local.wiltshire.amesbury-east.2016-05-05 but there's no PostExtraElection linking them
There was a membership for John Burns with post Haverhill Cangle and election local.suffolk.haverhill-cangle.2016-05-05 but there's no PostExtraElection linking them
There was a membership for Maureen Gwendoline Byrne with post Haverhill Cangle and election local.suffolk.haverhill-cangle.2016-05-05 but there's no PostExtraElection linking them
There was a membership for Tim Marks with post Haverhill Cangle and election local.suffolk.haverhill-cangle.2016-05-05 but there's no PostExtraElection linking them
There was a membership for Ken Rolph with post Haverhill Cangle and election local.suffolk.haverhill-cangle.2016-05-05 but there's no PostExtraElection linking them
There was a membership for David George Brookes with post Uttoxeter Town and election local.staffordshire.uttoxeter-town.2016-05-05 but there's no PostExtraElection linking them
There was a membership for Julian D'arcy Lee with post Uttoxeter Town and election local.staffordshire.uttoxeter-town.2016-05-05 but there's no PostExtraElection linking them
There was a membership for John McKiernan with post Uttoxeter Town and election local.staffordshire.uttoxeter-town.2016-05-05 but there's no PostExtraElection linking them
There was a membership for Ashley Wakeling with post Mynyddbach and election local.swansea.mynyddbach.2016-05-05 but there's no PostExtraElection linking them
There was a membership for Charlene Anika Webster with post Mynyddbach and election local.swansea.mynyddbach.2016-05-05 but there's no PostExtraElection linking them
There was a membership for Colin Brian Boyd with post Bixley and election local.suffolk.bixley.2016-05-05 but there's no PostExtraElection linking them
There was a membership for Tony Gould with post Bixley and election local.suffolk.bixley.2016-05-05 but there's no PostExtraElection linking them
There was a membership for Paul Bissett with post Edenthorpe & Kirk Sandall and election local.doncaster.edenthorpe-and-kirk-sandall.2016-05-05 but there's no PostExtraElection linking them
There was a membership for Martin Damian Greenhalgh with post Edenthorpe & Kirk Sandall and election local.doncaster.edenthorpe-and-kirk-sandall.2016-05-05 but there's no PostExtraElection linking them
There was a membership for Stephen Neil Porter with post Edenthorpe & Kirk Sandall and election local.doncaster.edenthorpe-and-kirk-sandall.2016-05-05 but there's no PostExtraElection linking them
There was a membership for Andrea Robinson with post Edenthorpe & Kirk Sandall and election local.doncaster.edenthorpe-and-kirk-sandall.2016-05-05 but there's no PostExtraElection linking them
There was a membership for Philip Charles with post Anderston/City and election local.glasgow.anderston-city.2016-05-05 but there's no PostExtraElection linking them
There was a membership for Karen King with post Anderston/City and election local.glasgow.anderston-city.2016-05-05 but there's no PostExtraElection linking them
There was a membership for Steven Livingston with post Anderston/City and election local.glasgow.anderston-city.2016-05-05 but there's no PostExtraElection linking them
There was a membership for Christy Mearns with post Anderston/City and election local.glasgow.anderston-city.2016-05-05 but there's no PostExtraElection linking them
There was a membership for Angus Millar with post Anderston/City and election local.glasgow.anderston-city.2016-05-05 but there's no PostExtraElection linking them
There was a membership for Ryan Ross with post Anderston/City and election local.glasgow.anderston-city.2016-05-05 but there's no PostExtraElection linking them
There was a membership for Shan Couch with post Mynyddbach and election local.swansea.mynyddbach.2016-05-05 but there's no PostExtraElection linking them
There was a membership for Mike Lewis with post Mynyddbach and election local.swansea.mynyddbach.2016-05-05 but there's no PostExtraElection linking them
There was a membership for Patrick Morgan with post Mynyddbach and election local.swansea.mynyddbach.2016-05-05 but there's no PostExtraElection linking them
There was a membership for Noel West with post Mynyddbach and election local.swansea.mynyddbach.2016-05-05 but there's no PostExtraElection linking them
There was a membership for Peter Alfred Appleford with post Staines South and Ashford West and election local.surrey.staines-south-and-ashford-west.2016-05-05 but there's no PostExtraElection linking them
There was a membership for Christopher Leslie Bateson with post Staines South and Ashford West and election local.surrey.staines-south-and-ashford-west.2016-05-05 but there's no PostExtraElection linking them
There was a membership for Matthew David Clarke with post Staines South and Ashford West and election local.surrey.staines-south-and-ashford-west.2016-05-05 but there's no PostExtraElection linking them
There was a membership for Andrew McLuskey with post Staines South and Ashford West and election local.surrey.staines-south-and-ashford-west.2016-05-05 but there's no PostExtraElection linking them
There was a membership for Iain Lochiel Raymond with post Staines South and Ashford West and election local.surrey.staines-south-and-ashford-west.2016-05-05 but there's no PostExtraElection linking them
There was a membership for Denise Turner-Stewart with post Staines South and Ashford West and election local.surrey.staines-south-and-ashford-west.2016-05-05 but there's no PostExtraElection linking them
There was a membership for Rob Bridgeman with post Bixley and election local.suffolk.bixley.2016-05-05 but there's no PostExtraElection linking them
There was a membership for Paul Martin West with post Bixley and election local.suffolk.bixley.2016-05-05 but there's no PostExtraElection linking them

Ideally there should be constraints to stop this happening, and the check could be added to: https://github.com/DemocracyClub/yournextrepresentative/blob/9d3d7287c98c2be76b8981039d408003950107e8/candidates/models/constraints.py

And these individual cases should be fixed.

mhl commented 7 years ago

The elections found via that method are:

{u'local.doncaster.edenthorpe-and-kirk-sandall.2016-05-05',
 u'local.glasgow.anderston-city.2016-05-05',
 u'local.gloucestershire.churchdown.2016-05-05',
 u'local.staffordshire.uttoxeter-town.2016-05-05',
 u'local.suffolk.bixley.2016-05-05',
 u'local.suffolk.haverhill-cangle.2016-05-05',
 u'local.surrey.staines-south-and-ashford-west.2016-05-05',
 u'local.swansea.mynyddbach.2016-05-05',
 u'local.wiltshire.amesbury-east.2016-05-05'}

This is nearly the same list as you get from:

In [63]: [e.slug for e in Election.objects.filter(posts__isnull=True)]
Out[63]: 
[u'local.wiltshire.amesbury-east.2016-05-05',
 u'local.suffolk.haverhill-cangle.2016-05-05',
 u'local.surrey.staines-south-and-ashford-west.2016-05-05',
 u'local.swansea.mynyddbach.2016-05-05',
 u'local.doncaster.edenthorpe-and-kirk-sandall.2016-05-05',
 u'local.castle-point.st-georges.2016-05-05',
 u'local.glasgow.anderston-city.2016-05-05',
 u'local.staffordshire.uttoxeter-town.2016-05-05',
 u'local.suffolk.bixley.2016-05-05',
 u'local.gloucestershire.churchdown.2016-05-05']

... except that that also includes u'local.castle-point.st-georges.2016-05-05'

mhl commented 7 years ago

This has been fixed now (see https://github.com/DemocracyClub/yournextrepresentative/issues/51 )