Posts in the Southend-on-Sea and Portsmouth local elections in 2016 have disappeared

mhl commented 7 years ago

Southend-on-Sea local election 2016

@sjorford spotted this worrying case of Jason Pilley's candidacy in a local election disappearing:

https://candidates.democracyclub.org.uk/person/7633

The candidacy was originally added on the 28th of March 2015 as standing in the Southend-on-Sea local election (St Luke's ward) for the Green Party. However, on the 9th of May 2017 an edit to that profile looks as if removed the St Luke's ward candidacy. That doesn't necessarily mean that that was actually done as part of that change - if something had caused the candidacy to be removed without recording a LoggedAction, it would be recorded as part of the next change made to the candidate's profile - so it might have been deleted between 28th of March 2015 and 9th of May 2017.

Investigating further, it looks like there are now no candidates marked as standing in Southend-on-Sea: https://candidates.democracyclub.org.uk/numbers/election/local.southend-on-sea.2016-05-05/posts

... and worse, the election has no posts associated with it:

In [3]: e = Election.objects.get(name__contains='Southend-on-Sea')

In [5]: e.posts.all()
Out[5]: []

It seems that these Post objects have been genuinely deleted, not just lost their association with the election - I've checked to see if a few posts for that election (from http://www.southend.gov.uk/info/200400/elections_and_registering_to_vote/308/election_results/2 ) exist, but none of those I've checked do. :unamused:

We need to work out what might have caused these posts to be deleted - whether it was a mistaken manual intervention, or a bug causing the post to be deleted.

Portsmouth local election 2016

@jf1 spotted this problem which appears to be very similar. He reported:

It has happened with the change made at 2017-05-17T08:39:04.855812 in https://candidates.democracyclub.org.uk/person/14287/menno-groen

Note the ward is now missing from: https://candidates.democracyclub.org.uk/election/local.portsmouth.2016-05-05/party/party:63/green-party

Unlike: https://webcache.googleusercontent.com/search?q=cache:x4cmIGz1j1cJ:https://candidates.democracyclub.org.uk/election/local.portsmouth.2016-05-05/party/party:63/green-party+&cd=6&hl=en&ct=clnk&gl=uk

Broken links and missing result: https://www.google.com/search?q=site%3Acandidates.democracyclub.org.uk+Eastney+Craneswater https://candidates.democracyclub.org.uk/uk_results/local.portsmouth.2016-05-05

Menno Groen did stand in 2016: https://portsmouth.greenparty.org.uk/elections/2016-local-election.html http://democracy.portsmouth.gov.uk/mgElectionAreaResults.aspx?XXR=0&ID=51&RPID=502524359

As with the previous example, it seems that this Post in the Portsmouth local election from 2016 has been deleted. Going by:

In [21]: for p in Election.objects.get(slug='local.portsmouth.2016-05-05').posts.order_by('base__label'):
    print p
   ....:     
Baffins ward
Central Southsea ward
Charles Dickens ward
Copnor ward
Cosham ward
Drayton and Farlington ward
Fratton ward
Hilsea ward
Milton ward
Nelson ward
Paulsgrove ward
St Jude ward
St Thomas ward

... compared to http://democracy.portsmouth.gov.uk/mgElectionResults.aspx?ID=5&V=1&RPID=502526907

It is the only one from Portsmouth to have been deleted.

Possible other cases

These past local elections have no candidates associated with them, so might also have been subject to the same problem:

Castle Point by-election: St George's ward https://candidates.democracyclub.org.uk/numbers/election/local.castle-point.st-georges.2016-05-05/posts ** Actual results here: https://www.castlepoint.gov.uk/download.cfm?doc=docm93jijm4n2525.pdf&ver=3663
Glasgow by-election: Anderston City ward https://candidates.democracyclub.org.uk/numbers/election/local.glasgow.anderston-city.2016-05-05/posts Candidates: https://www.glasgow.gov.uk/CHttpHandler.ashx?id=33124&p=0 Actual results here: https://www.glasgow.gov.uk/CHttpHandler.ashx?id=33124&p=0
South Derbyshire local election https://candidates.democracyclub.org.uk/numbers/election/local.south-derbyshire.2017-05-04/posts Candidates: https://www.derbyshire.gov.uk/council/elections/local-elections/default.asp Results: https://www.derbyshire.gov.uk/council/elections/local-elections/results/default.asp#9/53.1014/-1.7194

(Of course, given the Portsmouth example above, there might be other elections that have had some but not all posts deleted.)

mhl commented 7 years ago

A couple of other notes:

Post has ForeignKey fields referencing Organization and Area, so if the linked Organization or Area objects in Django's ORM had been deleted then that would cause the Post to be deleted as well.
These posts haven't been directly deleted in the Django admin (the only LogEntry objects relating to posts are accounted for). There's only one LogEntry for deletion of an Organization (so it couldn't account for all of these problems) and none for deletion of an Area. So think I think we can rule out someone deleting these in the admin (although they still might have been deleted in the Django shell, or in the database shell).
I don't think the uk_create_elections_from_every_election deletes objects ever - just creates or updates them, so I think we can rule out an change in EveryElection causing the posts to be deleted.

mhl commented 7 years ago

I've checked the current source code, and I can't see any way by which posts / organizations / areas could end up being deleted.

@symroe and I discussed this today, and decided that the next step should be a regularly run script to detect if any post has been deleted, so we can find out whether this is still happening, and if so localize in time the events that might have caused it. With us both being so busy with the imminent election, this seems like the most important step - the recovery of data can be done from these past local elections can happen after the election.

mhl commented 7 years ago

An update on the posts that have disappeared: because ResultEvent still is from the PopIt era, it stores post IDs and post names as text, so we can use this find some other posts that have been deleted. These are:

      2 Milton Keynes local election no post from: UTW:E05009411 Campbell Park and Old Woughton ward
      2 Milton Keynes local election no post from: UTW:E05009424 Woughton and Fishermead ward
      2 Portsmouth local election no post from: UTW:E05002447 Eastney and Craneswater ward
      2 Southampton local election no post from: UTW:E05002463 Millbrook ward
      2 Southend-on-Sea local election no post from: UTW:E05002212 Belfairs ward
      2 Southend-on-Sea local election no post from: UTW:E05002213 Blenheim Park ward
      2 Southend-on-Sea local election no post from: UTW:E05002214 Chalkwell ward
      2 Southend-on-Sea local election no post from: UTW:E05002215 Eastwood Park ward
      2 Southend-on-Sea local election no post from: UTW:E05002218 Milton ward
      2 Southend-on-Sea local election no post from: UTW:E05002219 Prittlewell ward
      2 Southend-on-Sea local election no post from: UTW:E05002220 St Laurence ward
      2 Southend-on-Sea local election no post from: UTW:E05002221 St Luke's ward
      2 Southend-on-Sea local election no post from: UTW:E05002223 Southchurch ward
      2 Southend-on-Sea local election no post from: UTW:E05002224 Thorpe ward
      2 Southend-on-Sea local election no post from: UTW:E05002225 Victoria ward
      2 Southend-on-Sea local election no post from: UTW:E05002227 West Leigh ward
      6 Warrington local election no post from: UTW:E05011028 Chapelford and Old Hall ward
      6 Warrington local election no post from: UTW:E05011032 Great Sankey North and Whittle Hall ward
      6 Warrington local election no post from: UTW:E05011036 Lymm North and Thelwall ward
      4 Warrington local election no post from: UTW:E05011037 Lymm South ward

sjorford commented 3 years ago

I've been through the SOPN and I believe these are all the candidacies missing from local.warrington.2016-05-05:

Chapelford and Old Hall https://candidates.democracyclub.org.uk/person/8902/allan-keith-bird https://candidates.democracyclub.org.uk/person/8903/phil-hayward https://candidates.democracyclub.org.uk/person/8908/faisal-rashid (merge required) https://candidates.democracyclub.org.uk/person/8911/mike-wass https://candidates.democracyclub.org.uk/person/8912/ian-anthony-wilson

Grappenhall https://candidates.democracyclub.org.uk/person/5635/ryan-bate

Great Sankey North & Whittle Hall https://candidates.democracyclub.org.uk/person/450/stephanie-davies https://candidates.democracyclub.org.uk/person/9008/dan-price

Lymm North & Thelwall https://candidates.democracyclub.org.uk/person/5069/bob-barr

Lymm South https://candidates.democracyclub.org.uk/person/854/james-ashington https://candidates.democracyclub.org.uk/person/9351/ed-gough

Warrington has all-out elections in the 2016-2020 cycle, so many of these candidates will be standing again in the postponed election in May 2021.

VirginiaDooley commented 3 years ago

These are all the candidacies still missing from local.warrington.2016-05-05

https://candidates.democracyclub.org.uk/person/450/stephanie-davies https://candidates.democracyclub.org.uk/person/9008/dan-price https://candidates.democracyclub.org.uk/person/5069/bob-barr https://candidates.democracyclub.org.uk/person/854/james-ashington https://candidates.democracyclub.org.uk/person/9351/ed-gough https://candidates.democracyclub.org.uk/person/7633

DemocracyClub / yournextrepresentative