My results from "invoke datasource.elections --state il" out of sync with live data

nrrb commented 10 years ago

I'm working off of the il branch. About half an hour ago I changed the direct_link values in the data admin from Candidate Totals to Candidate Totals By Counts. For example, for the 2012-03-20 General Primary election, I changed it from:

http://www.elections.il.gov/Downloads/ElectionInformation/VoteTotals/GP2012Tot.txt

to:

http://www.elections.il.gov/Downloads/ElectionInformation/VoteTotals/GP2012Cty.txt

This change is properly reflected in the metadata API endpoint (http://openelections.net/api/v1/election/?format=json&state__postal=IL):

 {
      "absentee_and_provisional": true,
      "cong_dist_level": false,
      "county_level": true,
      "direct_link": "http://www.elections.il.gov/Downloads/ElectionInformation/VoteTotals/GP2012Cty.txt",
      "end_date": "2012-03-20",
      "gov": false,
      "house": true,
      "id": 168,
      "organization": {
        "city": "Chicago",
        "fec_page": "http://www.fec.gov/pubrec/cfsdd/ildir.htm",
        "gov_agency": true,
        "gov_level": "state",
        "id": 23,
        "name": "Illinois State Board of Elections",
        "resource_uri": "/api/v1/organization/23/",
        "slug": "illinois-state-board-elections",
        "state": "IL",
        "street": "100 W. Randolph,Suite 14-100",
        "url": "http://www.elections.il.gov/"
      },
      "portal_link": "http://www.elections.il.gov/ElectionInformation/DownloadVoteTotals.aspx",
      "precinct_level": false,
      "prez": true,
      "primary_note": "",
      "primary_type": "closed",
      "race_type": "primary",
      "resource_uri": "/api/v1/election/168/",
      "result_type": "certified",
      "senate": true,
      "special": false,
      "start_date": "2012-03-20",
      "state": {
        "name": "Illinois",
        "postal": "IL",
        "resource_uri": "/api/v1/state/IL/"
      },

However, when I run "invoke datasources.elections --state il" I get the following response:

 {u'absentee_and_provisional': True,
  u'cong_dist_level': False,
  u'county_level': True,
  u'direct_link': u'http://www.elections.il.gov/Downloads/ElectionInformation/VoteTotals/GP2012Tot.txt',
  u'end_date': u'2012-03-20',
  u'gov': False,
  u'house': True,
  u'id': 168,
  u'organization': {u'city': u'Chicago',
                    u'fec_page': u'http://www.fec.gov/pubrec/cfsdd/ildir.htm',
                    u'gov_agency': True,
                    u'gov_level': u'state',
                    u'id': 23,
                    u'name': u'Illinois State Board of Elections',
                    u'resource_uri': u'/api/v1/organization/23/',
                    u'slug': u'illinois-state-board-elections',
                    u'state': u'IL',
                    u'street': u'100 W. Randolph,Suite 14-100',
                    u'url': u'http://www.elections.il.gov/'},
  u'portal_link': u'http://www.elections.il.gov/ElectionInformation/DownloadVoteTotals.aspx',
  u'precinct_level': False,
  u'prez': True,
  u'primary_note': u'',
  u'primary_type': u'closed',
  u'race_type': u'primary',
  u'resource_uri': u'/api/v1/election/168/',
  u'result_type': u'certified',
  u'senate': True,
  'slug': u'il-2012-03-20-primary',
  u'special': False,
  u'start_date': u'2012-03-20',
  u'state': {u'name': u'Illinois',
             u'postal': u'IL',
             u'resource_uri': u'/api/v1/state/IL/'},
  u'state_leg': True,
  u'state_leg_level': False,
  u'state_level': True,
  u'state_officers': True,
  u'user_fullname': u'Novikova, Anna'}

It's especially peculiar because other election results from the invoke run do have the correct and updated direct_link value.

I have also been able to get the correct and updated direct_link value if I go into a python shell and call openelex.api.elections directly:

In [1]: from openelex.api import elections as elec_api
In [2]: results = elec_api.find("IL")
In [3]: [r for r in results if r['start_date']=='2012-03-20']
Out[3]: 
[{u'absentee_and_provisional': True,
  u'cong_dist_level': False,
  u'county_level': True,
  u'direct_link': u'http://www.elections.il.gov/Downloads/ElectionInformation/VoteTotals/GP2012Cty.txt',
  u'end_date': u'2012-03-20',
  u'gov': False,
  u'house': True,
  u'id': 168,
  u'organization': {u'city': u'Chicago',
   u'fec_page': u'http://www.fec.gov/pubrec/cfsdd/ildir.htm',
   u'gov_agency': True,
   u'gov_level': u'state',
   u'id': 23,
   u'name': u'Illinois State Board of Elections',
   u'resource_uri': u'/api/v1/organization/23/',
   u'slug': u'illinois-state-board-elections',
   u'state': u'IL',
   u'street': u'100 W. Randolph,Suite 14-100',
   u'url': u'http://www.elections.il.gov/'},
  u'portal_link': u'http://www.elections.il.gov/ElectionInformation/DownloadVoteTotals.aspx',
  u'precinct_level': False,
  u'prez': True,
  u'primary_note': u'',
  u'primary_type': u'closed',
  u'race_type': u'primary',
  u'resource_uri': u'/api/v1/election/168/',
  u'result_type': u'certified',
  u'senate': True,
  u'special': False,
  u'start_date': u'2012-03-20',
  u'state': {u'name': u'Illinois',
   u'postal': u'IL',
   u'resource_uri': u'/api/v1/state/IL/'},
  u'state_leg': True,
  u'state_leg_level': False,
  u'state_level': True,
  u'state_officers': True,
  u'user_fullname': u'Novikova, Anna'}]

zstumgoren commented 10 years ago

Hey Nick, That's gotta be a caching problem. I suspect the page fell out of cache and was re-cached right before you added that link. TTL is currently set to 2 hours, so it should fall out of cache soon. If you still notice it not updating, let me know and I can manually purge after I get back home tomorrow (laid up in Chicago tonight).

Serdar

On Sun, Mar 2, 2014 at 2:20 PM, Nick Bennett notifications@github.comwrote:

I'm working off of the il branch. About half an hour ago I changed the direct_link values in the data admin from Candidate Totals to Candidate Totals By Counts. For example, for the 2012-03-20 General Primary election, I changed it from:

http://www.elections.il.gov/Downloads/ElectionInformation/VoteTotals/GP2012Tot.txt

to:

http://www.elections.il.gov/Downloads/ElectionInformation/VoteTotals/GP2012Cty.txt

This change is properly reflected in the metadata API endpoint ( http://openelections.net/api/v1/election/?format=json&state__postal=IL):

{ "absentee_and_provisional": true, "cong_dist_level": false, "county_level": true, "direct_link": "http://www.elections.il.gov/Downloads/ElectionInformation/VoteTotals/GP2012Cty.txt", "end_date": "2012-03-20", "gov": false, "house": true, "id": 168, "organization": { "city": "Chicago", "fec_page": "http://www.fec.gov/pubrec/cfsdd/ildir.htm", "gov_agency": true, "gov_level": "state", "id": 23, "name": "Illinois State Board of Elections", "resource_uri": "/api/v1/organization/23/", "slug": "illinois-state-board-elections", "state": "IL", "street": "100 W. Randolph,Suite 14-100", "url": "http://www.elections.il.gov/" }, "portal_link": "http://www.elections.il.gov/ElectionInformation/DownloadVoteTotals.aspx", "precinct_level": false, "prez": true, "primary_note": "", "primary_type": "closed", "race_type": "primary", "resource_uri": "/api/v1/election/168/", "result_type": "certified", "senate": true, "special": false, "start_date": "2012-03-20", "state": { "name": "Illinois", "postal": "IL", "resource_uri": "/api/v1/state/IL/" },

However, when I run "invoke datasources.elections --state il" I get the following response:

{u'absentee_and_provisional': True, u'cong_dist_level': False, u'county_level': True, u'direct_link': u'http://www.elections.il.gov/Downloads/ElectionInformation/VoteTotals/GP2012Tot.txt', u'end_date': u'2012-03-20', u'gov': False, u'house': True, u'id': 168, u'organization': {u'city': u'Chicago', u'fec_page': u'http://www.fec.gov/pubrec/cfsdd/ildir.htm', u'gov_agency': True, u'gov_level': u'state', u'id': 23, u'name': u'Illinois State Board of Elections', u'resource_uri': u'/api/v1/organization/23/', u'slug': u'illinois-state-board-elections', u'state': u'IL', u'street': u'100 W. Randolph,Suite 14-100', u'url': u'http://www.elections.il.gov/'}, u'portal_link': u'http://www.elections.il.gov/ElectionInformation/DownloadVoteTotals.aspx', u'precinct_level': False, u'prez': True, u'primary_note': u'', u'primary_type': u'closed', u'race_type': u'primary', u'resource_uri': u'/api/v1/election/168/', u'result_type': u'certified', u'senate': True, 'slug': u'il-2012-03-20-primary', u'special': False, u'start_date': u'2012-03-20', u'state': {u'name': u'Illinois', u'postal': u'IL', u'resource_uri': u'/api/v1/state/IL/'}, u'state_leg': True, u'state_leg_level': False, u'state_level': True, u'state_officers': True, u'user_fullname': u'Novikova, Anna'}

It's especially peculiar because other election results from the invoke run do have the correct and updated direct_link value.

I have also been able to get the correct and updated direct_link value if I go into a python shell and call openelex.api.elections directly:

In [1]: from openelex.api import elections as elec_apiIn [2]: results = elec_api.find("IL")In [3]: [r for r in results if r['start_date']=='2012-03-20']Out[3]: [{u'absentee_and_provisional': True, u'cong_dist_level': False, u'county_level': True, u'direct_link': u'http://www.elections.il.gov/Downloads/ElectionInformation/VoteTotals/GP2012Cty.txt', u'end_date': u'2012-03-20', u'gov': False, u'house': True, u'id': 168, u'organization': {u'city': u'Chicago', u'fec_page': u'http://www.fec.gov/pubrec/cfsdd/ildir.htm', u'gov_agency': True, u'gov_level': u'state', u'id': 23, u'name': u'Illinois State Board of Elections', u'resource_uri': u'/api/v1/organization/23/', u'slug': u'illinois-state-board-elections', u'state': u'IL', u'street': u'100 W. Randolph,Suite 14-100', u'url': u'http://www.elections.il.gov/'}, u'portal_link': u'http://www.elections.il.gov/ElectionInformation/DownloadVoteTotals.aspx', u'precinct_level': False, u'prez': True, u'primary_note': u'', u'primary_type': u'closed', u'race_type': u'primary', u'resource_uri': u'/api/v1/election/168/', u'result_type': u'certified', u'senate': True, u'special': False, u'start_date': u'2012-03-20', u'state': {u'name': u'Illinois', u'postal': u'IL', u'resource_uri': u'/api/v1/state/IL/'}, u'state_leg': True, u'state_leg_level': False, u'state_level': True, u'state_officers': True, u'user_fullname': u'Novikova, Anna'}]

Reply to this email directly or view it on GitHubhttps://github.com/openelections/core/issues/79 .

ghing commented 10 years ago

I suspect caching as well, I just reran the command and got:

 {u'absentee_and_provisional': True,
  u'cong_dist_level': False,
  u'county_level': True,
  u'direct_link': u'http://www.elections.il.gov/Downloads/ElectionInformation/VoteTotals/GP2012Cty.txt',
  u'end_date': u'2012-03-20',
  u'gov': False,
  u'house': True,
  u'id': 168,
  u'organization': {u'city': u'Chicago',
                    u'fec_page': u'http://www.fec.gov/pubrec/cfsdd/ildir.htm',
                    u'gov_agency': True,
                    u'gov_level': u'state',
                    u'id': 23,
                    u'name': u'Illinois State Board of Elections',
                    u'resource_uri': u'/api/v1/organization/23/',
                    u'slug': u'illinois-state-board-elections',
                    u'state': u'IL',
                    u'street': u'100 W. Randolph,Suite 14-100',
                    u'url': u'http://www.elections.il.gov/'},
  u'portal_link': u'http://www.elections.il.gov/ElectionInformation/DownloadVoteTotals.aspx',
  u'precinct_level': False,
  u'prez': True,
  u'primary_note': u'',
  u'primary_type': u'closed',
  u'race_type': u'primary',
  u'resource_uri': u'/api/v1/election/168/',
  u'result_type': u'certified',
  u'senate': True,
  'slug': u'il-2012-03-20-primary',
  u'special': False,
  u'start_date': u'2012-03-20',
...

That is, for the March 20, 2012 primary, the direct_link field reflects @tothebeat's correction to http://www.elections.il.gov/Downloads/ElectionInformation/VoteTotals/GP2012Cty.txt. I'm going to close this for now, but please feel free to reopen if you're still finding that you're getting stale values for some reason.

openelections / openelections-core

My results from "invoke datasource.elections --state il" out of sync with live data #79