palewire / django-calaccess-raw-data

A Django app to download, extract and load campaign finance and lobbying activity data from the California Secretary of State's CAL-ACCESS database
http://django-calaccess.californiacivicdata.org/
MIT License
64 stars 143 forks source link

Can we get the state to also include the tables that link elections to filers? #62

Closed palewire closed 9 years ago

palewire commented 10 years ago

The official CAL-ACCESS site includes a helpful page that lists all the candidates certified for this November's election. As far as I can tell, there is no way to reproduce this using the raw data dump.

However, the official documentation included with the download suggests there is a set of tables related to elections and filers.

cal-access-about-p148-large

cal-access-about-p149-large

cal-access-about-p150-large

Can we get the state to add these tables to the dump? If they are what I think they are, they seem very useful!

palewire commented 10 years ago

I attempted to send an email to Sean Jensen at the Secretary of State's office but I'm unsure if I targeted the correct address. If you have his email, please pass it along.

palewire commented 10 years ago

Here is the response I received. So much for asking nicely. I guess I have to file a CPRA request.

Hello Ben,

Thank you for your interest in CAL-ACCESS... Sean Jensen shared your inquiry with me. 
 You are correct: while there is a lot of good info, the bulk download does not include the
data from the elections-related tables. Given our current resource constraints, staff cannot
modify the database export to include that other data. 

Best,

Nicole

Nicole Winger  
Deputy Secretary of State, Communications 
California Secretary of State    
(916) 653-6575  
www.sos.ca.gov 
Twitter @CASOSvote
palewire commented 10 years ago

Yesterday I received this response from Constituent.Affairs@sos.ca.gov

Dear Mr. Welsh:

Thank you for contacting the Secretary of State with your Public Records 
Act request for certain elections-related CAL-ACCESS database tables.

We know that other private parties are gathering this data through
 “screen scraping” scripts.  The Secretary of State, however, 
does not have or maintain any such scripts.   

Secretary of State staff had previously researched which elections-related 
tables might be accessed to assist database administrators working with 
the DBExport.  This research resulted in the creation of records that we 
believe may be responsive to your request.  Attached please find a zip 
file and entity relationship diagram (ERD) that are the products of this 
research.  These records were created on an ad hoc basis and are not 
otherwise maintained in the normal course of business or available as a 
part of the DBExport at this time.  Including tables in the DBExport 
will require modification of the code (script) and we do not currently 
have the resources to do so. 

Please note that one of the tables you requested, ELECTION_CANDIDATES, 
is no longer used and has no recent data. In its place is FILER_ELECTIONS, 
which now serves as the link between candidates/propositions and 
their associated contests.  

Keep in mind that the tables provided in this export represent data 
as it existed on October 7, 2014.  That said, the tables are generally 
static by this point in the election cycle.  FILER_ELECTIONS can be 
an exception as candidates/propositions for special or future elections 
may be added at any time; it typically changes just a few times per month.

We included the high-level ERD to assist you in understanding the 
content of the tables and their relationships with other pertinent 
database tables.  The Secretary’s Political Reform Division staff may 
be able to answer questions regarding the data and relationships, but 
cannot provide detailed technical support related to the use of these 
tables or the DBExport in general.  If you have questions about the 
data, you may contact Bruce Maffia at (916) 651-5312 or bruce.maffia@sos.ca.gov.   

We hope this information is helpful to you. If you have questions 
about this or another matter related to the Secretary of State's 
office, please contact us again.

Sincerely,

Constituent Affairs
Secretary of State 

It came with two files attached. First this image.

elections_erd

And then this ZIP file.

palewire commented 9 years ago

So that sounds like a no. It looks like we're have to scrape it.