Make flask driven endpoint for API

[evz]

@fgregg Might be a stupid question but were you intending to back this with #4 ?

[fgregg]

yup, any reason not too?

On Tue, Oct 8, 2013 at 4:38 PM, Eric van Zanten notifications@github.comwrote:

@fgregg https://github.com/fgregg Might be a stupid question but were you intending to back this with #4https://github.com/datamade/elections/issues/4?

— Reply to this email directly or view it on GitHubhttps://github.com/datamade/elections/issues/3#issuecomment-25930002 .

773.888.2718 2231 N. Monticello Ave Chicago, IL 60647

[evz]

How many records do we have so far? Weren't you saying we were in the millions? Might be worth getting into PostgreSQL territory if we want it to perform reasonably. Either way, Flask has a pretty nice SQLAlchemy extension that should make the whole thing pretty painless.

[fgregg]

whatever you think is best, I'll keep the scraping populating the sqlite.db so we can keep the data in the repo

[fgregg]

we have about 4.5 million records, and about 90% done.

[evz]

Nice. Can't wait to get deeper into this.

[evz]

@fgregg The response for the endpoint we talked about (ballots cast by ward for a given election) is looking like this:

{

    "ballots_cast": [
        {
            "count": 10129,
            "ward": 6
        },
        {
            "count": 3558,
            "ward": 15
        },
        {
            "count": 3660,
            "ward": 16
        },
        {
            "count": 6075,
            "ward": 17
        },
        {
            "count": 4528,
            "ward": 20
        },
        {
            "count": 5509,
            "ward": 24
        },
        {
            "count": 7299,
            "ward": 25
        },
        {
            "count": 10079,
            "ward": 36
        },
        {
            "count": 7883,
            "ward": 38
        },
        {
            "count": 14474,
            "ward": 41
        },
        {
            "count": 9656,
            "ward": 43
        },
        {
            "count": 12149,
            "ward": 45
        },
        {
            "count": 9976,
            "ward": 46
        },
        {
            "count": 9716,
            "ward": 50
        }
    ],
    "election": {
        "date": "2011-04-05",
        "id": 1,
        "name": "April, 2011 Supplementary Municipal"
    }

}

I made it a bit more verbose (adding the count and ward keys) because it makes it a bit less of a headache to deal with in Javascript. I also thought it might be handy to echo back the election info.

[fgregg]

Looks great!

I want to be able to make a query to get the last primary (non special) results.

On Tue, Oct 15, 2013 at 9:21 AM, Eric van Zanten notifications@github.comwrote:

@fgregg https://github.com/fgregg The response for the endpoint we talked about (ballots cast by ward for a given election) is looking like this:

{

"ballots_cast": [
    {
        "count": 10129,
        "ward": 6
    },
    {
        "count": 3558,
        "ward": 15
    },
    {
        "count": 3660,
        "ward": 16
    },
    {
        "count": 6075,
        "ward": 17
    },
    {
        "count": 4528,
        "ward": 20
    },
    {
        "count": 5509,
        "ward": 24
    },
    {
        "count": 7299,
        "ward": 25
    },
    {
        "count": 10079,
        "ward": 36
    },
    {
        "count": 7883,
        "ward": 38
    },
    {
        "count": 14474,
        "ward": 41
    },
    {
        "count": 9656,
        "ward": 43
    },
    {
        "count": 12149,
        "ward": 45
    },
    {
        "count": 9976,
        "ward": 46
    },
    {
        "count": 9716,
        "ward": 50
    }
],
"election": {
    "date": "2011-04-05",
    "id": 1,
    "name": "April, 2011 Supplementary Municipal"
}

}

I made it a bit more verbose (adding the count and ward keys) because it makes it a bit less of a headache to deal with in Javascript. I also thought it might be handy to echo back the election info.

— Reply to this email directly or view it on GitHubhttps://github.com/datamade/elections/issues/3#issuecomment-26338198 .

773.888.2718 2231 N. Monticello Ave Chicago, IL 60647

[evz]

@fgregg Any particular key you want to use to get the last primary results? Right now the ballots endpoint uses the name of the election:

http://ec2-54-226-71-129.compute-1.amazonaws.com/ballots/?election=March 2012 Democratic Party Primary Election

but that seems a bit silly. I could do some kind of string/pattern matching perhaps and return all the ones that match. What do you think?

[fgregg]

Well, I'd like to be able to do a kind of sql like thing

show be all the primary elections give them to me sorted return the first one

-Forest

On Thu, Oct 17, 2013 at 11:30 AM, Eric van Zanten notifications@github.comwrote:

@fgregg https://github.com/fgregg Any particular key you want to use to get the last primary results? Right now the ballots endpoint uses the name of the election:

http://ec2-54-226-71-129.compute-1.amazonaws.com/ballots/?election=March2012 Democratic Party Primary Election

but that seems a bit silly. I could do some kind of string/pattern matching perhaps and return all the ones that match. What do you think?

— Reply to this email directly or view it on GitHubhttps://github.com/datamade/elections/issues/3#issuecomment-26524248 .

773.888.2718 2231 N. Monticello Ave Chicago, IL 60647

[fgregg]

Maybe you can just fake all that for now, and make query that looks like it be an alias for the last primary

[evz]

I think I can do it for reals. How do you want the results aggregated? By ward? Do you want registered voters and ballots cast, too?

[fgregg]

Let's start with aggregated by ward.

Yes, I want registered voters and ballots cast.

On Thu, Oct 17, 2013 at 11:41 AM, Eric van Zanten notifications@github.comwrote:

I think I can do it for reals. How do you want the results aggregated? By ward? Do you want registered voters and ballots cast, too?

— Reply to this email directly or view it on GitHubhttps://github.com/datamade/elections/issues/3#issuecomment-26525139 .

773.888.2718 2231 N. Monticello Ave Chicago, IL 60647

[evz]

Hey, I can deploy something like this in a few minutes. Let me know if this meets your expectation:

{

    "ballots_cast": [
        {
            "count": 3687,
            "ward": 1
        },
        {
            "count": 6648,
            "ward": 2
        },
        {
            "count": 5322,
            "ward": 3
        },
        {
            "count": 8135,
            "ward": 4
        },
        {
            "count": 7316,
            "ward": 5
        },
        ... snip ...
    ],
    "results": [
        {
            "votes": 2062,
            "ward": 16,
            "race": "Alderman 16th Ward",
            "option": "JOANN THOMPSON"
        },
        {
            "votes": 3119,
            "ward": 38,
            "race": "Alderman 38th Ward",
            "option": "TOM CARAVETTE"
        },
        {
            "votes": 2871,
            "ward": 17,
            "race": "Alderman 17th Ward",
            "option": "DAVID H. MOORE"
        },
        ... snip ...
    ],
    "election": {
        "date": "2012-03-20",
        "election_type": "Primary",
        "id": 22,
        "name": "March 2012 Democratic Party Primary Election"
    },
    "voters": [
        {
            "count": 26893,
            "ward": 1
        },
        {
            "count": 33810,
            "ward": 2
        },
        {
            "count": 19912,
            "ward": 3
        },
        {
            "count": 24081,
            "ward": 4
        },
        {
            "count": 23721,
            "ward": 5
        },
        ... snip ...
    ]

}

[fgregg]

Looks good. can we filter for just voters, races, or ballots cast

On Thu, Oct 17, 2013 at 12:43 PM, Eric van Zanten notifications@github.comwrote:

Hey, I can deploy something like this in a few minutes. Let me know if this meets your expectation:

{

"ballots_cast": [
    {
        "count": 3687,
        "ward": 1
    },
    {
        "count": 6648,
        "ward": 2
    },
    {
        "count": 5322,
        "ward": 3
    },
    {
        "count": 8135,
        "ward": 4
    },
    {
        "count": 7316,
        "ward": 5
    },
    ... snip ...
],
"results": [
    {
        "votes": 2062,
        "ward": 16,
        "race": "Alderman 16th Ward",
        "option": "JOANN THOMPSON"
    },
    {
        "votes": 3119,
        "ward": 38,
        "race": "Alderman 38th Ward",
        "option": "TOM CARAVETTE"
    },
    {
        "votes": 2871,
        "ward": 17,
        "race": "Alderman 17th Ward",
        "option": "DAVID H. MOORE"
    },
    ... snip ...
],
"election": {
    "date": "2012-03-20",
    "election_type": "Primary",
    "id": 22,
    "name": "March 2012 Democratic Party Primary Election"
},
"voters": [
    {
        "count": 26893,
        "ward": 1
    },
    {
        "count": 33810,
        "ward": 2
    },
    {
        "count": 19912,
        "ward": 3
    },
    {
        "count": 24081,
        "ward": 4
    },
    {
        "count": 23721,
        "ward": 5
    },
    ... snip ...
]

}

— Reply to this email directly or view it on GitHubhttps://github.com/datamade/elections/issues/3#issuecomment-26530930 .

773.888.2718 2231 N. Monticello Ave Chicago, IL 60647

[evz]

yup, I'll get it up and going after I eat something. Lunch is important.

[evz]

So, I made the endpoint 8aa254460460cba39a57ab364e666869e9143272 The server is kinda overwhelmed at the moment by the process of loading in all the data. I might actually just kill that process and finish loading it on my local setup so you and @derekeder can start getting into the data. The only thing that you'll need to remember is that it's not a complete dataset.

[evz]

Oh, forgot to mention how that works. If you call /elections/ it'll just respond with all the available elections including their ID. Then you can get the details for a particular election by calling /elections/?election=22 (that's the ID for the last primary) to get all the details. I realize this isn't entirely ideal but it's working for the moment.

@fgregg I'll get the filters you're requesting setup now.

[evz]

@fgregg As of 40a2804162413d399bea5be83b5c9beddde5d576 I modified the way this thing works, a bit.

First off, I made it so that you should just visit /elections/22/ instead of appending a query string. Secondly, I added a /ballots/ endpoint that works in a similar way. Visit /ballots/22/ to get the ballots cast by ward for the last primary. And the /elections/ endpoint works in the same way.

[evz]

It would be trivial to rewrite this function to aggregate by anything that makes sense to aggregate by. So, precinct, race ... um, I guess that's all that really makes sense. Might endeavor to do that on the next pass.

[fgregg]

Be careful about aggregating over precincts. We know they can change between elections. Wards too, but much more slowly.

[evz]

I guess when I said aggregate by I meant aggregate the vote count for a single election by. Which should be safe given the boundary changes over the years, right? Oh, speaking of which, any forward movement on #6?

[fgregg]

Yes, that should be fine. Still waiting to hear back about GIS files.

On Fri, Oct 18, 2013 at 7:20 AM, Eric van Zanten notifications@github.comwrote:

I guess when I said aggregate by I meant aggregate the vote count for a single election by. Which should be safe given the boundary changes over the years, right? Oh, speaking of which, any forward movement on #6https://github.com/datamade/elections/issues/6 ?

— Reply to this email directly or view it on GitHubhttps://github.com/datamade/elections/issues/3#issuecomment-26591254 .

773.888.2718 2231 N. Monticello Ave Chicago, IL 60647

[fgregg]

But what does it mean to 'aggregate' over a precinct within a particular race?

[evz]

Alright further clarification. I was merely thinking that you could aggregate by one of the three at a time (ward, precinct, race) for a given election. Aggregating by a race would just basically give you the results of the election across the whole city. Instead of giving you counts by ward in the example above, you get them by the other breakdowns. Just an idea.

[fgregg]

I don't really get, but I trust you.

On Fri, Oct 18, 2013 at 7:47 AM, Eric van Zanten notifications@github.comwrote:

Alright further clarification. I was merely thinking that you could aggregate by one of the three at a time (ward, precinct, race) for a given election. Aggregating by a race would just basically give you the results of the election across the whole city. Instead of giving you counts by ward in the example abovehttps://github.com/datamade/elections/issues/3#issuecomment-26530930, you get them by the other breakdowns. Just an idea.

— Reply to this email directly or view it on GitHubhttps://github.com/datamade/elections/issues/3#issuecomment-26592728 .

773.888.2718 2231 N. Monticello Ave Chicago, IL 60647