Create auto-import of NGP Van turfs (riders data) to the CPV database

[PamelaJohn]

(This request was made at the 2018, March 18th CPV upgrade planning and strategy session.)

Create auto-import of NGP Van turfs which identify riders' data to the CPV database, including:

• Rider's Name
• Rider's Address
• Rider's Zip Code
• Rider's Phone Number
• Rider's Polling Place Address + Zip Code

@acqsysandrea please let me know if I've missed anything data to import from turfs.

[aaghevli]

Digging into this a bit and am a bit confused. NGP Van does have API access capabilities, but I am not clear what the "turfs" refers to.

Here is the endpoint to fetch all the contacts in a single NGP Van database: https://developers.ngpvan.com/van-api#people

We also require an API-KEY to do this work. Is this something that can be provided?

[jkbits1]

@aaghevli thanks for taking a look at this 👍 😄 much appreciated. Good point about "turfs", it has a clear meaning for election organisers @PamelaJohn @acqsysandrea can you help to clarify this, please?

@PamelaJohn @acqsysandrea @andrewcuan @aaghevli when this issue has been discussed, I've wondered a bit about the context it fits into. If it's an occasional task, it might be easiest to have a standalone task that is provided with a text file with the data, and inserts it into the db. In which case, we run the standalone app manually (or as a background task, etc.) as and when we create the revised text file. Or does this data changes frequently and in small chunks, so we need a method that grabs the data on a frequent basis?

[acqsysandrea]

Attached is a sample "Turf" file. This is what is generated when we look at everyone who has requested a Ride to the Polls. It produces a map which I then organize by Polling location so we get a listing of everyone that needs to go to a particular polling location.

While we can certainly generate a list of everyone who needs a ride, I think it is much more valuable to organize the list by where people are going rather than who is simply going. This will be especially useful when we are using church buses and vans; we can perhaps come up with a route that the drivers can take that will be most efficient rather than randomly driving around and picking people up.

Andrea

On Sat, Apr 28, 2018 at 3:58 PM, Jon Kelly notifications@github.com wrote:

@aaghevli https://github.com/aaghevli thanks for taking a look at this 👍 😄 much appreciated. Good point about "turfs", it has a clear meaning for election organisers @PamelaJohn https://github.com/PamelaJohn @acqsysandrea https://github.com/acqsysandrea can you help to clarify this, please?

@PamelaJohn https://github.com/PamelaJohn @acqsysandrea https://github.com/acqsysandrea @andrewcuan https://github.com/andrewcuan @aaghevli https://github.com/aaghevli when this issue has been discussed, I've wondered a bit about the context it fits into. If it's an occasional task, it might be easiest to have a standalone task that is provided with a text file with the data, and inserts it into the db. In which case, we run the standalone app manually (or as a background task, etc.) as and when we create the revised text file. Or does this data changes frequently and in small chunks, so we need a method that grabs the data on a frequent basis?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/voteamerica/voteamerica.github.io/issues/344#issuecomment-385201757, or mute the thread https://github.com/notifications/unsubscribe-auth/AgEfTn2oX5Qv8CwN2QTb8kWhsIy4rF72ks5ttMnTgaJpZM4S6YC8 .

-- Andrea Miller Executive Director, People Demanding Action 443-878-2071 www.peopledemandingaction.org

[aaghevli]

@acqsysandrea and I synced offline and she sent me the file, which is in PDF.

What generates this file? If this is from NGP Van CRM [1] then I wonder if they have a more computer friendly format like CSV/JSON/XML since PDF is not ideal and can lead to brittle import logic. Otherwise, the direct API access is still an option.

@jkbits1, I am happy to help with either the stand alone task and/or polling process.

[1] - https://blog.ngpvan.com/product/introducing-ngp-vans-new-turf-management-tool

[acqsysandrea]

I can certainly generate this is a .CSV file but there would be a lot of manual copying and pasting. VAN does not let you export data, it is a permission based process that is hit and miss. The .PDF files or lists are native.

I can run the list and then copy and past the names into Excel. I would then need to run the turf to get the polling locations and copy and paste those in.

It is doable.

Andrea Miller

On Sat, Apr 28, 2018, 9:41 PM aaghevli notifications@github.com wrote:

@acqsysandrea https://github.com/acqsysandrea and I synced offline and she sent me the file, which is in PDF.

What generates this file? If this is from NGP Van CRM [1] then I wonder if they have a more computer friendly format like CSV/JSON/XML since PDF is not ideal and can lead to brittle import logic. Otherwise, the direct API access is still an option.

@jkbits1 https://github.com/jkbits1, I am happy to help with either the stand alone task and/or polling process.

[1] - https://blog.ngpvan.com/product/introducing-ngp-vans-new-turf-management-tool

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/voteamerica/voteamerica.github.io/issues/344#issuecomment-385218841, or mute the thread https://github.com/notifications/unsubscribe-auth/AgEfTnwxsPfPqM8KnD1wQCFfsOZXsFTJks5ttRowgaJpZM4S6YC8 .

[aaghevli]

Thanks @acqsysandrea! I don't think copy and pasting is worth your time. There are two solutions:

• parse the PDF and import into the database
• directly access the VAN data via their API

I can see how far I get with the parsing of the PDF, but I would suggest thinking through the API access as it would reduce all manual work on your end entirely.

[acqsysandrea]

There may be a third option. If we load the polling locations and then match those with the precincts. We should be able to get that data through ESRI. Then I could just run the list and include voting precinct.

Andrea Miller

On Sat, Apr 28, 2018, 10:11 PM aaghevli notifications@github.com wrote:

Thanks @acqsysandrea https://github.com/acqsysandrea! I don't think copy and pasting is worth your time. There are two solutions:

• parse the PDF and import into the database
• directly access the VAN data via their API

I can see how far I get with the parsing of the PDF, but I would suggest thinking through the API access as it would reduce all manual work on your end entirely.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/voteamerica/voteamerica.github.io/issues/344#issuecomment-385220052, or mute the thread https://github.com/notifications/unsubscribe-auth/AgEfTr0WYW5U5d_9sV-sjBwQR73gcEexks5ttSFLgaJpZM4S6YC8 .

[aaghevli]

I have been able to extract the data from the PDF. I still think its really brittle and could break easily, but the output is as follows. The zip and state of the polling location is not in the polling location which makes that output incomplete:

{
    "name": "Harvey, Larry S",
    "phone_number": "(123) 123-1234",
    "address_lines": [
      "1234 Young Larry Rd NW",
      "Roanoke, VA 24017"
    ],
    "polling_location": "NINTH BAPTIST CHURCH, 1111 Z St"
  }

It is unlikely that I'll have time to finish the push to the database this weekend, but hopefully I can post a PR for at least the parsing.

[acqsysandrea]

Don't worry about posting since the primarybixn'to up ntil June. We have a working concept.

If I send you a .CSV then you're in good shape. I will work on getting precinct added to the output.

Andrea Miller

On Sun, Apr 29, 2018, 8:00 PM aaghevli notifications@github.com wrote:

I have been able to extract the data from the PDF. I still think its really brittle and could break easily, but the output is as follows. The zip and state of the polling location is not in the polling location which makes that output incomplete:

{ "name": "Harvey, Larry S", "phone_number": "(123) 123-1234", "address_lines": [ "1234 Young Larry Rd NW", "Roanoke, VA 24017" ], "polling_location": "NINTH BAPTIST CHURCH, 1111 Z St" }

It is unlikely that I'll have time to finish the push to the database, but hopefully I can post a PR for at least the parsing.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/voteamerica/voteamerica.github.io/issues/344#issuecomment-385291998, or mute the thread https://github.com/notifications/unsubscribe-auth/AgEfTrPAFC31Ygtp6JlBcrUje3p3w82Tks5ttlQ3gaJpZM4S6YC8 .

[aaghevli]

Thanks @acqsysandrea, the extraction into JSON from the PDF should avoid the need for you to manually extract the CSV values by hand.

[acqsysandrea]

When I generate the list, I can just do a quick swipe and copy the names. There will never be more than a few hundred in each state.

Andrea Miller

On Sun, Apr 29, 2018, 8:07 PM aaghevli notifications@github.com wrote:

Thanks @acqsysandrea https://github.com/acqsysandrea, the extraction into JSON from the PDF should avoid the need for you to manually extract the CSV values by hand.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/voteamerica/voteamerica.github.io/issues/344#issuecomment-385292562, or mute the thread https://github.com/notifications/unsubscribe-auth/AgEfTuRgD2yTmIVvKLDuK84_9lcdI47yks5ttlWwgaJpZM4S6YC8 .

[jkbits1]

@PamelaJohn changed the title for clarity 🙂

[jkbits1]

@stephenscapelliti think this is ready to close - do you want to archive the conversation?