Get access to RI Open Government data

codeisland / ri-lobbying

A site to help people analyze and visualize public lobbying data in Rhode Island

MIT License

3 stars 0 forks source link

Get access to RI Open Government data #1

Open rofreg opened 6 years ago

rofreg commented 6 years ago

In order to do anything with RI Open Government data, first we need access to the data! I've just contacted Terrance from the Secretary of State's office to ask for access to the state data, and I'll update this issue as I learn more.

In the meantime, if we're curious about what kind of data the DB includes, there's a web interface to the data on this page.

rofreg commented 6 years ago

Updates from Terrance:

In the short term, we will get the data via a one-time data dump. This will probably happen within the next week.
In the long term, we should be able to get data updates via SFTP. There is an existing SFTP server where data is uploaded on a regular schedule.
- Access to that SFTP server is restricted based on IP. Terrance says he can add our IP address to the access whitelist, but we'll need a static IP in order to do that.
It is probably ok to host the data dump publicly in this GitHub repo (which will hopefully make development more convenient, hooray!). However, this needs to be verified with the Secretary of State's leadership team first.

paulopperman commented 6 years ago

@rofreg , the Newport Data Project hosts static datasets on github at https://github.com/newportdataportal. We could use that to stage the raw datafiles while building the app. Assuming it's ok with the SoS, that is!

rofreg commented 6 years ago

Just a quick update here: I'm still working on this :) Terrance gave me some AWS login creds to access a backup of the pre-2017 database, but the creds only had permission to view the filename, not to actually download the file. Hopefully we'll get that straightened out shortly. Looks like the data dump will be several GB in size.

dj0nes commented 6 years ago

Thanks for the update @rofreg, keep us posted, and let me know if you need backup :)

rofreg commented 6 years ago

Good timing @dj0nes – we just got the AWS creds working! I now have access to:

A backup of the old database (~3.3 GB)
A PDF of the old database schema (857 pages, wheeeee)
A PDF of the current database schema (231 pages, a little more digestible)

Notably, we do not yet have a backup of the current database (it contains some private info that needs to be stripped first).

Also, I'm not clear if I have permission to share these files publicly, so I'm not adding anything to the repo just yet. If anyone wants personal access, let me know and I'll find a way to send you the files. 📂

dj0nes commented 6 years ago

I'll send a DM for issue #9