Data Scraper Poc

[mkalish]

As an alternative to directly uploading data to the portal, data can be loaded via the ckan api. Combining that with AWS lambda, we can write scapers to continuously push new data.

Note: AWS Lambda currently supports Python, Javascript and Java

Tasks

• Write an ETL script to pull your chosen dataset here and convert into a csv that can be uploaded to csv
• Use the CKAN API to create a new dataset
• Use the CKAN API in combination with ETL script to push new resources to the newly created dataset
• Once the process had been sanity checked, commit the complete script (scraping and pushing the data to CKAN) and work with @mkalish to get it deployed to AWS Lambda

[mkalish]

This would be a good choice to get started as a dataset that could be scraped. Its just HTML and I have personal interest in it being available.

http://dccouncil.us/calendar.

I would be interested in grabbing the following fields:

• Date and Time
• Room Number
• Meeting Title
• Meeting Description