search

openaustralia / morph

Take the hassle out of web scraping
https://morph.io
GNU Affero General Public License v3.0
461 stars 74 forks source link

Search API #888

Open drzax opened 9 years ago

drzax commented 9 years ago

Provide an (unauthenticated) API endpoint for searching available datasets.

equivalentideas commented 9 years ago

Thanks @drzax . What's the use-case / reason why you'd like this?

drzax commented 9 years ago

I'm planning a system which automatically consumes new and/or updated datasets from CKAN instances based on specified search criteria. It's likely to be useful to do the same with morph.io datasets. As it stands I would need to hard-code a defined list of datasets.

On Tue, Aug 11, 2015 at 5:04 PM, Luke Bacon notifications@github.com wrote:

Thanks @drzax . What's the use-case / reason why you'd like this?

Reply to this email directly or view it on GitHub: https://github.com/openaustralia/morph/issues/888#issuecomment-129735107

henare commented 8 years ago

I'm not sure what it looks like but maybe morph.io could return a similar result to CKAN. This way tools build to search CKAN could automatically work with morph.io - neat.

jesserobertson commented 8 years ago

All I'd need would be an endpoint at http://morph.io/api/3/action/package_search&q=<query>&rows=<n_rows>. Should return some JSON where you have a 'results',key mapping to a list of datasets, where each dataset is a dictionary with 'description', 'url' and 'format'.

The other thing that would be cool to include is a recently changed list: CKAN API puts that at api/3/action/recently_changed_packages_activity_list.

jesserobertson commented 8 years ago

FYI: here's what my scraper bot is doing: https://github.com/jesserobertson/databot/blob/master/app/bot.py

jesserobertson commented 8 years ago

...and some queries from data.gov.au for reference:

https://data.gov.au/api/3/action/package_search?rows=5&q=walking

https://data.gov.au/api/3/action/recently_changed_packages_activity_list