Search within datasets with support for relatedness based on unique IDs

[tlevine]

I heard that you've been thinking about how to search through all of these spreadsheets at once (because ordinary keyword searches aren't great). I've been working a bit on this.

• general thoughts
• special snowflake
• broader things

Wanna talk about this some time? Or write down your thoughts here?

[philipashlock]

Thanks Tom! Any possibility you (or others listening) would be interested in getting paid for helping out with this?

As you probably already know there's already much better search capability available on data.gov than what is really being utilized or promoted. Full solr queries (#62) are possible across all federated sources which includes data catalogs from many cities, states, universities, and intergovernmental organizations. I think the more loosely coupled approach is to allow structured search within wholesale search indexes based on schema.org terms (#327) and to generally align with and promote better interoperability across data catalogs.

Data.gov currently doesn't provide much data hosting itself so there aren't many opportunities to search within datasets. While we will likely be doing more of this, I think the more widely applicable intermediary step is to better promote machine readable descriptions of data structure with really simple conventions like the Simple Data Format (which obviously wouldn't preclude anyone from publishing or promoting more sophisticated schema descriptions as well)

[HobokenMartha]

I just put the link to the job on Facebook. Looks like fun.

On Thu, Mar 6, 2014 at 6:00 PM, Philip Ashlock notifications@github.comwrote:

Thanks Tom! Any possibility you (or others listening) would be interested in getting paid for helping outhttps://www.usajobs.gov/GetJob/ViewDetails/363324500with this?

As you probably already know there's already much better search capability available on data.gov than what is really being utilized or promoted. Full solr queries (#62 https://github.com/GSA/data.gov/issues/62) are possible across all federated sources which includes data catalogs from many cities, states, universities, and intergovernmental organizations. I think the more loosely coupled approach is to allow structured search within wholesale search indexes based on schema.org terms (#327https://github.com/GSA/data.gov/issues/327) and to generally align with and promote better interoperability across data catalogs.

Data.gov currently doesn't provide much data hosting itself so there aren't many opportunities to search within datasets. While we will likely be doing more of this, I think the more widely applicable intermediary step is to better promote machine readable descriptions of data structure with really simple conventions like the Simple Data Formathttp://data.okfn.org/standards/simple-data-format(which obviously wouldn't preclude anyone from publishing or promoting more sophisticated schema descriptions as well)

Reply to this email directly or view it on GitHubhttps://github.com/GSA/data.gov/issues/329#issuecomment-36948741 .

Martha Garvey Writer/Editor/Digital Strategy

---

Web Editorial Director @ Ogilvy Books: My Fat Dog, Hatherleigh Presshttp://www.amazon.com/My-Fat-Dog-Simple-Weight/dp/1578261988/ref=sr_1_1?ie=UTF8&qid=1307200305&sr=8-1 My Fat Cat, Hatherleigh Presshttp://www.amazon.com/My-Fat-Cat-Simple-Weight/dp/157826197X/ref=ntt_at_ep_dpt_1 Yarnbombing Documentary: Looons Create Art or Else http://bit.ly/looons http://www.linkedin.com/in/marthagarvey

[tlevine]

Woah cool, I didn't know you had all this stuff.

Can you pay me to write Simple Data Format files for all of the datasets?

[JeanneHolm]

Confirmed to complete search upgrades in Milestone 2.7.

[tlevine]

Add this.

[philipashlock]

Flagging related discussion on CKAN mailing list - https://lists.okfn.org/pipermail/ckan-dev/2016-March/009800.html