Search functionality should have more visibility into datasets

[rcheetham]

This is report from @Kistine about trouble finding data sets:

as I went to search for a dataset I know is listed under a data entry, no results showed. Is there any way to improve the search feature so that it will display dataset entries based on details within the entry, not just the title of the entry itself?

LNI has been updating a dataset (currently called 'subcontractors') and it's to be replaced with the updated dataset called 'Permit contractors.' When I search on the ODP page for 'subcontractors,' it displays 0 results. In this case, I know that the subcontractors dataset is filed under the LNI permits dataset. This will significantly affect end-users being able to find relevant data. On the editing end, I usually search before I add a dataset just to make sure it doesn't already exist somewhere to avoid duplicative entries.

[BryanQuigley]

IIRC it's based on the title and the description.

I don't disagree this might be useful generally - but in this specific case having some text describing the Subcontractors resources in that description would be helpful as well (and should make it searchable).

[Kistine]

Thank you Bryan. Just tested by adding 'subcontractors' to the description of the dataset and that did indeed make it appear in the search. From here on out, we'll aim to list out the dataset names that may fall under one overarching dataset in the description.

If you're able to implement broader search functionality in the future that would be great as I don't know when/if we'll be able to go back and do this for all of our previously released datasets.

Best, Kistine

[cid:27ebbc6c-d69b-4f4d-9e7f-77901ed64618]

Kistine Carolan

Senior Program Manager, PHL Open Data Programhttps://www.phila.gov/programs/open-data-program/

Office of Innovation & Technology, City of Philadelphia

o: 1234 Market Street | Philadelphia, PA 19107

e: @.***

---

From: BryanQuigley @.> Sent: Thursday, May 11, 2023 3:10 PM To: azavea/opendataphilly-jkan @.> Cc: Kistine Carolan @.>; Mention @.> Subject: Re: [azavea/opendataphilly-jkan] Extend Search to include Dataset Descriptions and the Title of the Resource records (Issue #57)

External Email Notice. This email comes from outside of City government. Do not click on links or open attachments unless you recognize the sender.

---

IIRC it's based on the title and the description.

I don't disagree this might be useful generally - but in this specific case having some text describing the Subcontractors resources in that description would be helpful as well (and should make it searchable).

— Reply to this email directly, view it on GitHubhttps://github.com/azavea/opendataphilly-jkan/issues/57#issuecomment-1544539549, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ADTMWPF4XED7XZZIYLPBFOLXFU2QRANCNFSM6AAAAAAX6NVLRU. You are receiving this because you were mentioned.Message ID: @.***>

[Kistine]

Is it possible to also check my privileges? Currently, I don't seem to be able to create branches for submitting pull requests.

Best, Kistine

[cid:1860aa79-fe96-4030-a73c-0a11f79d51cf]

Kistine Carolan

Senior Program Manager, PHL Open Data Programhttps://www.phila.gov/programs/open-data-program/

Office of Innovation & Technology, City of Philadelphia

o: 1234 Market Street | Philadelphia, PA 19107

e: @.***

---

From: Kistine Carolan @.> Sent: Thursday, May 11, 2023 4:15 PM To: azavea/opendataphilly-jkan @.>; azavea/opendataphilly-jkan @.> Cc: Mention @.> Subject: Re: [azavea/opendataphilly-jkan] Extend Search to include Dataset Descriptions and the Title of the Resource records (Issue #57)

Thank you Bryan. Just tested by adding 'subcontractors' to the description of the dataset and that did indeed make it appear in the search. From here on out, we'll aim to list out the dataset names that may fall under one overarching dataset in the description.

If you're able to implement broader search functionality in the future that would be great as I don't know when/if we'll be able to go back and do this for all of our previously released datasets.

Best, Kistine

[cid:27ebbc6c-d69b-4f4d-9e7f-77901ed64618]

Kistine Carolan

Senior Program Manager, PHL Open Data Programhttps://www.phila.gov/programs/open-data-program/

Office of Innovation & Technology, City of Philadelphia

o: 1234 Market Street | Philadelphia, PA 19107

e: @.***

---

From: BryanQuigley @.> Sent: Thursday, May 11, 2023 3:10 PM To: azavea/opendataphilly-jkan @.> Cc: Kistine Carolan @.>; Mention @.> Subject: Re: [azavea/opendataphilly-jkan] Extend Search to include Dataset Descriptions and the Title of the Resource records (Issue #57)

External Email Notice. This email comes from outside of City government. Do not click on links or open attachments unless you recognize the sender.

---

IIRC it's based on the title and the description.

I don't disagree this might be useful generally - but in this specific case having some text describing the Subcontractors resources in that description would be helpful as well (and should make it searchable).

— Reply to this email directly, view it on GitHubhttps://github.com/azavea/opendataphilly-jkan/issues/57#issuecomment-1544539549, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ADTMWPF4XED7XZZIYLPBFOLXFU2QRANCNFSM6AAAAAAX6NVLRU. You are receiving this because you were mentioned.Message ID: @.***>

[BryanQuigley]

Improving the search is definitely on the TODO, but I have no idea when we can get to it. The goal is actually to completely switch to a new powerful system.

It really doesn't matter if they are created in your fork or in this repo, - in fact, for anyone else we definitely want them doing the fork/PR/review setup - not needing write access here.

That has me thinking for the case of editing a single dataset by you or @rcheetham @lydiascarf if we should not require a 2nd reviewer (or otherwise lower our requirements to just CI passing). Thoughts? (I can mock something up)

[lydiascarf]

Regarding permissions:

• @Kistine I've invited you to collaborate via your work email, which should fix your permissions issue, but please let me know if it doesn't!
• Also, @Alexander-M-Waldman and @jrmidkiff, your invites expired without being accepted. Would you like me to send them to different email addresses?
• @BryanQuigley I can also handle setting up laxer reviewer requirements for trusted people. I'm already working on a CODEOWNERS file so that someone from the city gets requested if a PR touches a city dataset. I think what you're describing can be covered by that file as well but I'll dig in more today.

Regarding search:

• @BryanQuigley I've got some availability to look into overhauling search. Was there a particular solution/plugin you were looking into? Let's focus this ticket on that search overhaul going forward (changing the title now)

[BryanQuigley]

permissions. That sounds good to me @lydiascarf . Thanks!

As for search here is the JKAN issue: https://github.com/timwis/jkan/issues/225

There aren't specific requirements but I've also been looking at projects like https://stork-search.net/ in addition to lunr.js. I think the first thing is to come up with a list of possible goals:

• Handle more data - this ticket (search by filetype as well?)
• Synonyms
• Highlight found text
• Be able to have search page from home page <- I think this is actually the most important on the list.

Ideally with a similar flow to the current site - I really like how fast it is and how it doesn't break the view. But many of these other search engines might not work like that - they might be separate pages or pop-outs. Is that worth the trade-off?

[jrmidkiff]

Hi Lydia,

Sorry it slipped by me to accept the invitation - can you resend it? My gmail address is fine

Thanks!

On Tue, May 16, 2023 at 8:09 AM Lydia Scarf @.***> wrote:

Regarding permissions:

• @Kistine https://github.com/Kistine I've invited you to collaborate via your work email, which should fix your permissions issue, but please let me know if it doesn't!
• Also, @Alexander-M-Waldman https://github.com/Alexander-M-Waldman and @jrmidkiff https://github.com/jrmidkiff, your invites expired without being accepted. Would you like me to send them to different email addresses?
• @BryanQuigley https://github.com/BryanQuigley I can also handle setting up laxer reviewer requirements for trusted people. I'm already working on a CODEOWNERS file so that someone from the city gets requested if a PR touches a city dataset. I think what you're describing can be covered by that file as well but I'll dig in more today.

Regarding search:

• @BryanQuigley https://github.com/BryanQuigley I've got some availability to look into overhauling search. Was there a particular solution/plugin you were looking into? Let's focus this ticket on that search overhaul going forward (changing the title now)

— Reply to this email directly, view it on GitHub https://github.com/azavea/opendataphilly-jkan/issues/57#issuecomment-1549864123, or unsubscribe https://github.com/notifications/unsubscribe-auth/AGIJKQDIZRZTW7KZERUKXA3XGOKELANCNFSM6AAAAAAX6NVLRU . You are receiving this because you were mentioned.Message ID: @.***>

-- Best,

James Midkiff