Enable search by bounding box

mradamcox commented 1 week ago

This is the beginning of integrating the "search this area" functionality, based on the current view of the map's bounding box. The pieces of this are:

bbox param, which already updates whenever the map is moved
bboxSearch param, which is already linked to the "Search this area" checkbox
the locn_geometry is the field in Solr that we run an Intersect operation against, and its alias is geometry

The desired behavior is that if bboxSearch is true, an extra fq clause is generated based on the bbox param, and this clause is added to the query filters. This much is completed already on this PR.

@pengyin, what still needs to be done is setting up the proper triggers.

Whenever bboxSearch is changed (by clicking the "Search this area" box) the query needs to be re-run, using the content in the URL params.
Whenever bbox param is changed (by map move) the query should also be re-run, using the content within the URL params.

I'm just not familiar enough with where/how the Solr queries are initiated to be able to add this quickly enough. FWIW It seems like conceptually at least, we could have a single function call to Solr that is somewhere within the ParameterList functions (edit: Looking at the code, I'm thinking this would be like moving the handleSearch apparatus up the chain to only be called somewhere within ParameterList). I.e. Solr's /select endpoint is only ever called in one place within the code base, and it is always only populated with query information that is directly drawn from URL params. Then, any updates to those params, which could come from any downstream components, like checkboxes, etc, would automatically trigger an update on the search results. I may be missing something, but as far as I can tell the /select endpoint is only ever passed information that also exists in our URL params.

For reference on this geospatial query, pasting the following text into the fq box in the Solr web admin Query interface, will properly return one item (or more, soon, when we add more Geometry values to the records)

locn_geometry:"Intersects(ENVELOPE (-106.498,-96.315,40.879,35.504))"

To test completion of this

New requests to Solr should be apparent whenever the box is checked, and the map is moved.
Zooming into an area far outside of the US should give 0 results when the filter is enabled.

I updated the test methods in my comments below

netlify[bot] commented 1 week ago

Deploy Preview for cheerful-treacle-913a24 ready!

Name	Link
Latest commit	4768e875f342a6bc6bc89988c5087423adae66a4
Latest deploy log	https://app.netlify.com/sites/cheerful-treacle-913a24/deploys/673ced408b5bf100083df976
Deploy Preview	https://deploy-preview-385--cheerful-treacle-913a24.netlify.app
Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site configuration.

pengyin-shan commented 1 week ago

Thanks @mradamcox, I'll take this PR and continue working on the rest. Could you provide an example of a /select URL that I can refer to for the complete URL format? For instance, if the URL is http://localhost:3000/search?bbox=-125.788,-40.118,-59.301,-4.974&layers=state&bboxSearch=true, what would the corresponding Solr query look like?

Also just for your reference in the future, updating the result list would typically involve a combination of:

    const urlParams = GetAllParams();
    const filterQueries = reGetFilterQueries(urlParams); // update this if necessary
    props.setQuery(userInput);
    urlParams.setSubject(null); // Optional, here’s an example of how the subject parameter can be removed.
    props.handleSearch(urlParams, userInput, filterQueries); // This method is in discoveryApp.tsx, where it takes URL, query term and filters, then fetch Solr and set the 'fetchResults' on the home page. We then pass these results to the sub-components.

Unfortunately I haven't been able to decouple the handleSearch method further, so it is still in the discoveryApp.tsx file. Hopefully there will be time in the future to optimize this further and make it more manageable.

mradamcox commented 1 week ago

Disrgarding the specific coordinates, here is an example that works on our Solr instance:

/select?q=*&fq=(gbl_suppressed_b:false)&rows=1000&fq=locn_geometry:"Intersects(ENVELOPE (-106.498,-96.315,40.879,35.504))"

pengyin-shan commented 5 days ago

I have completed this PR, and it implements the bboxSearch feature to enable searches when the bbox parameter is present. If bboxSearch is not enabled, even if the bbox parameter changes (e.g., when the map is moved), no search will be triggered.

Additionally, I improved the search mechanisms by validating the URL and reducing loading times for queries involving multiple nuqs parameters. I noticed that the nuqs parameter collection introduces a time gap, which can cause searches to trigger before all parameters are collected. To address this, I added validation methods. After completing the prioritized features, I plan to optimize this further, as the time gap may increase with more parameters in the URL, potentially impacting the user experience.

How to Test

Navigate to the Search Page: Move around the map. No search should be triggered since bboxSearch is not enabled. However, the 'Clear All Filters' button should remain active to allow users to clear the map movements.
Click 'Search this Area': This should trigger a search:
Move the Map Again: After enabling bboxSearch, searches should now be triggered based on map movements (data testing not available at the moment).
Click 'Clear All Filters': Clearing all filters removes both bbox and bboxSearch parameters, returning all results. Optionally, the bbox parameter can remain in the URL to persist as a default value, allowing users to clear filters while keeping the bbox context intact.

mradamcox commented 4 days ago

@pengyin-shan As discussed, one thing to fix after this merge is the encoding issue we encountered on Firefox. I have confirmed that it does work on Chrome, so this is a minor issue but something to make a ticket for and address next month.

With regard to the other issue of the spatial record being returned even when the bounding box shouldn't intersect it: This is a problem with the geometry in the metadata record, and I'll have to come back to it. For reference, the current field value is a WKT-formatted rough boundary of the US (including AK and HI) that looks like this: (it is very low resolution in order to reduce the number of characters needed to store it).

I split this up in to each polygon separately and indexed the record into Solr, but it still always returned, no matter what the bounding box is.

However, when I replaced the geometry field with a single point POINT (-90 30), a point near New Orleans, the bounding box search did work as intended: the record was only returned when the map view included that point.

So it seems like there is something about the operation on Solr's side when it is run against polygons or multipolygons, OR there is a problem with all three of the individual polygons above (perhaps right-hand rule isn't followed?) so I'll have to follow up on that in the coming days.

pengyin-shan commented 4 days ago

@pengyin-shan As discussed, one thing to fix after this merge is the encoding issue we encountered on Firefox. I have confirmed that it does work on Chrome, so this is a minor issue but something to make a ticket for and address next month.

With regard to the other issue of the spatial record being returned even when the bounding box shouldn't intersect it: This is a problem with the geometry in the metadata record, and I'll have to come back to it. For reference, the current field value is a WKT-formatted rough boundary of the US (including AK and HI) that looks like this: (it is very low resolution in order to reduce the number of characters needed to store it).

I split this up in to each polygon separately and indexed the record into Solr, but it still always returned, no matter what the bounding box is.

However, when I replaced the geometry field with a single point POINT (-90 30), a point near New Orleans, the bounding box search did work as intended: the record was only returned when the map view included that point.

So it seems like there is something about the operation on Solr's side when it is run against polygons or multipolygons, OR there is a problem with all three of the individual polygons above (perhaps right-hand rule isn't followed?) so I'll have to follow up on that in the coming days.

@mradamcox cool, I think there might be something w/ the encoding (probably " or :, I will test and fix it in a separate PR). For area search, I found https://solr-user.lucene.apache.narkive.com/2KWVFfTD/polygon-search-query-working-but-not-multipolygon (its multi-polygon not work, just polygon) probably we can refer in the future?

mradamcox commented 4 days ago

@pengyin-shan Ok, I realized: this is not an actual issue with the app code, it's just what happens in the console! So, no real problem here.

pengyin-shan commented 4 days ago

@pengyin-shan Ok, I realized: this is not an actual issue with the app code, it's just what happens in the console! So, no real problem here.

Nice! then let's keep the current encoding mechanism

healthyregions / SDOHPlace