Custom Year searches returning odd results

karenmajewicz commented 1 year ago

What should happen:

A user defines a range of Years using the widget. For example: https://geo.btaa.org/?q=&range%5Bgbl_indexYear_im%5D%5Bbegin%5D=1830&range%5Bgbl_indexYear_im%5D%5Bend%5D=1860&search_field=all_fields
All items returned should be confined to those years as defined in gbl_dateRange_drsim

What is happening:

A user defines a range of Years using the widget. For example: https://geo.btaa.org/?q=&range%5Bgbl_indexYear_im%5D%5Bbegin%5D=1830&range%5Bgbl_indexYear_im%5D%5Bend%5D=1860&search_field=all_fields
Many items that have no year defined are being returned. This is easier to see when refining just to Datatsets. For example: https://geo.btaa.org/?f%5Bgbl_resourceClass_sm%5D%5B%5D=Datasets&q=&range%5Bgbl_indexYear_im%5D%5Bbegin%5D=1830&range%5Bgbl_indexYear_im%5D%5Bend%5D=1860&search_field=all_fields

I am reviewing Solr's scoring algorithm on this search (https://geo.btaa.org/?f%5Bgbl_resourceClass_sm%5D%5B%5D=Datasets&q=&range%5Bgbl_indexYear_im%5D%5Bbegin%5D=1830&range%5Bgbl_indexYear_im%5D%5Bend%5D=1860&search_field=all_fields&score=true&explain=true), but not seeing helpful information.

ewlarson commented 1 year ago

Seeing a lot of Unpermitted parameter: :range in the application logs here. Will keep troubleshooting, but apparently this logic has shifted...

ewlarson commented 1 year ago

Wait... the REAL issue here is that my logic for building dateRange values for Solr was bad. I essentially created a bunch of [* TO *] (infinity) values here for items without a dateRange value, so those hits are just overwhelming results.

I'm updating the code so those documents don't have a dateRange value and I'm reindexing geomgdev.

ewlarson commented 1 year ago

Okay... after adjusting the indexing logic to guard against arrays of blank entries things seem much improved.

Before

After

It also worth pointing out our dateRange search in Solr uses the default "intersects" value, so big ranges [1800-2020] are going to appear in a lot of focused range queries, ex. [2000-2001]. https://solr.apache.org/guide/8_11/working-with-dates.html#more-daterangefield-details

@karenmajewicz Let me know what you think.

ewlarson commented 1 year ago

^ This change and reindex is complete on geodev: https://geodev.btaa.org/?search_field=all_fields&q=&range%5Bgbl_indexYear_im%5D%5Bbegin%5D=2000&range%5Bgbl_indexYear_im%5D%5Bend%5D=2001&commit=Limit

karenmajewicz commented 1 year ago

This seems good and makes sense to me. I think that the Contains and Within might cause frustration for users.

I am curious what the possibilities are though. For example, could we set it up so that a user can specify Intersects or Within? Or, develop an algorithm that preferences items Within at the top?

ewlarson commented 1 year ago

Yeah, we could add some kinda toggle/radio button/control to let the end user choose between the three options: intersects, contains, and within. How they "work" might be hard to explain to an end user. Maybe there's better labels/language we can use?

Also, it might be a bit challenging to implement, as we'd be adding functionality to the blacklight_range_limit plugin and that plugin is another one of those seldomly maintained Blacklight gems without clear consensus regarding its future.

I'm gonna close this issue now. If you do want to add that query option toggle, let's open another ticket for that work.

karenmajewicz commented 1 year ago

Sounds good - We don't need to add that extra functionality now, but it's good to know what is possible.

geobtaa / geoportal