avalonmediasystem / avalon

Avalon Media System – Samvera Application

http://www.avalonmediasystem.org/

Apache License 2.0

94 stars 51 forks source link

Block deep pagination #6038

Open joncameron opened 1 month ago

joncameron commented 1 month ago

Description

Continuing investigation work for Solr responses and paging to research real world load and performance issue mitigation. See #5358 for preliminary investigation issue.

Done Looks Like

[ ] Block deep pagination in Avalon code
[ ] Present a reasonable interface to the user so they can contact mcohelp@iu.edu if they legitimately need access to the data
[ ] Would be good to grab one or 2 days' worth of logs and capture just the deep pagination requests so we can verify they look like bot activity.

joncameron commented 1 month ago

Correlation can be found; causation would be a lot harder, especially with our current tools.

Could also be worth waiting until Blacklight 8 and then looking at this after that point. Will removing the deep pagination indicator have an effect?

elynema commented 1 month ago

@joncameron From my notes, I thought we would update this ticket to block deep pagination and present a nice info page to any user about what is happening. We would need to define how many pages / results represent deep pagination, both for facets and for results. We could also include a done looks like of pulling a day's worth (or more) of deep pagination from the logs for scanning by humans so we are comfortable that it looks like bot activity and not humans.

Does this match your take-away from the conversation? We could also run this policy of blocking deep pagination past the MCO Working Group to see if there are any concerns.

joncameron commented 1 month ago

Yes, that sounds right. I'll update the text in this issue to reflect that.