codice / ddf

DDF Distributed Data Framework - an open source, modular integration framework.
http://ddf.codice.org
Other
132 stars 181 forks source link

[2.26.x] Support ID query paging #6737

Closed derekwilhelm closed 1 year ago

derekwilhelm commented 1 year ago

What does this PR do?

When issuing a query for IDs that exceed the requested page size, solr returns the full list of results rather than the requested page size. This causes the catalog framework to throw away those results beyond the page size limit and give the impression to any calling code that the next page could be requested. However, since ID queries are changed to solr real time get requests, requesting the next page doesn't work. Solr ignores all of the paging parameters when utilizing the real time get handler. This means that solr will again return the full list of documents and cause the catalog framework to return the same list of results it did for page 1 as page 2. This PR works around this limitation such that if paging parameters are received it will not do a real time get request.

Who is reviewing it?

@pklinef @clockard

Select relevant component teams:

@codice/solr

Ask 2 committers to review/merge the PR and tag them here.

@glenhein @jlcsmith

How should this be tested?

  1. Ingest or create several metacards
  2. Query the catalog for all of the created metacards by their IDs and set the page size to smaller than the total number of metacards. For example: if there are 10 metacards, set the page size to 5
  3. Verify the first page of results are returned
  4. Adjust the start index to the second page and issue the query again
  5. Verify the second page of results are returned as expected (previously this would be the same list of results as the first query)

Any background context you want to provide?

What are the relevant tickets?

Fixes: #____

Screenshots

Checklist:

Notes on Review Process

Please see Notes on Review Process for further guidance on requirements for merging and abbreviated reviews.

Review Comment Legend:

derekwilhelm commented 1 year ago

build now

cxddfbot commented 1 year ago

Internal build has been started, your results will be available at build completion.

clockard commented 1 year ago

✅ Successfully tested functionality through a downstream project.

cxddfbot commented 1 year ago

Build SUCCESS See the job results in legacy Jenkins UI or in Blue Ocean UI.