search

usnationalarchives / OPAProd

Tracking enhancements to OPAProd
1 stars 0 forks source link

Raise/remove offset limit #91

Open DominicBM opened 9 years ago

DominicBM commented 9 years ago

I think it's pretty clear we have a desire to substantially raise or remove the offset limit in the long term. This limit makes it impossible to ever see an entire results set, whether it is in the search UI or the API (and possibly bulk export, but I am not sure if that is the same setting). Currently this limit is set at 500 for logged-out users, 2000 for regular registered users, and 20000 for power users. ST treats this as a non-starter when brought up, but I thought we should track it here anyway.

WaxCylinderRevival commented 9 years ago

Agreed. As a public institution, it is unacceptable to limit search returns in this manner. I have never seen a peer institution require logging in to view results -- perhaps a high volume of downloads and API calls, but not browsing and search results in our open catalog.

The step ladder of 500, 2000, and 20,000 also makes no sense, especially given one record group can easily have over 2000 series -- not to mention large-scale digitization efforts. DOCUMERICA, for instance, boast over 15,000 images that people should be able to scroll through without logging in: https://catalog.archives.gov/id/542493?q=documerica

WaxCylinderRevival commented 9 years ago

Is there a "vital" tag to add to this discussion?

DominicBM commented 9 years ago

@WaxCylinderRevival: The labels are all custom, and can be created by anyone. ;-)

https://github.com/usnationalarchives/OPAProd/labels

WaxCylinderRevival commented 9 years ago

Would you support "vital" or other verbiage? I thought "critical" might be reserved for issues related to "critical" defects so I was shying away from creating/using that label.

DominicBM commented 9 years ago

Or "high priority", or something, sure. In other bug trackers there is usually a way for users to vote for an issue, but we don't have that here.

WaxCylinderRevival commented 9 years ago

Okay, I'll add "high priority."

WaxCylinderRevival commented 9 years ago

NASA uses Socrata as its data catalog host with a built-in SODA API. For comparison, SODA APIs support a maximum of 50,000 records per page.

http://dev.socrata.com/consumers/getting-started.html "For performance, SODA APIs are paged, and return a maximum of 50,000 records per page."