adsabs / biblib-service

ADS library service
https://ui.adsabs.harvard.edu
MIT License
4 stars 8 forks source link

Improve management of large libraries #156

Open aaccomazzi opened 2 years ago

aaccomazzi commented 2 years ago

A number of problems appear in library operations when the number of records grow beyond a certain threshold (a few thousand). This becomes a problem for people using ADS libraries to maintain bibliographies, and renders them far less useful. An extreme case is this one (26K records): https://ui.adsabs.harvard.edu/public-libraries/vJf4U0AGS8-NIqDp6gqEdQ

In particular:

  1. "View library in search results" times out
  2. Metrics and visualizations (even though they take the top N results) sometime fail to complete
  3. Export seems flakey (not sure what's going on, but paging then exporting seems to fail?)

One initial step would be to implement the change that @romanchyla advocated when exporting the library, which at least allows for further manipulation of the bibcodes in the main application: https://github.com/adsabs/solr-service/blob/f5d85d6d4cd75ed26a7f00a5c294d360b7b954df/solr/views.py#L446