search

cedadev / search-futures

Future Search Architecture
BSD 2-Clause "Simplified" License
0 stars 0 forks source link

How to scan the whole archive? #161

Closed rhysrevans3 closed 2 years ago

rhysrevans3 commented 2 years ago

How to parallelise?

  1. Cut up the archive ourselves
  2. Distribute across threads instead of machines (celery)

How to store the items/collections

  1. Save to a file
  2. Cache