search

likeawizard / polyglot-composer

A library to compose polyglot opening books from PGN game collections
GNU General Public License v3.0
5 stars 0 forks source link

Concurrent source parsing #13

Open likeawizard opened 1 year ago

likeawizard commented 1 year ago

Allow concurrent reading of sources

  1. Introduce a new CLI boolean flag '-c'
  2. When present and multiple sources are provided parse each file in a separate go routine
  3. Introduce a hard-coded limit on max concurrency. i.e. if there are 10 sources and maxConcurrecny is 5 open 5 concurrent reads and hold the rest.
  4. For 3. a semaphore could be an appropriate sync data structure.
  5. Currently there is a progress bar for a single file and multiple reads would overlap. Find and implement a multi-line updateable CLI printing method.
  6. For 5. display all the sources indicating their status: complete / pending / active progress bar.
  7. The source list order should not change and start with all pending and the parsers working downwards through them.