alecandido / partons

Distribute and evaluate partonic distributions
1 stars 0 forks source link

Cache all remote resources #7

Open alecandido opened 1 year ago

alecandido commented 1 year ago

All remote resources should be cached locally, to limit network issues.

The cache should be maintained in the suitable cache folder, with the following structure:

source-1/
  index-file
  set-a/
    ...
  set-b/
    ...
source-2/
  index-file
  set-c/
    ...

where source-1 can be identified by its URL, and set-a by its ID-name (not name only, since it is not guaranteed to be unique, nor ID only, to improve manual inspection).

alecandido commented 1 year ago

Manually invalidate cache for one or all elements.

Best, at least for development, is to maintain a trash bin-like folder, in which to move old files. Once already filled, it will be overwritten (it is already easy to make a back-up, just copying or moving the trash bin).

Add also the --force flag to drop cache without passing from trash bin.

alecandido commented 1 year ago

Resources are now cached, but path have to be better organized (as described in the OP), and trash bin has to be implemented.

Moreover, I want to implement a utility at configuration level that probe sequentially all the remotes for the given resource, stopping at the first one actually providing it. Since configurations are not the best (conceptually) for this application, maybe better to implement it in a further remote module (and make "remote" a subpackage).

alecandido commented 1 year ago

No longer sure we want to have a thrash bin, since it's more maintenance burden with little benefit...