search

OasisLMF / OasisDataManager

BSD 3-Clause "New" or "Revised" License
0 stars 0 forks source link

Update file cache logic #17

Open sambles opened 1 month ago

sambles commented 1 month ago

Issue Description

The file caching in the OasisDataManager package is based on the filename rather than the hash of the file data. This causes a problem with filenames like analysis_1_inputs.tar.gz which is the input tar (oasis files)

Example

  1. Generate inputs With loc_file = A --> File stored analysis_1_inputs.tar.gz
  2. Run Generate Losses loc_file = A --> File cached analysis_1_inputs.tar.gz
  3. Update portfolio with loc_file = B
  4. Generate inputs With loc_file = B --> File overwritten with same name analysis_1_inputs.tar.gz
  5. Generate Losses:
run_analysis[8fa67556-5a22-4c79-885c-d78f7c4c90d7]: {'oasislmf': '2.3.4', 'ktools': 'fmcalc : version: 3.12.2 - git update: ', 'platform': '2.3.4'}
[2024-05-02 09:12:49,079: INFO/ForkPoolWorker-1] Get from Cache: oasis/server/analysis_1_inputs.tar.gz

(This is the original analysis_1_inputs.tar.gz Tar file from Step 1

sambles commented 1 month ago

Workaround to prevent the above from happening: https://github.com/OasisLMF/OasisPlatform/pull/1037

However, still might be an issue with filenames we don't have control over (like model data)