search

OasisLMF / OasisPlatform

Loss modelling platform.
BSD 3-Clause "New" or "Revised" License
41 stars 17 forks source link

File not found issue when file exists #970

Open benhayes21 opened 7 months ago

benhayes21 commented 7 months ago

Issue Description

File not found issue - Oasis fails to start an analysis randomly due to missing file (which is actually not missing and exists). This cluster is 2 years old and has lots of old files stored. Timestamps show the file is created after the missing file error is raised (maybe an issue with write speed to the file system which has too many files in it)

Note, old files can be deleted if analyses are never revisited after the run but this is messy. Deleting the analysis is the proper way to go and should remove the files (but there are lots of analyses there which have never been deleted)

Oasis to consider retention policy on file storage, but possibly not a good idea for users who do go back to older analyses

Oasis to investigate object store solution (see lot 3 approach) and migrate from Azure files

Users to consider adding a deletion process into the workflow once the results have been obtained, either immediately after the analysis or as part of a timed batch cleanup.

benhayes21 commented 7 months ago

placing on hold as can't recreate the issue. Suggested user remove old files and retest