Closed Lukas113 closed 7 months ago
Attention: 210 lines
in your changes are missing coverage. Please review.
Comparison is base (
367cf11
) 65.72% compared to head (a4a599a
) 64.74%.
:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.
closes #535
This PR addresses the issues of #535 . The main work was to make sure, that the disk-caching was not performed in a read-only filesystem, since in Singularity you're not the same user as the creator of the image. Therefore, I refactored
FileHandler
to have a short-term-memory and long-term-memory disk-cache available. Long-term is mainly for downloaded objects, where short-term-memory is for intermediate data-products, if no file/dir-path was provided in the according API-function. For details, see code & docstring ofFileHandler
.In addition, I've seen two instances where I was not happy how it was implemented (in addition with the disk-caching), and saw a good opportunity to refactor the code. The first one was, how
DaskHandler
was implemented. It had some twisted code with SLURM, where the separation of concerns was not fulfilled. In addition, I've refactored some telescope-function code to be increase the stability & re-usability of the code.This PR should be merged after #526 is merged, because it's a branch created from
512_sarus
.The testing-setup was, I tested our tests in singularity- sarus & docker-container, and they passed there. This is of course no guarantee that I got everything, however, I've looked through the entire repository & the tests passed which is good enough I think.