search

radiocosmology / alpenhorn

Alpenhorn is a service for managing an archive of scientific data.
MIT License
2 stars 1 forks source link

Remove Lustre I/O from MainThread #183

Closed ketiltrout closed 2 months ago

ketiltrout commented 2 months ago

It's just not reliable enough, and tends to slow the entire thing down.

These sorts of things just can't happen in the main thread if we want to keep it responsive. (Probably not all of these are in need of fix: some of these methods are likely only called from within a task already):

In LustreHSMNodeIO.check https://github.com/radiocosmology/alpenhorn/blob/742165e5f0177363acdf30daa92717b4c5d0db96/alpenhorn/io/lustrehsm.py#L275

In LustreHSMNodeIO.auto_verify https://github.com/radiocosmology/alpenhorn/blob/742165e5f0177363acdf30daa92717b4c5d0db96/alpenhorn/io/lustrehsm.py#L304

https://github.com/radiocosmology/alpenhorn/blob/742165e5f0177363acdf30daa92717b4c5d0db96/alpenhorn/io/lustrehsm.py#L322

In LustreHSMNodeIO.filesize https://github.com/radiocosmology/alpenhorn/blob/742165e5f0177363acdf30daa92717b4c5d0db96/alpenhorn/io/lustrehsm.py#L401

In LustreHSMNodeIO.open https://github.com/radiocosmology/alpenhorn/blob/742165e5f0177363acdf30daa92717b4c5d0db96/alpenhorn/io/lustrehsm.py#L439

In LustreHSMNodeIO.ready_path https://github.com/radiocosmology/alpenhorn/blob/742165e5f0177363acdf30daa92717b4c5d0db96/alpenhorn/io/lustrehsm.py#L457