search

audeering / audb

Manage audio and video databases
https://audeering.github.io/audb/
Other
23 stars 1 forks source link

Use platform-dependent lock mechanism #269

Open frankenjoe opened 1 year ago

frankenjoe commented 1 year ago

As discussed in https://github.com/audeering/audb/issues/201 we have switched to a soft lock mechanism as a platform-independent lock mechanism. Unfortunately, this is not always safe and we can run into dead locks when a process is killed and the lock file is not removed. Since this is a very unfortunate behavior we should consider to switch back to a platform-dependent lock mechanism.

hagenw commented 1 year ago

The main possible downside of a platform-dependent lock file (FileLock on unix based systems, SoftFileLock on Windows) will be cases of a shared cache that is used with different platforms, e.g. will the Unix process wait until the Windows process is finished? What will happen with dead lock files?

frankenjoe commented 1 year ago

The main problem I see is that the unix process will not remove the lock file, so for the window process it still looks like the folder is locked.

frankenjoe commented 1 year ago

Maybe we can encode in .audb.yaml if a folder is shared across platforms and only in that case use the unsafe lock mechanism?

hagenw commented 1 year ago

When setting up a shared folder, you do not necessarily know which platforms will access the folder, e.g. if it is a network share, so I don't see how you can store it inside the config. What we could do instead maybe is adding a hidden file to the shared folder the first time any Windows machine accessed it. Afterwards we know that we have to use lock files for that particular folder.

frankenjoe commented 1 year ago

Yes, that's sounds like a better solution.