search

PavlidisLab / Gemma

Genomics data re-analysis
Apache License 2.0
23 stars 6 forks source link

Add a mechanism when writing data files to mark them as ready #1249

Open arteymix opened 1 week ago

arteymix commented 1 week ago

I did experiment a little bit with Java FileChannel API which allows a Java process to acquire an exclusive lock on a file (not but a directory, which is problematic for MEX).

In addition, within Gemma, we should have a read-write lock for reading the contents of the file. The FileLock object is not meant for protecting a file within the JVM context.

There might be implication of doing that with NFS.

Until this is implemented, users are subject to receive potentially incomplete/corrupted files.

arteymix commented 1 week ago

This is going to be important for single-cell data since those file take significantly longer to produce.

arteymix commented 1 week ago

I've added a read-write lock to wait when a file is being generated and prevent multiple threads from generating the same file.

The next step is to support file-level locking. This is less of an issue though because most of the volume will occur on the Web server.