PacificBiosciences / FALCON_unzip

Making diploid assembly becomes common practice for genomic study
BSD 3-Clause Clear License
30 stars 18 forks source link

wait_for_file and nfs file systems #27

Closed flowers9 closed 7 years ago

flowers9 commented 8 years ago

In unzip.py, wait_for_file(), the check for a "done" file can fail even if the done file is present if the remote file system updates the presence of the "exit" file before the "done" file (this is probably more likely when the remote file system is heavily loaded). Introducing a simple delay in the function can prevent a lot of extraneous restarting.

            # give done file a little time to show up
            for count in range(1, 5):
                if os.path.exists(filename):
                    break
                time.sleep(wait_time)

(On the plus side, all the restarting led me to spot the bug I posted about earlier. ;)

pb-jchin commented 8 years ago

We will eventually change the tracking mechanism to use code developed recently by @pb-cdunn which can solve this problem if the latency of the file system synchronization through the different nodes is not too bad.