search

mllg / batchtools

Tools for computation on batch systems
https://mllg.github.io/batchtools/
GNU Lesser General Public License v3.0
171 stars 51 forks source link

Store a backup of registry.rds #154

Closed jakob-r closed 6 years ago

jakob-r commented 6 years ago

So that if the file is corrupted you can come back to the backup

mllg commented 6 years ago

There typically already is a backup if something went wrong while writing the registry (https://github.com/mllg/batchtools/blob/master/R/saveRegistry.R). Thoughts?

jakob-r commented 6 years ago

I think in the case the "registry.new.Rds" existed but was 0kb.

So what is happening?

  1. registry.new.Rds will be removed (probably not existent)
  2. registry.new.Rds will be written
  3. renaming registry.new.Rds -> registry.Rds (this apparently overwrites registry.Rds without warning)

What can go wrong?

  1. Nothing
  2. Corrupt file written? Probably/hopefully not possible
  3. It looks like when the renaming fails only a warning is issued and FALSE is returned. You probably want to throw an error if renaming fails here. Now the question is: Can file.rename destroy the from and the to at the same time?

How I would do it:

  1. write "registry.Rds" and overwrite it
  2. remove "registry.old.Rds" if it exists
  3. copy "registry.Rds" to "registry.old.Rds"
mllg commented 6 years ago

This issue came up because the file system quota was exhausted. Maybe the file.rename() operation failed here for some reasons. This is very hard to reproduce.

Your proposed solution would require an extra copy of the registry. This could slow things down. I'll wait if others have problems, too. Never experienced this myself.

jakob-r commented 6 years ago

At least you should stop() if the result of file.rename() is not TRUE. So in case it happens again we have a way to trace it back.

mllg commented 6 years ago

Ok this in now in the master. However, there is no real way to deal with this.