ToransuShoujo / quesoqueue_plus

A queue system for Mario Maker 2
GNU General Public License v3.0
5 stars 3 forks source link

Reduce risk of loosing/currupting data #12

Closed liquidnya closed 2 years ago

liquidnya commented 2 years ago

If the software or the PC crashes the queue might have been in the middle of writing data to the file system, which could lead to a corrupted file.

To prevent this, the queue could write the data to a shadow file, wait for a file system sync, and then atomically rename the new file to the actual file name.

A solution to the problem could be to use this algorithm: https://stackoverflow.com/a/7957634 But with the following changes:

liquidnya commented 2 years ago

I am going to implement this like suggested here for a different project: https://github.com/simonlast/node-persist/issues/94#issuecomment-405127513 Using the libraries https://github.com/isaacs/node-graceful-fs and https://github.com/npm/write-file-atomic.

skybloo commented 2 years ago

Another option that would probably be simpler to implement would be using the database as a pseudo application directory as in this article, which would provide atomic updates more or less for free, and avoid having to go through the whole file to read/write state. The shadowing would probably be somewhat more performant overall, just requires more work and file system stuff.

liquidnya commented 2 years ago

I think the suggestion to use a database with a pseudo application directory is very good, since it will greatly improve performance. Changing the write from just a regular write to an atomic write was actually quite easy by using a library that does exactly that, which is currently implemented by #22. The most work was to move all the code in a new source file, and to convert the old save file format to the new one.

liquidnya commented 2 years ago

Fixed by #22. This could be further improved with the following:

However, the underlying issue of loosing data is solved, and all these improvements are performance improvements only.