Document NFS Incompatibility

[grantjenks]

Document incompatibility of DiskCache and NFS due to SQLite.

This is relevant to PythonAnywhere users.

[grantjenks]

Fixed and deployed in v2.0.2 on PyPI.

[mungojam]

It didn't end up well, so I concur that it isn't really do-able :)

With NFSv4, it is possible to get this working because it has much better locking. I wanted to use this package with AWS EFS and lambda functions and was able to get it working reliably (so far), with this horrible monkey-patch before importing FanoutCache:

def _safe_connect(database, timeout, isolation_level):
    return _original_connect(
        f"file:{database}?vfs=unix-excl",
        timeout=timeout,
        isolation_level=isolation_level,
        uri=True,
    )

_original_connect = sqlite3.connect
sqlite3.connect = _safe_connect

the key change is to use a SQLite uri and then specify the alternative linux VFS which has support for NFS v4 locking:

?vfs=unix-excl

There's a bit of a summary here:

https://aws.amazon.com/about-aws/whats-new/2017/03/amazon-elastic-file-system-amazon-efs-now-supports-nfsv4-lock-upgrading-and-downgrading/

[SoundsSerious]

@mungojam can you comment on what broke in your application? I've read a number of cautionary tales as well as a number of success stories using sqlite on NFSv4 so I would love to not reinvent the wheel :)

I am trying to understand this issue better, and see that you have tried something I was very close to what I was about to try: EFS(NFSv4) + vfs=unix-excl SQL mode

Referencing the linked sqlite documentation the relevant passage is

SQLite uses reader/writer locks to control access to the database. (
Under Win95/98/ME which lacks support for reader/writer locks, a probabilistic simulation is used instead.) 
But use caution: this locking mechanism might not work correctly if the database file is kept on an NFS filesystem. 
This is because fcntl() file locking is broken on many NFS implementations. 
You should avoid putting SQLite database files on NFS if multiple processes might try to access the file at the same time. 

This seems to suggest this should work with an enterprise grade NFSv4 implementation like EFS, especially in light of answers like this supporting flock on EFS https://stackoverflow.com/questions/53177938/is-it-safe-to-use-flock-on-aws-efs-to-emulate-a-critical-section https://stackoverflow.com/questions/34464880/what-happen-when-i-lock-file-located-on-remote-storage-via-fcntl

I see some pain points with processes dying and corrupting the database as well, however the counterpoint to that is that certain NFS can recover deadlocks

[mungojam]

@SoundsSerious it feels like a long time since I tried that and I'm afraid I don't remember what happened. I wish I had been less vague in my edit!

Have a go and see how you get on. It might well have been deadlocks, particularly if my lambda functions were timing out and leaving locks on EFS

[SoundsSerious]

Thanks for the feedback, I am thinking of a few ideas such as putting an outer lock around the DB for the application since writing is sparse and so that I can safely delete it in the case of a deadlock.