Open 9999years opened 10 months ago
I don't think it is possible for a sqlite database to be written to concurrently, even with the WAL and WAL2 modes. In particular, the COMMIT steps require exclusive access to the database.
https://sqlite.org/cgi/src/doc/begin-concurrent/doc/begin_concurrent.md
That's correct. I don't want concurrent writes to the database, I want hiedb
to wait for the database lock to be released instead of exiting with an error.
Currently the behavior is this:
hiedb
attempts to open the sqlite databsehiedb
sees the database is already locked and exits with an errorThe behavior I want is this:
hiedb
attempts to open the sqlite databasehiedb
sees the database is already lockedhiedb
outputs a warning or notice to the user that the database is lockedhiedb
waits, periodically checking if the database can be opened, until the lock is released and hiedb
can continueI'm not sure hiedb
is the right place for this functionality, because different clients might expect different behaviour from hiedb
depending on their own needs and usage pattern, like whether to retry, how many times to retry, the interval between retries etc. Instead of making this configurable in hiedb, a client can implement this functionality for their own use case.
For example, HLS implements retrying functionality here: https://github.com/haskell/haskell-language-server/blob/66cf40033cf6d1391622ee4350696656ea0c39d9/ghcide/session-loader/Development/IDE/Session.hs#L361
When I run
hiedb index .hiefiles
concurrently, the second process errors because the first has already locked the database:Context
We're using
ghciwatch
for development in combination withstatic-ls
for language intelligence.ghciwatch
handles loading and compiling modules inghci
for rapid reloads, andstatic-ls
leverageshiedb
to provide language intelligence. (Our project is too big forhaskell-language-server
.)We're using
ghciwatch
's lifecycle hooks to reindex the hiefiles on reloads:Because
ghciwatch
runs these commands asynchronously (so that it can keep reloading and responding to changes while the files are indexed), it's possible for multiplehiedb
processes to run at once (especially on the first load when 7500+ modules are being indexed), which has been triggering this bug.I have a few ideas for working around this in
ghciwatch
(only allowing one of each hook to run at once, killing old hook processes before launching new ones, etc.), but I think we should fix this inhiedb
as well.