search

conda / conda-index

conda index, formerly part of conda-build. Create channels from collections of packages.
BSD 3-Clause "New" or "Revised" License
10 stars 9 forks source link

Sqlite cache issues with network file system and/or concurrent conda_index processes #177

Open meesepyter opened 3 months ago

meesepyter commented 3 months ago

Checklist

What happened?

We are storing an in-house conda channel on a NFS drive. After a recent upgrade from conda-build 3.27.0 to conda-index 0.5.0 (used conda index before, now python -m conda-index), we are facing problems with "database is locked" or "file is not a database" error messages from the sqlite3 module.

conda-index/index/common.py uses

conn.execute("PRAGMA journal_mode = WAL")
conn.execute("PRAGMA synchronous = 1")

The sqlite docs list as first disadvantage "All processes using a database must be on the same host computer; WAL does not work over a network filesystem", which is both not satisfied in our scenario.

Can the journal mode be changed to SQLite's default DELETE (not sure about PRAGMA synchronous ) to allow NFS storage of the channel (and concurrent conda index from different client machines)?

Conda Info

No response

Conda Config

No response

Conda list

No response

Additional Context

No response

dholth commented 2 months ago

Sounds feasible. We may have to check that it doesn't interfere with local multiprocess indxing.