search

conda-incubator / conda-store

Data science environments, for collaboration. ✨
https://conda.store
BSD 3-Clause "New" or "Revised" License
144 stars 46 forks source link

[BUG] - SQLite experience with conda-store is not great #728

Open costrouc opened 9 months ago

costrouc commented 9 months ago

Describe the bug

The root of the issue is the length of transactions and when conda-store-server is initially starting it attempts to update the sqlite database with bulk inserts. I believe bulk inserts are the key problem here.

The sqlite database currently gets locked frequently.

Expected behavior

SQLite should be a great database choice for sqlite. Especially for the local use case.

How to Reproduce the problem?

Run conda-store-server locally.

Output

No response

Versions and dependencies used.

No response

Anything else?

No response

nkaretnikov commented 8 months ago

@costrouc Could you please add more thoughts here from today's meeting? You had some other ideas on what might help and some differences between SQLite and Postgres.