search

ecmwf / earthkit-data

A format-agnostic Python interface for geospatial data
Apache License 2.0
53 stars 12 forks source link

Crash when caching used in multiple processes #381

Open sandorkertesz opened 4 months ago

sandorkertesz commented 4 months ago

What happened?

When earthkit-data is used simultaneously in multiple processes to retrieve fdb data it crashes with:

OperationalError('database is locked')

It comes from the sqlite database used for indexing the cached files.

This happens when caching-policy is "user". In this case each process running under the same account uses the same cache area/database.

What are the steps to reproduce the bug?

See above

Version

lates

Platform (OS and architecture)

all

Relevant log output

No response

Accompanying data

No response

Organisation

No response

sandorkertesz commented 4 months ago

Could not reproduce this issue when retrieving MARS data or downloading data from URLS using 12 processes on a MacBook.