Closed robbert-harms closed 1 year ago
I have gotten this error as well - the initial stack trace is equivalent to the one in this report. Unfortunately it seems to have locked up everything and one flow has been "running" for 2 days with everything else completely halted / stuck on "late". Perhaps relevant: flows have been running fine for quite some time and the database become pretty large (176MB) which I did not expect.
Same here, running a flow with simple tasks and .map()
locks up the database and fails all the tasks.
Hi! This is because you are using the SQLite backend with an ephemeral API. When using the ephemeral API, each task run starts its own copy of the API server in-memory. Each of these API servers creates a connection to the database. When you run many tasks concurrently, you have many connections to the SQLite database which does not perform well with concurrent writes.
Instead, you should start a standalone API server (with prefect orion start
) then set PREFECT_API_URL
so your flow and task runs connect to it. Then, the single instance of the API will manage communication with the database reducing strain during concurrent runs. I'd also recommend switching to using PostgreSQL for the backing database instead, as it'll perform much better at scale.
Thanks @madkinsz, I suspected something like you've explained taking place, however is there a way to run flow as a script without setting all the infa? Would SequentialTaskRunner
be a remedy here?
If you run tasks sequentially yeah you're unlikely to run into database locking issues. You could still have problems running flow runs concurrently though.
For anyone else encountering this issue - in my setup, I was running a standalone "prefect orion start &", an agent as well, PREFECT_API_HOST was set (so I assumed the URL setting was set as well) - and all the flows were using SequentialTaskRunner (because I am using Prefect as a replacement for cron w/ a nicer UI).
But only upon explicitly setting PREFECT_API_URL did the server type switch from ephemeral to hosted.
Is it possible to solve this issue without setting up all this infrastructure? My main use of Prefect was actually being able to smoothly parallelize my pipeline while running only locally on my development machine. This used to work before in Prefect version 2.0.3, after that version it keeps crashing. Is there a way to disable this database system? Can we have an in memory database working with multiple threads/processes?
Hello - I'm also seeing the effects of SQLite database locking with the ephemeral API in Prefect (specifically with ConcurrentTaskRunner
) (many thanks @shntnu). Based on my understanding of the conversation in this issue, it seems like this locking may occur with any scenario which leverages task.map()
without SequentialTaskRunner
or prefect orion
server running somewhere outside of memory (is this accurate?).
Like @robbert-harms mentions, I would also seek a solution that doesn't rely on additional complexities with new or added infrastructure. More broadly I'd say that prefect's task.map()
capabilities are one reason I find inspiration in this library, in addition to a lot of the other fantastic work taking place, and I hope there's a way to resolve these challenges.
A few additional questions in the interest of solutions:
task.map
) outlining the risks of using the functionality without SequentialTaskRunner
or a prefect orion
server?| WARN | This workflow uses task.map() capabilities without non-ephemeral orion or SequentialTaskRunner, tasks may cause SQLite locking or failures!
Thanks for the comments!
I'm not sure adding warnings to .map
specifically makes sense to me. Any highly concurrent usage of the database is going to cause problems. We're going to be adding more concurrency heavy features in the future. I think that we should highlight that the ephemeral API and SQLite database is not intended for highly concurrent workflows. Perhaps there are some good places in the documentation for this cc @tpdorsey
There's also the SQLite HCTree project https://sqlite.org/hctree/doc/hctree/doc/hctree/index.html. DuckDB might make sense, but it'll be hard for us to justify development time for another database backend at this time. Writing migrations and testing both databases is already quite a bit of work.
I'm working on changes to improve concurrency handling. Perhaps I can expose automatic throttling of concurrency when using the empheral API with SQLite or make the settings easy to reach.
I think the fastest and simplest fix may be to adjust our task runs on Ray to share a single client instance which would avoid creating multiple ephemeral APIs and consequently create less concurrent SQLite connections. I'm a little fuzzy on the details of how that would work, but it is probably worth exploring.
Thank you @madkinsz for the thorough reply and thoughts towards addressing this topic! I'm excited to see the concurrency heavy features and capabilities you mentioned. The controls with automated throttling for concurrency seem like a great way to balance the challenges at hand, though I'd still hope for a simplified solution via the database technologies (perhaps through SQLite HCTree as you suggested). I'd also wonder: are there are there any ways to BYOD "bring your own database" when leveraging the ephemeral API (with an eye towards experimentation or decoupling the problem space)?
This issue is stale because it has been open 30 days with no activity. To keep this issue open remove stale label or comment.
Has this been resolved yet? It would be wonderful if we can run prefect pipelines standalone.
This issue is stale because it has been open 30 days with no activity. To keep this issue open remove stale label or comment.
This issue was closed because it has been stale for 14 days with no activity. If this issue is important or you have more to add feel free to re-open it.
Has anyone come up with a solution for this? I am experiencing a similar issue while running a large number of concurrent tasks.
@blakete first switch to using a hosted server (i.e. prefect server start
or Prefect Cloud) and if you're already doing that switch to Postgres as a backing database.
@zanieb Thanks for getting back on this! When you say Postgres as a hacking database do you mean for the application or for the Prefect Orion database?
@blakete for the Prefect Server (previously known as Orion) database, yes.
Hi @zanieb Just a quick question on this issue, as I am experiencing it too. Is there a way to switch to Postgres while still retaining my flow history and run deployments? I dont want to switch over to Postgresql and lose out on all the blocks and everything that was already built out. Thanks!
@cleung1996 I don't work at Prefect anymore, but I don't think there's automated migration. cc @serinamarie
First check
Bug summary
With the latest Prefect v2 library, if I run a flow with RayTaskRunner, I get the following error:
sqlite3.OperationalError: database is locked
after that the error:
followed by:
sqlalchemy.exc.InvalidRequestError: Can't operate on closed transaction inside context manager. Please complete the context manager before emitting further commands.
ending with:
sqlalchemy.exc.InvalidRequestError: Can't operate on closed transaction inside context manager. Please complete the context manager before emitting further commands.
It doesn't happen with the DaskTaskRunner or SequentialTaskRunner. I don't know if I should post this bug here or in the prefect-ray github.
Reproduction
Error
Versions
Additional context
No response