Open hamr opened 2 years ago
Hi @hamr thanks for posting this question. It does appear we haven't fully documented this case. We'll add that. In the meantime here is how you configure the Scheduler
to use a persistent database (e.g. sqlite). By default, Scheduler()
will use an in-memory database that will not persist the cache between executions (which is what you are seeing).
from redun import Scheduler
from redun.config import Config
scheduler = Scheduler(config=Config({
"backend": {
"db_uri": "sqlite:///redun.db",
}
}))
scheduler.load() # Auto-creates the redun.db file as needed and starts a db connection.
result = scheduler.run(main())
In our own code, where we embed a redun Scheduler inside a larger python application we instantiate Scheduler
like above. RedunClient()
is really only used in tests.
Let me know if that helps.
Excellent, thank you!
The documentation helpfully provides examples of how to instantiate a Scheduler and run tasks, e.g.
However, that does not appear to take advantage of caching -- tasks run every time -- so it's not quite analogous with running something like
I'm guessing that's because scheduler object isn't making use of the database. And I expect it's relevant that I'm seeing this message when calling
scheduler.run()
.How do you set up a scheduler object so that it behaves more like calling redun on the command line? Or do you recommend using RedunClient instead? And could you please add that to the docs?