This series moves Moonraker's underlying database implementation from LMDB to Sqlite3. This is essentially a rewrite of database.py and a significant refactor of history.py and authorization.py. As such, I'm creating a pull request for these changes before merging. Users should see no difference, Moonraker will convert the existing lmdb database. Moonraker will not delete the prior lmdb database in the event that a bug is discovered that I missed in testing.
In addition, this should not break frontends, nor should it impact any components using internal APIs. This is accomplished by creating a namespace_store table, which is effectively a key-value store that mimics the way LMDB stores data.
The motivation for this change is several fold:
LMDB databases are architecture dependent. This has been an issue for users looking to transfer their database. Sqlite databases have no such dependency.
Sqlite backups don't require a special script. Simply stop the Moonraker service, then copy or restore the database file. Sqlite3 supports live backup and restore as well, which are implemented as API endpoints in this PR.
Moonraker is beginning to outgrow the limitations imposed by the LMDB database. The /server/history/list endpoints includes search/sort parameters that are easily applied using SQL. Future components that need to store, categorize, and search data can create their own tables to simplify access.
Sqlite3 is generally delivered as part of the standard python distribution, eliminating some external dependencies.
As mentioned above, this PR adds endpoints to backup and restore the sqlite database. In addition an endpoint is available to delete backups. Backups cannot be downloaded as they main contain sensitive information. Users will need to use physical media or another protocol (SCP, SMB, etc) to transfer backups off the host.
The one change that could affect frontend developers is that the existing debug endpoints can no longer be used to manipulate the job history, job totals, or authorized users. These items have been migrated from "namespaces" to tables, so /debug/database/item can't access them. This PR does include code for debug endpoints to manipulate rows, however I decided against enabling them for the time being. Given that SQL is ubiquitous, I think it might be easier for devs to write a script to manipulate these tables (ie: make fake entries) if the need arises.
This series moves Moonraker's underlying database implementation from LMDB to Sqlite3. This is essentially a rewrite of
database.pyand a significant refactor ofhistory.pyandauthorization.py. As such, I'm creating a pull request for these changes before merging. Users should see no difference, Moonraker will convert the existing lmdb database. Moonraker will not delete the prior lmdb database in the event that a bug is discovered that I missed in testing.In addition, this should not break frontends, nor should it impact any components using internal APIs. This is accomplished by creating a
namespace_storetable, which is effectively a key-value store that mimics the way LMDB stores data.The motivation for this change is several fold:
/server/history/listendpoints includes search/sort parameters that are easily applied using SQL. Future components that need to store, categorize, and search data can create their own tables to simplify access.As mentioned above, this PR adds endpoints to backup and restore the sqlite database. In addition an endpoint is available to delete backups. Backups cannot be downloaded as they main contain sensitive information. Users will need to use physical media or another protocol (SCP, SMB, etc) to transfer backups off the host.
The one change that could affect frontend developers is that the existing debug endpoints can no longer be used to manipulate the job history, job totals, or authorized users. These items have been migrated from "namespaces" to tables, so
/debug/database/itemcan't access them. This PR does include code for debug endpoints to manipulate rows, however I decided against enabling them for the time being. Given that SQL is ubiquitous, I think it might be easier for devs to write a script to manipulate these tables (ie: make fake entries) if the need arises.