Beets History Logging

[ab623]

Hi there.

So I've been looking to implement a beets work history log. This will essentially log all the actions beets completes, along with any supplementary information. A user then is able to see this via the beet interface and have a more granular view of the updates beets makes and to what. It will in the future allow the user to rollback changes (within reason) to a specific point in time. This means users can "test" commands, and commit changes. In addition if a feature to run beets commands in the background is implemented, a log of work can be recorded.

I've penned up an initial design to this functionality, with the hopes others can contribute and we can work as a community to see this implemented. This is a work in progress so anyone can suggest changes, etc.

I think this development should happen in 2 phases. They are as follows:

• Phase 1 - Implement the database and logging of beet's actions
• Phase 2 - Implement the rollback functionality

  Phase 1

  > Database Design

I believe the following columns will be required for the new history database.

• id - auto incremented ID number
• action - This will hold the action that beets is currently doing. This will be discussed more in the Logs section
• description/path - will hold the the item which the action is acting on
• active - this is a flag to indicate which row is active. This will come into effect, when we implement phase 2, which has the ability to rollback to a previous history. Rolling back, will invalidate all the logs below it, if a new command is run from a rolled back position. Further thought will need to go into then on implementation.

  > Logs

The logs will be able to log the following actions:

File Actions

• Move - A move of a file
• Copy - A copy of a file
• Link - Creation of hard or soft link
• Unlink - Remove of a hard or soft link
• Rename - Physical rename of a file
• Delete - The physical deletion of a file - This will unable to be reversed!

Library Actions

• Add - Addition of item in beets
• Delete - Deletion of an item
• Write - Essentially every change to meta data is a write. The addition of lyrics, is a write, but the removal of lyrics is still a write of blank text

General Actions

• Command - The history should be able to log the command that was executed with beets. The description will contain the command and the flags that were presented to it.

Each of these actions can and should be logged, and with each (except delete) a reversal process exists, which can be used to rollback changes.

With each action we would also needs to store a serialised version of the item, before the change is made. e.g. before a beet write writes to a file, we should log the full data of the item, and then run the command. This means when we rollback, we have have the information to rollback to.

> Config

For this phase I see only the following configuration items being relevant at the minute. Happy to accept more

• history -> enabled - A boolean flag to enable the logging functionality
• history -> loglimit - A number defining the maximum number of logs to hold. Database will delete any older logs. It may be beneficial to also either limit logs by DB size ( which can be difficult ) or date, but again that introduces its own issues. Number of rows should be sufficient.

  > Process

The process in which this would work would be simple. For each write to the database, the required information is stored. As the transaction class is abstracted, this write to the history can be implemented in a central location. It is important to note that there will be some slowdown in writing due to each request being logged. However this slowdown, is the tradeoff for implementing such feature.

As the logs go in, a process will be required to remove old logs, via a SQL command.

> UI

I believe the history UI should be fairly simple, called with something similar to beet history it will display a table of information, which will be pretty much a dump of the new history table. This will allow the user to see what beets has done.

Phase 2

I wont discuss much of phase 2 at the moment, but the premise is to effectively pass beets a number of the history item, and beets will start a process to run though every log from the present back to the specified number, and reverse every action to get beets back to that specific point in time.

Further details and design of this will be created at a later point.

Conclusion

I would like to hear all of your thought. Also there are some architectural challenges to face, like how we can add another table to beets but separate it from the library. Right now item and album are subclasses of Database. And we really need to figure out the best method of sorting and accessing the logs.

[sampsyo]

Interesting idea—I can see the use case.

Since versioning and rollback is such a complex concern, I would love to see the strategy here be based on prior work. Database versioning is a problem closely related to replication, which is in turn useful for beets' network API (see #736). It's common for web APIs to sync via revision numbers: essentially the same concept that's necessary here.

I don't think there's any off-the-shelf library we can use, but it would at least be good to look at other people's approaches: for example, here's a useful article about database revisions in the form of history tables. The main difference here is that we'd need to track filesystem changes in concert with the database itself.

Overall: A good first step here would be to prototype the basic versioning system for the database. It should be able to do this with minimal API disruption across the codebase, and it should add the (theoretical) ability to view the database's complete state at any revision in the past with something like beet ls -r5 or something. Once that proof of concept seems feasible and the abstractions are right, we can start worrying about the practical concerns.

[ab623]

I know the idea is ambitious, but I think beets can be taken to never levels by implementing these types of features.

I like your idea of basing it off prior work, however I think there might be a slight variance in how we store the information.

• Because we have to store historical data for not only items but also albums, commands and file system, we could either have a history table, which has all the existing columns in item, and album. When inserting a new history item, we populate the relevant fields. The advantages are we can easily query this table on any field. The disadvantage would be it would create a lot of redundant columns, and I'm not sure how big it would become when with fields created by plugins.
• We could serialise the item/album row in question and store it in a string and store this. This would remove the ability to search by specific field (not that you would want to)(it wouldn't remove it entirely, but just slow it down as you would need to unserialise each time), but it means beets can still store and restore all the data for an item, and be flexible for plugin fields.

Due to beets nature, if the 2 items, went thought 10 changes, we cant allow one item to revert back to revision 5, the entire database and field structure would have to reverse back to revision 5 for all items. This is to make the implementation MUCH simpler. reverting individual changes would be a nightmare to keep track of.

As I mentioned, and you agree with :), is we implement this in 2 phases. With phase 1 being the implementation of the revision logging/history feature. Is it best to develop this feature in a new branch?

[sampsyo]

Yes, whole database revisions seem like the way to go.

And also yes: a branch would be the right place to play with this. In the design, the overriding concern should be simplicity: the feature should be as orthogonal as possible to the other functionality in dbcore and, ideally, should not fundamentally change the Library API when doing ordinary (non-version-related) manipulations.

[goretkin]

I should be sorry to revive an old issue, but it's still open, so I figured I could give it a chance. If I understand correctly, sqlite is used to persist all of the music library state. Does anyone use something like https://ongardie.net/blog/sqlite-in-git/ or https://stackoverflow.com/a/38271631/415404 to version the sqlite database?

The diffs would be less useful (less "semantic") than a bespoke schema for versioning, or logging all actions done on the database, with beets-specific metadata. But it would be easier to implement since it's a beets-agnostic solution.