Backup and other feature requests

[Elmit2015]

I feel increasingly uncomfortable if there is:

• no reliable backup
• parallel channels or both side funded channels
• channel top up payments

Maybe we can setup a voting system to get features ranked by importance, like the video voting of Rene.

I know there are hundreds of things to do.

[Coinomatron]

As a test, I'm mirroring the .lightning/ folder via a bash script:

while true; do inotifywait -r -e modify,create,delete /home/user/.lightning/; rsync -avz /home/user/.lightning/ /home/user/LN_folder_backup/; done

Seems to work, just restarted the client using the backup dir and made a payment. Maybe someone else can verify if this is a sufficient method for backing up.

[rustyrussell]

The Right Way is to have a plugin with hooks at the db level, so you back up before c-lightning continues. That plugin could back up anywhere. I'm looking at it now, for 0.7.1; agreed it's vital!

[manreo]

Do that and I'm switching from lnd to c-lightning :)

[ZmnSCPxj]

The Right Way is to have a plugin with hooks at the db level

Does this mean the plugin needs an implementation of SQLite3?

What exactly are the db-level hooks we want? Do we ask SQLite to provide a correctly-escaped query for every db query we make, then pass that sanitized query to the hook before invoking SQLite on the sanitized query?

[trueptolemy]

What exactly are the db-level hooks we want?

I think it's db_write hook #2421 , and it passes database changes to hook. I also think about the plugin to backup, and aslo not sure if plugin should implemente a SQLite directly. It seems indeed need.

[trueptolemy]

Does this mean the plugin needs an implementation of SQLite3?

Another idea is: Based the database, we can extract the channel event and implement the compatibility with SCB for Lnd.

[ZmnSCPxj]

Based the database, we can extract the channel event and implement the compatibility with SCB for Lnd.

If I understand correctly, recovering an SCB in lnd simply marks the channel such that on next reconnect to the peer, lnd falsely accuses it of not following the protocol, causing the peer to unilaterally close in despair.

If the peer suspects your node is lnd, and that you were using an SCB, it could replay old state and the lnd node will not be able to revoke it.

The only safe backup in Poon-Dryja is replicating the database.

I would argue that a proper replicating self-healing filesystem would be more reliable than anything we could build on top of this mechanism, if only because well tested replicating self-healing filesystems already exist.

[trueptolemy]

@ZmnSCPxj Thank you for your explanation. I learned a lot from it!

[ZmnSCPxj]

The counterargument is that the lnd could pretend have a SCB recover, in which case a thief peer would post a revoked state whose revocation key the lnd actually knows and lose its money, but an honest peer would not. But for myself, I would not have released an SCB-like feature until I had code that would occassionally emulate an SCB recover to root out thief peers.

[trueptolemy]

I would not have released an SCB-like feature until I had code that would occassionally emulate an SCB recover to root out thief peers.

If I understand correctly, during the honest user who lost the recent data, the thief and you, a ‘police’ do entrapment, these 3 roles, you are the 100% safe.

[rustyrussell]

I'm leaving this open, since although 0.7.1 has support for database replication, it'd be nice to have a well-supported plugin which made it easy to do. But I'm removing it from the 0.7.1 milestone, since the c-lightning core part is done...

[jdabs]

I take snapshots of my VM running c-lightning. Although I have not had to recover one yet, is this a possible way to backup for now?

It feels like the time-sensitive aspect of bitcoin core and lightning channels would not make it easy for an old snapshot to sync back up to the network correctly as it comes from a point in the past.

[ZmnSCPxj]

I take snapshots of my VM running c-lightning. Although I have not had to recover one yet, is this a possible way to backup for now?

No, and is a risk in fact, due to the "toxic waste" problem. Old backups are actively dangerous as old state is the theft vector, thus all implementations must treat a channel counterparty showing old state (i.e. you after recovering from backup) as a thief and punish it.

We support data_loss_protect which is the same mechanism lnd SCB uses (so that old backups like this are effectively a mass SCB, though still not recommended as if the counterparty is offline you still cannot recover coins) but there were some data_loss_protect incompatibilities with lnd.

What you need is an on-going continuous backup process, which is what the db_hook is for and which is intended to have a plugin later on (but as I understand it the db_hook itself is currently insufficient --- it does not backup db_migrations and we still need a db_hook incrementing counter, and then @jb55 and @cdecker will still have to actually implement the backup plugin).

If you need on-going continuous backups now and have the equipment for it, install multiple permanent storage devices and use ZFS with RAID-Z across devices (you can turn off dedup especially if you use the ZFS mount only for the lightningd data; most of the data is cryptographic nonces so no duplication likely). That way failure of one storage device allows you to recover from non-failing storage.

[jdabs]

@ZmnSCPxj thanks, this is very good information.

It probably is worth repeating somewhere -- the danger of restoring a VM snapshot. I believe many people run lightning on a personal VPS. In that scenario, it is a common practice to think of snapshots as quick and dirty backups, and they probably don't realize it opens them up to fund loss (or at least I didn't know that).

[SimonVrouwe]

Just to let know, I am working on a dbbackup plugin. Will probably make PR in couple of days.

[ZmnSCPxj]

@SimonVrouwe you might want to coordinate also with @jb55 @cdecker who also want to do something similar.

Note in particular that from the C-lightning mailing list discussion, it seems the current db_hook is missing an incrementing number (so that backup plugins can know if they missed some changes, so they'd be better off copying the current db state and then adding the current changeset on top of that) as well as reporting db queries in the db_migrations array (which are needed else if somebody upgrades, the changes added in db_migrations will not be reflected in a backup at all). I do not know if @jb55 and @cdecker have plans or code for these.

[mloop1]

Hi everybody, any update on this? is the backup plugin still in the plans?

[darosior]

Hi @mloop1, SimonVrouwe had a shot but I think the @cdecker's database serie will make this easier (actually I think it aims to, see https://github.com/ElementsProject/lightning/pull/2924)

[whitslack]

If you need on-going continuous backups now and have the equipment for it, install multiple permanent storage devices and use ZFS with RAID-Z across devices

That doesn't help in the case that my computer is consumed by a fire. We need the ability to ship database updates off to a remote system.

The db_hook does sound like the Right Way.

[GlenCooper]

Lost a small amount of BTC (under $10 worth) permanently because I was setting up my first ever lightning node on an Odoid-HC2. Once I got the node running, I happily/stupidly snapped the case onto the heatsink... with the MicroSDHC still protruding out the back. With this case, you need to remove the MicroSDHC card before putting the case on. Oops. I didn't do that. It instantly snapped the MicroSDHC card in half and killed the lightning node.

Please build a backup plugin.

[cdecker]

There is at least one backup plugin in the community repository which does incremental backups. The only current backend however is a file backend that could be mounted on a separate machine. If you used that plugin and the backup file was on the same disk it wouldn't have helped however.

We are working on Dropbox and GDrive backends in order to provide offsite backups, but that's complicated by the fact that neither API allows appending to a file, so we'd have to write a file per DB transaction, and then aggregate later. If you have experience with python you're welcome to help :-)

[leshik]

Could Dqlite be leveraged somehow to the benefit of backups?

[ZmnSCPxj]

Depends on how well dqlite tracks sqlite.

C-lightning currently supports PostgreSQL, so if you have the ability and inclination to use that instead, and do replication on Postgre, you can use that instead.

there is also the WIP CLDCB