search

buckyos / CYFS

CYFS is the next-generation technology to build real Web3 by upgrading the basic protocol of Web (TCP/IP+DNS+HTTP),is short for CYberFileSystem. https://www.cyfs.com/, cyfs://cyfs/index_en.html.
https://www.cyfs.com/
BSD 2-Clause "Simplified" License
1.99k stars 276 forks source link

Problems with ood-daemon restore backup data #269

Open lurenpluto opened 1 year ago

lurenpluto commented 1 year ago

According to the feedback of some related products, OOD has been found to be missing data during gateway usage use after restoring data according to the following process:

  1. Initialize a new ood environment with ood-installer In this case, the ood is in an unbound state and only the ood-daemon process is available
  2. Call the remote-restore interface of ood-daemon to restore the data from other ood backups This process may take longer, depending on the size of the backup data, etc.
  3. The above process includes the binding process because the {cyfs}/etc/desc directory is backed up.
  4. After the recovery is completed, use gateway directly It's found that in some cases the data is missing, especially when accessing the root-state to get the dec app's state, the state is completely different from the state of the backed up source ood
lurenpluto commented 1 year ago

After analyzing the actual recovery data and the local data, as well as the reviedw of the whole process of remote-store, it was found that the problem should be caused by the restore step of uni restore, as well as the combination of the existing activation detection of ood-daemon

Restore process

The current uni restore recovery steps are defined as follows: https://github.com/buckyos/CYFS/blob/89110cca0f10409ee6958a85845c64436ead849f/src/component/cyfs-backup-lib/src/backup/restore_status.rs#L8-L16

It can be seen that the restore step restores the key-data first, then the objects and chunks, and the {cyfs}/etc/desc directory in the key-data of the device.desc/device.sec identity file, so it will lead to the whole restoration process in which the files in etc/desc are recovered first

Binding detection mechanism of ood-daemon

The binding detection logic of ood-daemon is that if the identity file (device.desc/device.sec) in the {cyfs}/etc/desc directory is found to be missing, a monitor process is started to periodically check whether the identity file is created, then it will pull up other services including gateway

So the problem comes, after the gateway is started, because the remote store process is still continuing, so the local objects and chunks are still incomplete at this time, the gateway loads the root-state, in the case of loading the root, there will be object meta exists but the corresponding object blob is missing, This leads to the belief that the object does not exist and get_object will return None:

https://github.com/buckyos/CYFS/blob/89110cca0f10409ee6958a85845c64436ead849f/src/component/cyfs-lib/src/prelude/named_object_cache.rs#L142-L148

https://github.com/buckyos/CYFS/blob/89110cca0f10409ee6958a85845c64436ead849f/src/component/cyfs-lib/src/prelude/named_object_cache.rs#L261-L281

So gateway will initialize a new global-state, the content is also completely empty, resulting in the above problem

lurenpluto commented 1 year ago

This problem is considered to be fixed from the following two sides:

1. Adjust the steps of uni restore

Give priority to restore objects and chunks, and restore key-data last, make sure {cyfs}/etc/desc is released after objects and chunks are restored to avoid the above problem

Also during the key-data recovery process, we need to improve the order of the files recovered, make sure the etc directory is recovered last, and the db files should be recovered first to ensure that ood-daemon has recovered all the data after detecting the identity files in etc/desc dir

2. Improves the binding detection logic of ood-daemon service

Since the remote-store is carried out inside ood-dameon, ood-daemon can stop detecting bindings when there is a remote store task, and wait until the restore task is completed, then continue with the binding monitor logic.

However, this logic improvement is limited and can only play a supplementary role. If an external independent process is used for restorer-store operation, then ood-daemon has no way to know the progress of restore.

lurenpluto commented 1 year ago

This problem is considered to be fixed from the following two sides:

1. Adjust the steps of uni restore

Give priority to restore objects and chunks, and restore key-data last, make sure {cyfs}/etc/desc is released after objects and chunks are restored to avoid the above problem

Also during the key-data recovery process, we need to improve the order of the files recovered, make sure the etc directory is recovered last, and the db files should be recovered first to ensure that ood-daemon has recovered all the data after detecting the identity files in etc/desc dir

2. Improves the binding detection logic of ood-daemon service

Since the remote-store is carried out inside ood-dameon, ood-daemon can stop detecting bindings when there is a remote store task, and wait until the restore task is completed, then continue with the binding monitor logic.

However, this logic improvement is limited and can only play a supplementary role. If an external independent process is used for restorer-store operation, then ood-daemon has no way to know the progress of restore.

The First option above has been adopted to make improvements, including the following two major changes. 1.

  1. In the uni restore logic, the restore order is adjusted to objects->chunks->key-data
  2. In the key-data restore logic, the etc/desc directory is restored last.

The above two points ensure that even with the existing bind detection logic of ood-daemon, it is possible to restore and bind ood at the same time by restore operation

See at 3cd354ab5c9406e032511bcc07415c874a105619