ERROR: cannot open .../oc_activity.ibd: No such file or directory

[psi-4ward]

I repetitive got this error:

Command execution failed (exitcode=1): `btrfs send -p /mnt/btrfs_root/_btrbk_snap/opt.20180423 /mnt/btrfs_root/_btrbk_snap/opt.20180424 | ssh -q -p 22022 -c aes256-ctr -i /opt/btrbk/secrets/h3-btrbk -C root@target 'btrfs receive /backups/server1/opt/'`
WARNING: [send/receive] (send=/mnt/btrfs_root/_btrbk_snap/opt.20180424, receive=/backups/server1/opt) At subvol /mnt/btrfs_root/_btrbk_snap/opt.20180424
ERROR: cannot open backups/server1/opt/opt.20180423/owncloud/_data/mariadb/nextcloud/oc_activity.ibd: No such file or directory
WARNING: [send/receive] (send=/mnt/btrfs_root/_btrbk_snap/opt.20180424, receive=/backups/server1/opt) At snapshot opt.20180424
ERROR: Failed to send/receive btrfs subvolume: /mnt/btrfs_root/_btrbk_snap/opt.20180424 [/mnt/btrfs_root/_btrbk_snap/opt.20180423] -> target:/backups/server1/opt
send/received failed, deleting (possibly present and garbled) received subvolume: target:/backups/server1/opt/opt.20180424

Deleting the problematic snapshots helps and the next 5-10 runs succeeds but the error come back after some time.

Probably related to #91

Kernel 4.14.8-1.el7.elrepo.x86_64 btrfs-progs v4.9.1 btrbk command line client, version 0.26.1

Does anyone have an idea why oc_activity.ibd is problematic?

[digint]

Yes, it's related/duplicate of #91: upstream bug, nothing that can be fixed in btrbk.

The only advice I can give here is "try a newer kernel". If it's still reproducible, the btrfs devs might be interested.

For the record: what kernel / btrfs-progs version do you use on the target machine?

[psi-4ward]

Target: 4.11.2-1.el7.elrepo.x86_64 btrbk 0.25 btrfs-progs v4.9.1

I'll try to update everything. Thx a lot!

[psi-4ward]

Updated src and target and created a full snapshot by hand. Let's see if the problem occurs again.

[aventrax]

Same here. It happens only on graphite .wsd files. It seems related to changing files as databases's ones. The error appear after a while (7 days?) than it persists for any futher run, until I remove the source (remote) snapshot. Then after some days the cycle restarts...

Here 2 errors on different days...

ERROR: cannot open storage/Backup/snapshots/@.20180601/var/lib/graphite/whisper/collectd/rpi/disk-sda1/pending_operations.wsp: No such file or directory

ERROR: cannot open storage/Backup/snapshots/@.20180525/var/lib/graphite/whisper/environment/home/bedroom/temperature.wsp: No such file or directory

The same volume has PostgreSQL files but they are not affected.

[psi-4ward]

Problem still occurs

[psi-4ward]

anything new here? still happens on 5 of 7 servers.

[digint]

@psi-4ward did you try new kernels? There has been some fixes (seen on the btrfs mailing list, but can't find it right now) which might have arrived in 4.19 / 4.20. Most probably you only need to upgrade the source side.

[psi-4ward]

Some updates here, problem still occurs even with kernel 5

[yarikoptic]

I think my case also relates to this?

/etc/cron.daily/btrbk:                                                                                                                                                                                                            
WARNING: Target subvolume "drogon10:/mnt/backup/smaug/home/home.20240605T0000" exists, but is not a receive target of "/mnt/btrfs/.snapshots/home.20240605T0000"                                                                  
WARNING: Please delete stray subvolumes: "btrbk clean drogon10:/mnt/backup/smaug/home"                                                                                                                                            
WARNING: Skipping backup of: /mnt/btrfs/.snapshots/home.20240605T0000                                                                                                                                                             
ERROR: Failed to send/receive subvolume: /mnt/btrfs/.snapshots/datasets.20240605T0000 [/mnt/btrfs/.snapshots/datasets.20240604T0000] -> drogon10:/mnt/backup/smaug/datasets/datasets.20240605T0000                                
ERROR: ... Command execution failed (exitcode=1)                                                                                                                                                                                  
ERROR: ... sh: btrfs send -p '/mnt/btrfs/.snapshots/datasets.20240604T0000' '/mnt/btrfs/.snapshots/datasets.20240605T0000' | ssh -i '/root/.ssh/id_rsa' root@drogon10 'btrfs receive '\''/mnt/backup/smaug/datasets/'\'''         
ERROR: ... chown datalad/ci/datalad-extensions/.git/index failed: No such file or directory                                                                                                                                       
ERROR: ... failed to check target subvolume: drogon10:/mnt/backup/smaug/datasets/datasets.20240605T0000                                                                                                                           
ERROR: ... Command execution failed (exitcode=1)                                                                                                                                                                                  
ERROR: ... sh: ssh -i '/root/.ssh/id_rsa' root@drogon10 'btrfs subvolume show '\''/mnt/backup/smaug/datasets/datasets.20240605T0000'\'''                                                                                          
ERROR: ... cannot find real path for '/mnt/backup/smaug/datasets/datasets.20240605T0000': No such file or directory                                                                                                               
ERROR: Error while resuming backups, aborting                                                                                                                                                                                     
ERROR: Failed to delete subvolume: drogon10:/mnt/backup/smaug/home/home.20240514T0000                                                                                                                                             
ERROR: ... Command execution failed (exitcode=1)                                                                                                                                                                                  
ERROR: ... sh: ssh -i '/root/.ssh/id_rsa' root@drogon10 'btrfs subvolume delete '\''/mnt/backup/smaug/home/home.20240514T0000'\'''                                                                                                
ERROR: ... Could not statfs: No such file or directory                                                                                                                                                                            
WARNING: Skipping cleanup of snapshots for subvolume "/mnt/btrfs/home", as at least one target aborted earlier                                                                                                                    
WARNING: Skipping cleanup of snapshots for subvolume "/mnt/btrfs/datasets", as at least one target aborted earlier                                                                                                                
--------------------------------------------------------------------------------                                                                                                                                                  
Backup Summary (btrbk command line client, version 0.32.5)

? client kernel 6.5.0-0.deb12.4-amd64 , server 6.1.0-0.deb11.11-amd64