yutayu
commented
3 years ago @Heiko-L My recognization is resync is just only sync , not for freeing memory.
Open Heiko-L opened 3 years ago
yutayu
commented
3 years ago @Heiko-L My recognization is resync is just only sync , not for freeing memory.
manorit2001
commented
2 years ago Maybe I don't understand it, but shouldn't everything be copied from the upper to the lower dir after the timer expired so that the overlayfs is "flushed" for lower memory consumption?
Ideally, this should be it but while running tests with clearing the upper directory. There seems to be some issues of data losses due to the internal caching mechanism of overlayFS. It doesn't update the content of the lower directory deterministically as the logic dictates that the lower directory is supposed to be read-only as per overlayFS. So if we try to copy all the content to lower directory and free the upper directory, there seems to be some inconsistencies in the directory mounted with overlayFS. I hope I come across some solution by which I'll be able to implement it but as of now am afraid i can't do much.
mox-mox
commented
2 years ago Hi, I believe this data loss has hit me. To be safe(er) for the future, should/could I disable the overlayfs mode?
manorit2001
commented
2 years ago No, I haven't implemented this so the data loss must not be due to this. The data loss can always occur while using ASD depending on what kind of data loss you're talking about. Think of ASD as a volatile memory with a timeout for syncing back stuff ( by default it is set to one hour ). Suppose you were working on something for the period of one hour and some power cut off occurred before the sync back ( not a proper shutdown ). Now if this occurs the data that was in the volatile memory wouldn't sync back to the disk. However, asd tried to keep the last synced backup intact and is focused on that part. I had created a lot of checks to try to minimize this data loss and all to keep the data intact from the last backup. Have you lost all your data or is it just the volatile data that am talking about? Also, are you on the latest version of ASD or some older version?
mox-mox
commented
2 years ago My current version of ASD is 6.0.0-1. The data loss might have ocured with the previous version, I'm not sure when the Archlinux repo was last updated. My setup involves one folder (scratch) where I do most of my development stuff. In the past few years I've had several unclean shutdowns (partly Out of Memory Exceptions) and have learned to recover my data (minus the last hour of course). A few months back, I've decided to try the overlayfs feature, so I uncommented the respective line in /etc/asd.conf to yield
WHATTOSYNC=('/home/mox/scratch' '/home/mox/projects/hardware/dodgy') # The 'dodgy' folder is irrelevant here because I have not touched it for quite a long time.
USE_OVERLAYFS="yes"Everything else was commented out. Over the last week(s), I've been working on some project. I did not turn off my computer but I have suspended it serval times. Then, I did a system update (including new kernel if that is relevant) and restarted the computer. When I looked into the scratch folder, the data there was older. At least a weeks worth of code was gone. So naturally, I've looked for a crash recovery snapshot. There were two snapshots (.scratch-backup_asd-crashrecovery-20220626_015926.tar.zstd and .scratch-backup_asd-crashrecovery-20220726_105434.tar.zstd). The older snapshot is too old to be relevant, so I've looked into the newer one. The code in there is quite a bit older than what I've had before the restart.
Next, I've looked into the journal to see if asd has reported any errors. It seems to me it did but I'm not sure how to interpret them.
The cropped output of journalctl --no-pager -u asd: (full output was much too long for a comment)
and
Thank you for taking time to respond and look into this. ASD is a wonderfull piece of software :)
manorit2001
commented
2 years ago I think I can guess what happened, thought I don't know why the I/O errors are occurring in rsync, that usually doesn't occur or I haven't seen it happen. Mostly faulty harddrives might cause that as per few google searches on it. Though regarding the data loss, I think I maybe have a hint on what might've happened. The new version of asd contains a major change which consist of changing the symlinking method of syncing to bind mounting which changed a lot of stuff in the codebase ( hence version upgrade to 6.0 ). Now since you updated the system, your initial setup was done through the previous version but once upgraded, the de-sync happened with the newer version which assumed that bind mounts was used as the setup mechanism. Though it was fault on my part that I didn't know asd install script in the arch repository had some hardcoded offsets for the version changes in which asd would stop ( i recently became a maintainer of this repository ).
$ cat anything-sync-daemon.install
[...]
pre_upgrade() {
for version in 5.69 5.76; do
if [ "$(vercmp "$2" "$version")" -lt 0 ]; then
_stop_asd_service
echo 'Start the asd service manually after upgrade is complete.'
fi
done
}
[...]Though once I got to know about it I had updated @graysky2 about this change and mentioned him to update the arch scripts to stop asd before any upgrade ( regardless of the version numbers ) though I can still see the old scripts only available in the repo.
mox-mox
commented
2 years ago Wow. That's quite a comprehensive analysis. If I understand this correctly, ignoring the I/O errors my data loss was mainly due to the (more or less faulty) update process in the Arch Linux package and this will soon be fixed. Thank you very much! :)
To reproduce:
Maybe I don't understand it, but shouldn't everything be copied from the upper to the lower dir after the timer expired so that the overlayfs is "flushed" for lower memory consumption?