Add sync option

[mstormi]

Although not recommended syncing without unmounting may be desirable and works in many cases so it should be available as an option.

[StuartIanNaylor]

I am not sure as I did speak to the OverlayFS and they where adamant that a online merge down could take the merge layer out of sync.

I need to do some updates as 4.19 has seen a lib name change from xatrr to attr so the merge requires update. I guess it could be added but would have to make a disclaimer.

[GenTarkin]

I just tried implementing this change on latest commit in my own fork of this repo. The result was the merge script deleted the ./upper directory and hosed the overlayfs upper mount point =(

Heres the snippet from zram-config log showing the action taken by the overlayfs-tools merge script:

• mv -T /opt/zram/zram1/upper/daemon.log /opt/zram/log.bind/daemon.log
• rmdir /opt/zram/zram1/upper

[mstormi]

:( Do you still have the log ? Any idea what went wrong ?

[GenTarkin]

:( Do you still have the log ? Any idea what went wrong ?

See the update in my first reply =) ... quoted from the log file, what looks like the culprit. To further clarify, the last line shown in the snippet.

[mstormi]

Hmm so it's the overlayfs-tools who wrongfully want to delete the main (upper) directory ? At least I wouldn't think that's ever meant to be deleted but I'm not the expert. Do you think we can file this as an issue with the overlay tools author/repo ? Could you reproduce or narrow it down somehow?

[GenTarkin]

Well, since they say merge while mounted is officially unsupported ... I doubt we would get any traction. Perhaps this is one of the reasons why =P Ive been pondering developing a file semaphore driven safe "Stop/Start" merge which would not mess w/ any swap entries. (if semaphore exists, skip swap, delete semaphore) The only problem I can see potentially being data written to logs(or other files folders for that matter) while "Stop" merge script is moving items objects from upper to lower. But, I guess even now that can potentially be a problem.

But I do notice another thing thats odd, if I stop service, /dev/zram1 seems to hang around in a limbo state and then when I start the service again(not reboot) ... there are 3 zram devices, 2 populated, the one unpopulated and unmounted is zram1... seems like zram1 is "stuck" ... but I guess everything is still working fine.

[GenTarkin]

I went ahead and logged an issue w/ overlayfs dev/repo. I reread the "info" posted there and the behaviour specified is that the upper should be left "empty" after a merge ... not outright deleted. So yeah, I think this is definitely a bug. In the meantime, perhaps sed or something can be used to post process the .sh file(to remove this line) prior to running it?

[GenTarkin]

Well, I tried the sed approach and while it worked as it intended ... the result is overlayfs did not like it at all. Since doing the successful sync ... while the "upper" dir still remains ... no files written to, ever go there anymore ... in fact I dont know where they are going lol! so yeah, it seems the "copy on write" to previously merged files since overlayfs was mounted, is completely borked when doing this sync approach.

[GenTarkin]

Well, Ive implemented, in my repo, the semaphore based "skip swapfile" approach for stop / start commands. Seems to be good enough for now, This way I can stop/start every half hour using crontab to get lower sync'd safely w/o interrupting swap. Im sure theres a better way to do this, was the easiest I could come up with for now lol!

[GenTarkin]

Well, I guess it doesnt work as well as Id hoped. Because I have it ignoring swap entries, they dont get to devicelist file =( on normal stop / starts. Grr... need to figure out logic to save current swap entry upon a skip swap stop / start

[GenTarkin]

Back to the overlayfs-tools "merge" sync idea.... I wonder if doing a remount command after the merge would restore full overlayfs functionality. Hrm...

[mstormi]

Good idea, but we need to find out why/when the tools misbehave. So they work when unmounted but fail when mounted ? Eventually it's as you say just a bug to remove upper dir that just does not show up because it's unmounted already at that stage.

[GenTarkin]

Sorry for all the updates, but another update on the "merge/sync" approach.... After some experimentation, it seems these are the results:

• doing a remount of TARGET_DIR after the merge seems to have the result of overlayfs refreshing lower and upper contents accurately enough for "dir" type.
• doin the same remount of TARGET_DIR after merge on "log" type (against /var/log) seems to result in ... files which were merged previously from upper to lower, are no longer recreated in upper when written to and I cant seem to find any evidence of them even being updated in lower... I do see the original /var/log folder being updated tho, so I dont know if overlayfs is being completely bypassed or if the filesystem of overlayfs just not showing updates properly. I can see that "watch zramctrl" shows the zram drive assigned to "/opt/zram/zramN/ ... has writes going to it ... I just see any evidence of that in ANYTHING in the "zramN" folder lol! All that being said, stop/start work and logs seem intact ... the only question is how far will logs be rolled back if I power fail the device.(since I have no idea really where these writes are going to) lol

[GenTarkin]

Good idea, but we need to find out why/when the tools misbehave. So they work when unmounted but fail when mounted ? Eventually it's as you say just a bug to remove upper dir that just does not show up because it's unmounted already at that stage.

I dont think "unmounted vs mounted" state is the tools fault .... I think overlayfs has a problem with the lower changing at all while the filesystem is mounted as such. I've read similar sentiments elsewhere and some saying aufs or similar should be used instead. Although, I DO think the removal of "upper" dir is a bug because its not functioning as technically advertised.

[mstormi]

the only question is how far will logs be rolled back if I power fail the device.(since I have no idea really where these writes are going to

Did you remember or see that old logs - unlike "dir"s - can be rotated to a standard directory ?

[GenTarkin]

the only question is how far will logs be rolled back if I power fail the device.(since I have no idea really where these writes are going to

Did you remember or see that old logs - unlike "dir"s - can be rotated to a standard directory ?

I dont follow what you are asking =/

[mstormi]

If one configures a directory of "log" type, files in there are moved to a normal (non-zram) directory although I don't know the exact mechanism when this happens. "oldlog_dir", last column in /etc/ztab. Maybe that's related to your "disappearing" writes.

[GenTarkin]

If one configures a directory of "log" type, files in there are moved to a normal (non-zram) directory although I don't know the exact mechanism when this happens. "oldlog_dir", last column in /etc/ztab. Maybe that's related to your "disappearing" writes.

Well I check the referenced old_dir in my config and it doesnt seem to have anything relevant to this really. The missing writes definitely arent there, from what I can tell (at least going off file timestamps when comparing /var/log vs /opt/zram/oldlog)

[mstormi]

The missing writes definitely arent there

I had a tail -f running on a logfile on zram and noticed that unmounting caused a whole lot of lines to be appended.

[alpe12]

So, any way I can periodically sync only one zram folder?