search

spion / adbfs-rootless

Mount Android phones on Linux with adb. No root required.
Other
908 stars 73 forks source link

How to preserve date created, date modified date accessed of folders? #56

Open anmac1789 opened 2 years ago

anmac1789 commented 2 years ago

How do I preserve using this method of folder date created, date modified, date accessed of android phone when it is connected to my PC? I want to copy folders inside android to PC preserving all these dates exactly

IzzySoft commented 1 year ago

Just noticed this now – seems to be the same issue I have in #59 just the other way around (you ask about "from device" aka adb pull, I about "to device" aka adb push).

anmac1789 commented 1 year ago

Just noticed this now – seems to be the same issue I have in #59 just the other way around (you ask about "from device" aka adb pull, I about "to device" aka adb push).

Omg last time i posted was april last year !!!! it took so long for a reply i gave up. Is it even possible anymore ?

IzzySoft commented 1 year ago

Is it even possible anymore ?

I cannot tell (I maybe can read a little C++ but certainly not write). @spion is still active (though maybe on different projects), so I hope the details on my issue plus the fact yours existing as well and even for a year now, most likely being fixed by whatever fixes the other one (thus killing 2 issues with 1 patch) might trigger him to give this project some love again.

anmac1789 commented 1 year ago

Is it even possible anymore ?

I cannot tell (I maybe can read a little C++ but certainly not write). @spion is still active (though maybe on different projects), so I hope the details on my issue plus the fact yours existing as well and even for a year now, most likely being fixed by whatever fixes the other one (thus killing 2 issues with 1 patch) might trigger him to give this project some love again.

is he even developing the software ??

spion commented 1 year ago

I've been thinking about switching this project to Rust, as I'm not very good at writing safe C++ code (in fact, I'm quite slow at writing C++ in general, which has made it very difficult to find the time to work on it).

One idea to help this move further is to try ./adbfs -f - that should show all the commands that adbfs runs. With that log we could try and identify what the differences are between adbfs and a normal push/pull more easily, which should make it easier to patch them up.

anmac1789 commented 1 year ago

I've been thinking about switching this project to Rust, as I'm not very good at writing safe C++ code (in fact, I'm quite slow at writing C++ in general, which has made it very difficult to find the time to work on it).

One idea to help this move further is to try ./adbfs -f - that should show all the commands that adbfs runs. With that log we could try and identify what the differences are between adbfs and a normal push/pull more easily, which should make it easier to patch them up.

What is rust ? is this related somehow to adbfs ?

spion commented 1 year ago

Another note: it seems like fuse has moved on - we now have fuse3, which is what most distros use. I tried porting over some of the changes that maz-1 did to get it to work on fuse3 (https://github.com/maz-1/adbfs-rootless) but AFAICT the timestamps appear to be completely lost.

spion commented 1 year ago

Rust is a programming language (i.e. I'd like to rewrite this project in Rust)

Its somewhat of an upfront cost but after that development should be much more of a breeze.

IzzySoft commented 1 year ago

Ah, and there's a difference indeed:

--*-- exec_command: adb push '/tmp/adbfs-aw7waR/-storage-emulated-0-Podcasts-cinnamon-restart' '/storage/emulated/0/Podcasts/cinnamon-restart'

adbfs gives the target name explicitly. But not only that, it does a bunch more things I have no idea why.

Log

--*-- exec_command: adb push '/tmp/adbfs-aw7waR/-storage-emulated-0-Podcasts-cinnamon-restart' '/storage/emulated/0/Podcasts/cinnamon-restart'
invalidate cache /storage/emulated/0/Podcasts/cinnamon-restart
--*-- exec_command: adb shell "sync"
invalidate cache /storage/emulated/0/Podcasts/cinnamon-restart
adb_getattr
--*-- exec_command: adb shell "ls -l -a -d '/storage/emulated/0/Podcasts/cinnamon-restart'" 2>&1
-- adb_open --/storage/emulated/0/Podcasts/cinnamon-restart /tmp/adbfs-aw7waR/-storage-emulated-0-Podcasts-cinnamon-restart
ls -l -a -d '/storage/emulated/0/Podcasts/cinnamon-restart'
--*-- exec_command: adb shell "ls -l -a -d '/storage/emulated/0/Podcasts/cinnamon-restart'"
--*-- exec_command: adb pull '/storage/emulated/0/Podcasts/cinnamon-restart' '/tmp/adbfs-aw7waR/-storage-emulated-0-Podcasts-cinnamon-restart'
flag is: 0
invalidate cache /storage/emulated/0/Podcasts/cinnamon-restart
--*-- exec_command: adb push '/tmp/adbfs-aw7waR/-storage-emulated-0-Podcasts-cinnamon-restart' '/storage/emulated/0/Podcasts/cinnamon-restart'
invalidate cache /storage/emulated/0/Podcasts/cinnamon-restart
--*-- exec_command: adb shell "sync"
Looks like it pushes it multiple times, and in between also pulls it (I actually copied it twice as the first time I had no output, using the same session in a Midnight Commander window – but I definitely did not pull it, I just removed it in between). And finally, the last command is resolving the riddle: ``` touch '/storage/emulated/0/Podcasts/cinnamon-restart' ``` **WHY?** It's not the way `adbfs` *pushes* the file – it explicitly sets the time stamp to "now" with an explicit final `touch` :scream: The only mention of `touch` I find in `static int adb_utimens(const char *path, const struct timespec ts[2])` – but I don't see where that is called; due to `adbfs_oper.utimens = adb_utimens;` my guess is that's called by FUSE. Doesn't FUSE pass the timespec along? And not fully related, why is that timestamp increased by `50` (according to the code) – or rather not passed at all (according to the debug output)? For reference: ```cpp static int adb_utimens(const char *path, const struct timespec ts[2]) { string path_string; path_string.assign(path); fileData[path_string].timestamp = fileData[path_string].timestamp + 50; shell_escape_path(path_string); queue output; string command = "touch '"; command.append(path_string); command.append("'"); cout << command<<"\n"; adb_shell(command); // If we forgot to mount -o rescan then we can remount and touch to trigger the scan. if (adbfs_conf.rescan) adb_rescan_file(path_string); return 0; } ``` Where is it passed? It's calculated by `fileData[path_string].timestamp = fileData[path_string].timestamp + 50;` – but where is that `fileData` used in the executed command? Further, the variable passed is called `timespec` (or `ts`? I'm a bit confused there) – I don't see that referenced and being used inside.
IzzySoft commented 1 year ago

PS: While I understand a rewrite in Rust, that's not installed on my machine with the default build tools. Checked with my package manager, no idea what packages I'd need for that – but it would be quite an extra for non-devs. Would you then provide binaries?

As for "timestamps lost" – maybe there could be an option turning adb_utimens into a stub, so the final touch is never executed? So if one just wants to run a sync (like me), one could simply mount with that option (e.g. adbfs -t droid) for that task. adb push deals fine with timestamps to my experience, so no touch would be needed for that.

anmac1789 commented 1 year ago

Okay so in comparison from what I understand, in NTFS windows, if you have a parent older with subfolders,changing anything inside the subfolders changes the date modified and date accessed but leaves the parent folder timestamps alone. If somehow, this could be tweaked with android to replicate the same behaviour for subfolders then we have a soution where the date and time won't change if you make changes within a subfodler. Mind you a few notes-- this COULD only works for parent folders not subfolders. Any considerations of my idea?

spion commented 1 year ago

I pushed an update to feat/merge-libfuse-3 as it looks like @maz-1 has done an incredible job at fixing some of these issues by passing the times to the touch command by default. I think I also managed to save the timestamps by relaxing the restrictions for their parsing a little.

https://github.com/spion/adbfs-rootless/tree/feat/merge-libfuse-3 is the new code - note that there is a slight change in the readme too, as this has been upgraded for fuse3 rather than fuse.

anmac1789 commented 1 year ago

I pushed an update to feat/merge-libfuse-3 as it looks like @maz-1 has done an incredible job at fixing some of these issues by passing the times to the touch command by default. I think I also managed to save the timestamps by relaxing the restrictions for their parsing a little.

https://github.com/spion/adbfs-rootless/tree/feat/merge-libfuse-3 is the new code - note that there is a slight change in the readme too, as this has been upgraded for fuse3 rather than fuse.

so all timestamps are preserved ?

spion commented 1 year ago

Gave it my best shot to test the end result, to the extent of my knowledge it looks OK. But please give it a try yourself. I could have missed something. (try the branch)

anmac1789 commented 1 year ago

Gave it my best shot to test the end result, to the extent of my knowledge it looks OK. But please give it a try yourself. I could have missed something. (try the branch)

Would it be okay if you could provide a screenshot as a test ?

IzzySoft commented 1 year ago

@spion that Readme needs another update I'm afraid:

fuse: failed to exec fusermount3: No such file or directory

And now, if I want to apt install fuse3, that will remove fuse. Not sure if that'd break something else depending on fuse (I don't know what I might have installed depending on it)… Looks like nothing in my case (else apt should have refused if it was installed via it), but for others:

 ntfs-3g depends on fuse.
 ifuse depends on fuse.
 gvfs-fuse depends on fuse.
 exfat-fuse depends on fuse.

However, for existing scripts using e.g. fusermount, there's /usr/bin/fusermount -> fusermount3 then as a "compatibility layer" (automatically created by fuse3). So the Readme lacks some hints on that. I'd say amend here:

sudo apt-get install libfuse3-dev android-tools-adb
sudo apt-get install build-essential git pkg-config
sudo apt-get install fuse3

and point out the above (e.g. "Note that you cannot do so if you use… In that case…"). Not sure what should be done "in that case", e.g. when ntfs-3g is being used. Preserve current master as branch?

OK, I dared doing it. But timestamps are still destroyed, sorry (just seems so, but read on). Though the output shows a proper touch this time, and that time seems to be used:

--*-- exec_command: adb shell "touch -am -t 201702042223.39 '/storage/emulated/0/Podcasts/deentity'" 2>/dev/null
..
caching /storage/emulated/0/Podcasts/deentity = -rw-rw----  1 root sdcard_rw  657 2017-02-04 22:23:39.000000000 +0100 deentity

… Oh. And a few seconds later that is reflected – so it just takes a little, and Midnight Commander was too fast. To reproduce: this time I used 2 terminal windows, let's call them T1 and T2. In T2 I use Midnight commander.

So it seems like this would solve #59 then. I certainly could have been less verbose (simply stating the fuse3 thing and confirm it's working then) – but @anmac1789 wanted some POW (proof-of-work), so having hi-jacked his issue I felt obligated :see_no_evil:

That said, maybe have the Readme mention the delay for proper timestamps? I'll meanwhile replace my working adbfs binary with the new build from the branch and keep an eye on it. Will yell should I find something weird in my daily usage then :wink:

spion commented 1 year ago

And now, if I want to apt install fuse3, that will remove fuse. Not sure if that'd break something else depending on fuse (I don't know what I might have installed depending on it)… Looks like nothing in my case (else apt should have refused if it was installed via it), but for others

Strange, on Ubuntu I got the exact same opposite message, as everything there seems to depend on fuse3. As far as I could tell, fuse3 has been the preferred way to go since ~2016-2017

Perhaps installing libfuse3-3 and libfuse3-dev should be enough to compile the new version without causing disruption ?

IzzySoft commented 1 year ago

For compiling yes, but if you want to use adbfs then it complains as it seems to explicitly call fusermount3 (would it call fusermount, with fuse3 the symlink would take care for it – but I'm not sure if that would work with the old fuse package installed – nevermind, nothing you can change as that call doesn't come from your code, at least I couldn't find it there).

I also had to (at least temporarily) switch back to master as with the new adbfs from the branch I was unable to mount a device in need of some "urgent maintenance" (will investigate that later, when time permits; IIRC that one is still on Android 9, while the one I successfully tested with was on Android 10). Not being able to have fuse and fuse3 installed in parallel gives another headache: whenever I want to switch, that implies re-generating initrd.img, taking a bit longer than being comfortable.

Btw: I'm on Linux Mint 20.3, which is not the very latest; it's still based on Ubuntu 20.04, which could explain the difference.

spion commented 1 year ago

You should be able to use sudo umount ~/path/to/x right? (instead of fusermount3)

IzzySoft commented 1 year ago

Please look again, Gorgi: I'm not able to MOUNT. It complains on adbfs /droid if fuse3 is not installed. So something is explicitly calling to fusermount3. Not me, not adbfs – I suspect it's the fuse3 libraries. But well, let me do a quick hack test… Ha! Great, works :rofl:

sudo ln -s /usr/bin/fusermount /usr/local/bin/fusermount3

(actually, I rather linked it into $HOME/bin – can be anything that's in the $PATH). Excellent. So no need to install fuse3, creating a symlink suffices. Good, so I can continue testing with a little ASAP unit, just switching binaries. Thanks for the hint! (If you wonder: it was your "instead of fusermount3" triggering the idea of just using a symlink to "instead using the other one" :rofl:)

PS: my test results might be tainted then, though. If something does not work as expected, the cause could be the wrong fusermount. I'll have to leave that to you then for interpretation.

PPS: that said, timestamps work the same as described above (i.e. with a slight delay).

spion commented 1 year ago

Sorry, I might make a few mistakes and miss a point or two! :grinning: but I try not to make claims when I'm unsure (just ask questions).

Ok. I will need some time to figure out the fuse/fuse3 dichotomy in terms of code (does it bring anything new to the table. I think it may be possible to backport the fix without upgrading to fuse3, but to test all that I'll either need to make a mess out of my system or prepare some docker containers with various versions of libraries + adb + android emulator :sweat_smile:

spion commented 1 year ago

Funny enough in rust land fuser (https://github.com/cberner/fuser/#linux) is compatible with both fuse and fuse3, whatever you have on your system :grinning: I already have a hello-world (non-ADB) filesystem working locally and had absolutely no trouble building it with rustup (I understand though that installing something like rustup brings a great difference in comfort for many situations... sigh)

IzzySoft commented 1 year ago

Yeah, I think it's rather an "error" (not sure if "bug" is to harsh here) on libfuse3's end. For now, I've set the latest build (fuse3 based) back active, with the symlink in place, and will keep on testing whenever I find something suitable. Until now I just successfully mounted an Android-10 device (Wileyfox Swift with LOS), for the last of the above experiments. From that I assume Android 10+ should work with it – give me a sec…

Nope. My Android-11 device (FP2 running LOS) does not mount, but I get no useful information via adbfs -f. My other Android-10 device (SHIFT6mq running SOS-L, their official Google-free ROM) mounts well, too – and -f output looks identical to that of the non-working 11. What's even more irritating is the syslog, which claims "rootless-droid.mount: Succeeded." despite the mount point being broken. It claims so whether it succeeded or not. Same for journalctl. Guess for my daily work I'll again have to go back to the pre-fuse3 build :cry:

I think it may be possible to backport the fix without upgrading to fuse3

With the fuse3 build not working on half of my devices, I can give that one some test runs as well. For "critical work" and can always switch back to the last "stable" build – I simply do a symlink -sf <binary> adbfs so I don't have to worry about breaking scripts etc.

spion commented 1 year ago

Hi @IzzySoft - I decided to go the docker + tests route and to try and keep fuse to version 1, at least for now. To that end I opened a PR where suggestions for the kind of tests would be helpful https://github.com/spion/adbfs-rootless/pull/62

Lets add some good timestamp tests, as well as brainstorm on some tools that we'd like to ensure continue to work (so long as they don't take too long). I would very much welcome input from prolific users :grinning:

IzzySoft commented 1 year ago

I gladly assist with testing new versions – but I have neither docker nor podman set up here (planned for years to take a look at but never found time).

Some things I'd have in mind, apart from "basic timestamp checks": rsync, both directions (and maybe compared against rsync of the very same tree on a "pure local" FS). Compared to adb push --sync as well maybe (you'll be shocked how much faster that one is; rsync with an adbfs mount as source is pretty slow especially with many files in a directory (here: TiBu/Seedvault backup locations), and pushing plus fixing timestamps is slow as well (which is why I was happy to find the --sync option to adb push working fine – in order to push with rsync you can override that by setting the delete flag wich is only supported by rsync)).

For automation, it might be considerable to integrate my adbsync which lets you easily define a bunch of sync pairs in JSON, grouped by devices, and toggle sync pairs or entire devices ("sync groups") on and off with a single 0=>1=>0. Sync pairs can be from or to device, or even bidirectional (experimental, but that shouldn't be a deal-breaker here).

anmac1789 commented 1 year ago

So when can we begin with the testing phase?

I gladly assist with testing new versions – but I have neither docker nor podman set up here (planned for years to take a look at but never found time).

Some things I'd have in mind, apart from "basic timestamp checks": rsync, both directions (and maybe compared against rsync of the very same tree on a "pure local" FS). Compared to adb push --sync as well maybe (you'll be shocked how much faster that one is; rsync with an adbfs mount as source is pretty slow especially with many files in a directory (here: TiBu/Seedvault backup locations), and pushing plus fixing timestamps is slow as well (which is why I was happy to find the --sync option to adb push working fine – in order to push with rsync you can override that by setting the delete flag wich is only supported by rsync)).

For automation, it might be considerable to integrate my adbsync which lets you easily define a bunch of sync pairs in JSON, grouped by devices, and toggle sync pairs or entire devices ("sync groups") on and off with a single 0=>1=>0. Sync pairs can be from or to device, or even bidirectional (experimental, but that shouldn't be a deal-breaker here).

IzzySoft commented 11 months ago

Just stumbled upon this issue again when backtracing why I was using -c (checksum verification) with rsync in my scripts, which slows down the process significantly… Seems this got stuck again due to lack of testers I guess? Or is there any "local" progress that simply didn't get pushed yet, @spion?

spion commented 11 months ago

Its due to lack of automatic tests, mainly. The tests PR I wrote works locally, its just that GitHub doesn't support the virtualization necessary to run android emulator tests in their free runners. I've been watching the issue from time to time and while they did add virtualization to paid runners, there is nothing for free ones yet.

I'd really like it for GH to enable this feature. Running a test matrix of multiple android versions locally is painful and it would be much nicer if these could automatically run for other people's PR contributions too. But since its not happening I can try and revisit this if I find some time in the following period.