Files modification time as create time, expect it as original modification time.

catscarlet commented 6 years ago

Hi. I'm using some software to sync all my files on most of my devices. (such as Freefilesync, Beyond Compare) It detect files modification time to check if a file has been edited.

I want to use prim-ftpd to make my android devices to be a sftp-server, but I find out when I copy a file from my computer to my phone through sftp, the files modification time is as its create time(aka now), not the original modification time, that causes Freefilesync thinks it's a new version of file.

Can you fix it?

Android version: 4.4 prim-ftpd version: 5.8

catscarlet commented 6 years ago

Is that a system behavior or what? I only get this problem on my android devices.

wolpi commented 6 years ago

I'm not aware of an app could control file creation time. I think java file API does not expose that. But I also realize that in the other direction. When downloading files from phone their creation-time on PC is download-time not original creation-time.

catscarlet commented 6 years ago

Don't care about the creation-time. That one is untrustworthy. Do care about modification-time pls.

wolpi commented 6 years ago

I'll have to check if file API allows to set it.

catscarlet commented 6 years ago

Tried on Android version: 7.1.2, and still got this problem Plain old filesystem seems to be read only and can't get tested.

catscarlet commented 6 years ago

Tried on Android version: 8.0 with SAF, still got this problem. Plain old filesystem seems to be read only and can't get tested.

I found something may be useful, you should check if you have time: https://forum.xda-developers.com/android/general/guide-timestamp-attributes-correct-t2960935

Seems it's a system issue about Androrid filesystem, every tool about backups or syncing not works fine on Android because of this.

Seems got fixed in Android 8.0, but with the test on prim-ftpd, it seems not.

Waiting for your response.

catscarlet commented 6 years ago

Something more about my digging out:

I installed an app called 'Ftp Server'(GP: https://play.google.com/store/apps/details?id=com.theolivetree.ftpserver&hl=en) , and tested:

On Android 8.0, the modification timestamp is correct!
On Android 7.1.2, the modification timestamp is as wrong as the create time.

Surely the timestamp problem belongs to the Android system 7.1.2 and lower because of FUSE, and seems we have no chance to fix it unless we got ROOT.

I wish you can fix this problem for prim-ftpd only on Android 8. Oreo uses 'SDCardFS' so it should be possible to keep the files' timestamp and attributes correct.

Waiting for your response.

wolpi commented 6 years ago

Java File API does not provide a way to set modification time. For this app that means plain old filesystem. SAF API does not allow it, too. The xda link you posted contains some hints how this could be done with linux commands. The app could launch those commands as external processes. That should work for plain old filesystem but probably not for SAF.

Another thing to check is how to get original modification time from ftp/sftp server libraries.

Interesting that com.theolivetree.ftpserver is able to do this. Would be nice to know how they implemented it.

wolpi commented 5 years ago

I found time to play a little with this. It seems that most clients do not send correct mtime. Can you tell me more about your setup:

which protocol do you use (ftp/sftp)?
which client?
which storage driver (filesystem, SAF, root)?

catscarlet commented 5 years ago

Phone: Sony XZ1 G8342 47.1.A.12.235 Android version: 8.0.0 Kernel 4.4.78-perf+ (The same device as https://github.com/wolpi/prim-ftpd/issues/145#issuecomment-404080907 , )

Client:

Filezilla (free and opensource, supports Win, Linux and Mac. If you don't have a credible ftp/sftp client, try this pls) (Notice: The option on menu_Transfer_Preserve timestamps of transferred files must be checked)
BCompare
SecureFX
FreeFileSync (This is not a sftp client but a file sync tool which supports sftp protocol)

They all send correct mtime when connect to most of linux sftp servers (based on openssh/sftp-server)

Storage Type:

Plain old filesystem (This one can't get all access to the external SD because there is not a chance to get it)
SAF (SAF can get the write access when choosing SD card at start)

I don't have rooted 8.0.0 device.

Case One:

Protocol: sftp

Plain old filesystem result: Failed. Modification time setted as creation time.
SAF: Failed. Modification time setted as creation time.

Case Two

Protocol: ftp

Plain old filesystem result: Limited Succeed. Modification time setted as right as it should be when writing file into internal storage. Can't write file into external SD anyway.
SAF: Failed.No matter internal storage or external SD, modification time setted as creation time.

Well, I guess this is a omission because of ftp's terrible efficiency. prim-ftpd can work as good as com.theolivetree.ftpserver when using ftp.

wolpi commented 5 years ago

As you might see in git commits I made some changes for this:

Setting of mtime for sftp was completely missing, has been added
Setting of mtime for root was missing and has been added
Some other bugs in sftp where found and fixed
Setting of mtime for SAF still does not work :cry:

catscarlet commented 5 years ago

If setting of mtime for SAF still does not work , can we have a chance to giant the access of external SD when using Plain old filesystem?

wolpi commented 5 years ago

access of external SD when using Plain old filesystem?

Short answer: no, it is blocked by the operating system.

Here is some documentation from google: http://www.androiddocs.com/guide/topics/providers/document-provider.html

Here are some issues of this app regarding SAF: #31, #51, #65, #67, #106

Longer answer: Google considers good old filesystem API insecure as basically all apps can access all files. They invented their new API called SAF which requires more user interaction (choosing of directory). That shall make it more secure as user has to choose which data an app can access. But obviously they failed to provide a way to set mtime with their new API.

catscarlet commented 5 years ago

access of external SD when using Plain old filesystem?

Short answer: no, it is blocked by the operating system.

Here is some documentation from google: http://www.androiddocs.com/guide/topics/providers/document-provider.html

Here are some issues of this app regarding SAF: #31, #51, #65, #67, #106

Longer answer: Google considers good old filesystem API insecure as basically all apps can access all files. They invented their new API called SAF which requires more user interaction (choosing of directory). That shall make it more secure as user has to choose which data an app can access. But obviously they failed to provide a way to set mtime with their new API.

Well, weird. Most of reliable 3rd-party gallery apps and file-managers can do it. When needs to access external SD, they tell 'select the root of 'external SD', and popup the system's default file-manager. After that, they can have all the access. Is that SAF too?

And I still don't understand why it does not work. Google should have already fixed it in 8.0 by using SDCardFS.

If it is not possible to access of external SD when using Plain old filesystem, please try Setting of mtime for SAF. It should works.

catscarlet commented 5 years ago

At least com.theolivetree.ftpserver can set mtime on external SD.

wolpi commented 5 years ago

Is that SAF too?

It is likely to be SAF

by using SDCardFS

That is something different. That is about mounting a filesystem to be accessed with old school filesystem API. But they don't allow apps to access that filesystem via filesystem API.

please try Setting of mtime for SAF. It should works.

Well, it is implemented. And it does not work on all devices. You can try yours when next version is released.

At least com.theolivetree.ftpserver can set mtime on external SD.

As I said before: would be interesting to know how they implemented it. My guess is that they bypass android stuff like java dalvik vm and app sandboxing by using low level linux API with plain C.

Did you try to launch server in filesystem mode and browse folders to access your sd card? On some devices you can see it when switching to directory /storage.

catscarlet commented 5 years ago

Did you try to launch server in filesystem mode and browse folders to access your sd card? On some devices you can see it when switching to directory /storage.

Tried, with Plain old filesystem, prim-ftpd-5.11, Android 8.0 The SD card directory can be seen (/storage/35BE-34CB/), and can be visit as expect. No trouble with visiting and downloading files, but can't upload file or delete file. Readonly.

wolpi commented 5 years ago

So your device is among those with readonly access. Other devices block it completely. While again others don't block it at all. See linked issues above for other weird behavior of devices regarding sdcard access via filesystem API.

catscarlet commented 5 years ago

So your device is among those with readonly access. Other devices block it completely. While again others don't block it at all. See linked issues above for other weird behavior of devices regarding sdcard access via filesystem API.

That's strange. I have two 4.4 phone, one 5.1.1 phone, one 7.1 pad, and one 8.0 phone. All those devices have readonly access. I have never met devices block read access.

catscarlet commented 5 years ago

I use update my XZ1 to 9.0 and, still can't keep file modification time.

wolpi / prim-ftpd

Files modification time as create time, expect it as original modification time. #145

Case One:

Case Two