Add media-scanning after upload (Writing files via root method produces different results from SAF method)

[tayl]

Hello,

My use case is that I transfer images to a phone running primitive ftpd which are then picked up by Google Photos running on the device and uploaded. I've been using SAF method for a while, and it works fine once the file is on the phone. That is to say once a remote client connects to the SFTP server, and transfers the file successfully, Google Photos immediately sees the file and begins uploading it.

Unfortunately, SAF is otherwise very unreliable on my phone. I don't suspect this is a primitive ftpd issue so will not focus on it here. Permission to the directory is seemingly revoked every couple days, resulting in the remote clients inability to connect to SFTP. So, I've switched to root method. This gets rid of the permissions and connection issue, but introduces a new issue. Files that are stored to the phone using root method seem to be missing something that files stored using SAF have. The Google Photos app does no longer immediately detect that new files exist in the folder it is watching, and will only start picking them up after Google Photos is force closed and reopen.

My best guess is that when stored via SAF, the OS or folder somehow notifies other processes that are watching that folder that files have changed and to scan for changes. When stored via root, this is not happening.

I've verified that the umask is the same for files stored via SAF and files stored via root (0660). Both are owned by the root user and everybody group.

Is there anything I can do to provide more information to debug this?

Thank you

[tayl]

Some more testing: switching from root method back to SAF method temporarily, and renaming a file via FTP that was stored via root method, causes it to immediately show up in Google Photos.

[tayl]

OH! I think it's related to the date that is being set on root method. When stored via SAF method, the date displayed in Android OS is correct (local time). According to client log in primitive ftpd, SAF method is actually getting an error during file transfer related to date. See below:

2024-03-14 12:16:29 SAF SFTP 10.0.1.52 LIST_DIR /
2024-03-14 12:16:50 SAF SFTP 10.0.1.52 UPLOAD /saf4.jpg
2024-03-14 12:16:51 SAF SFTP 10.0.1.52 ERROR /saf4.jpg could not set last modified time, error: java.lang.NullPointerException
2024-03-14 12:16:51 SAF SFTP 10.0.1.52 ERROR /saf4.jpg could not set last modified time, error: java.lang.NullPointerException
2024-03-14 12:16:51 SAF SFTP 10.0.1.52 ERROR /saf4.jpg could not set last modified time, error: java.lang.UnsupportedOperationException
2024-03-14 12:16:51 SAF SFTP 10.0.1.52 LIST_DIR /

However, when stored via root method, no errors are thrown in client log, so the date is likely being set without issue, which is actually what is causing the issue. The date it is setting is UTC, but maybe it is setting it without timezone info? So Android OS is just accepting it as local time. Google Photos then isn't seeing, or is choosing not to display the file, as it has a timestamp that has not happened yet.

[tayl]

Last post ha.. I've confirmed it's definitely related to the incorrect date being set. If I upload a file from root method, wrong date gets set. Google Photos does not see file. If I then rename file so last modified date is adjusted to correct time in local file manager, it immediately appears in Google Photos.

So, let me see if I can dig into the code and see what's going on. Not familiar with this code base so if someone else knows exactly where this is, feel free to jump in.. would be much appreciated.

The phone is running Android 10 btw. Not exactly recent, but I wouldn't expect that to affect something like dates and times.

[tayl]

I captured some logs during my troubleshooting and attached them here. The files I transferred to troubleshoot were saf.jpg, saf3.jpg, saf4.jpg, and root.jpg, root1.jpg, and root2.jpg. The names reference how they were transferred. The "could not set last modified time" errors are shown and may be of interest too.

The root cause seems to be in org.primftpd.filesystem.RootFile#setLastModified

The dates are correctly parsed and shown as UTC in the debug logs. However, when they are written to the file in RootFile#setLastModified, they are converted to a string and solidified as local time on the device, rather than as UTC. It's the call to 'touch' that is receiving UTC but expecting local time. The formatter doing the work is currently being told that the timestamp it's formatting has no timezone, when in reality is seems like it does (UTC). If that was set correctly, I wonder if that would resolve it. I don't have a build environment set up to test this unfortunately. Besides, I'm not sure what downstream effects changing this would have.

Thanks

primitive-ftpd-debug.txt

[tayl]

Oh boy, date may be a red herring after all. I got the build environment set up and started debugging around the date. Turns out it may not be related. If I remove 'touch' entirely, the original date makes it through, which the OS recognizes as being current, but it still doesn't appear to Google Photos. Yet renaming the files from within the OS still enables them to be seen. I guess I'm back to my original hypothesis... something SAF and the OS are doing when creating/copying files is being noticed by Google Photos, but when root does the same, the file is not seen. I tried manually copying the file out of and back into the folder from the terminal as root, but no luck. Also tried moving, and renaming as root. Still nothing.

[tayl]

I worked at this again today but I'm stumped. Not sure if it's a Google Photos issue or what. If anyone else has interest to look into this, I'd appreciate it. I'm not deeply familiar with Android as it operates as an OS. Someone else may take a look at this and know instantly what's going on. The only additional thing I noticed today is that Google Photos does eventually notice that the files have been added to the directory, but hours later. So I don't think it's a property of the file that's causing them to not be seen immediately.

[wolpi]

I guess I'm back to my original hypothesis... something SAF and the OS are doing when creating/copying files

I can imagine this.

However, when they are written to the file in RootFile#setLastModified

There have been issues with lastModifiedTimes before. As you noticed there are still issues with other storage types. With current impl there have been no more known issues. Interesting that you found the UTC issue. Maybe that would be worth another change in that code area.

I don't have an idea how your use case could be solved. As you might know, in recent versions Google has restricted filesystem access more and more. They invented SAF instead. IMHO it is nearly imposible on current Android to use SFTP server without user interaction. SAF is built in a way that requires a user to explicitly choose files and confrim access to it. You cannot just launch the server and access files remotely anymore. When you store files as root that works on some basic Linux level and works around Android mechanisms. Which is basically your suspicion above. For your use case to work according to Google's ideas you would probably need to choose SAF folder each time when starting the server to have the useer interaction Google wants to see.

In order to automate this it would probably be necessary to do something like "SAF touch" after files have been uploaded. That would probably be hard to implement. It would probably need user interaction as well.

[tayl]

I'm considering trying this with later versions of Android to see if it's even an issue later on. I'm on Android 10 which is the latest version my device supports natively, but there are LineageOS Android 14 builds for it.

In the meantime, I asked about it potentially being a master file table deal akin to NTFS in #android-dev and someone suggested investigating the following links..

https://developer.android.com/reference/android/provider/MediaStore https://developer.android.com/reference/android/content/Intent#ACTION_MEDIA_SCANNER_SCAN_FILE https://developer.android.com/training/data-storage/shared/media#add-item

.. which look promising, but of course these still assume you're using normal methods to interact with files 😅

[tayl]

https://developer.android.com/reference/android/media/MediaScannerConnection#scanFile(java.lang.String,%20java.lang.String)

I'm still playing with it, but this actually seems to achieve what I was going for. Passing the app context and path of file to MediaScannerConnection.scanFile causes it to immediately appear in Google Photos. I suspect this is the missing step.

[wolpi]

MediaScannerConnection.scanFile

Sounds good, I've added it.

[tayl]

I'm not sure java.io.BufferedOutputStream#close is being called as expected. I put some logging statements in there and didn't see any output.

During testing, I had to put the call to scanFile in org.primftpd.filesystem.RootFile#move because the FTP client I'm using is actually moving instead of creating. So that may be an edge case to consider.

public boolean move(RootFile<T> destination) {
    logger.trace("[{}] move({})", name, destination.getAbsolutePath());
    postClientAction(ClientActionEvent.ClientAction.RENAME);
    boolean result = runCommand("mv " + escapePath(absPath) + " " + escapePath(destination.getAbsolutePath()));
    if (result) {
        logger.trace("[{}] mediaScanFile({})", name, destination.getAbsolutePath());
        MediaScannerConnection.scanFile(pftpdService.getContext(), new String[]{destination.getAbsolutePath()}, null, null);
    }
    return result;
}

[wolpi]

It's great that those APIs don't have a reliable close(). Let's see how this turns out.

[wolpi]

let's see if more places are found where media scanning would be necessary.