Amaze 3.3.2 move/copy results are not reflect throught MTP immediately

[hman2]

I try to organize my images with Amaze. Therefore I go into the folder containing the images shot with OpenCamera, which is on my SD card "Pictures/Open Camera" and I create some folders and sort images into them.

Steps to reproduce the behaviour:

1. Go to /storage/[cardID]Pictures/OpenCamera, create a folder
2. Select some images
3. Since there is no "move" feature, click on the scissors icon
4. Enter the folder just created and click on the clipboard icon
5. Watch and see images appear every 10-20 seconds at a time
6. Go the the parent folder again and see images disappear 10-20 seconds at a time
7. Connect the phone to a PC and look into the Pictures/OpenCamera folder
8. Try to find the folder created in 1). Nowhere to be found!
9. Go into Amaze again, re-enter the folder, it is still there.
10. Look at the images cut in 3): They are gone (the expected result)
11. Check with the PC: there they are gone, too (expected). Again try to find the folder (nowhere, and that is unexpected).

Expected behaviour: Cutting and pasting should be executed immediately, not in ultra slow-motion. And folders created and populated in Amaze should be seen via USB on the card... Results of these operations should be noticeable on the storage media...

• Device: Fairphone 2
• OS: Fairphone Open OS 19.11.2 (Android Nougat 7.1.2)
• Rooted: No and Yes (not necessary on FPOS)
• Version: 3.3.2
• SD card: Sandisk Ultra 128 GByte SDXC SDSQUAR-128G-GN6IA, the one sold as “up to 100 MB/Sec, Class 10, U1". Card is very fast, as can be seen via USB...

First, the speed cannot be explained. It is many orders of magnitude (!) slower as could be expected, second, it seems that cut & copy is not operated on the OS level (where no data at all would be moved, just the pointer in the file system modified) but actual data are transmitted. But even that doesn't explain this level of slowness.

Where exactly are my images now? According to the PC, the are no longer in the OpenCamera folder. But since the newly created folder is missing, the images can't be there. But Amaze still shows them, so they must be somewhere. Does Amaze create temporary folders somewhere?

Exactly where are my images? I want to backup them to the PC...

[hman2]

Strange observation: After I viewed a moved image in Simple Gallery, which showed the new path to the image, then the image became visible via USB.

Then I did this with one of the newly created folders. The image showed up in Simple Gallery, and also appeared via USA. And its folder, too! Looking at the folder, it contains just one image image, the one viewed in Simple Gallery.

So it seems to be a caching issue. I will now try to see one of those folders in Simple Gallery, let's see what happens.

[hman2]

In Simple Gallery I can only see the folder, from which I opened an image from Amaze in Open Gallery. Other folders I created and populated, and which are invisble via USA, are also nowhere to be found in Simple Gallery. So I have to suspect that whatever Amaze did, was not properly written to the SD card.

[hman2]

Oh, it can be even watched "live". At the very moment that I tap on an image in Amaze to open it in Simple Gallery, it magically appears via USB...

[hman2]

Btw, Amaze's method to copy instead of moving, i.e. creating new (!) files destroys the files' datetime stamps. They are all reset to date and time of the move operation. Luckily as images, this data is also part of the filename, but the loss of the OS timestamps makes "funny" sort orders...

[TranceLove]

Sorry to hear you having bad user experience on using Amaze.

Strange observation: After I viewed a moved image in Simple Gallery, which showed the new path to the image, then the image became visible via USB.

I suppose Simple Gallery had triggered Android's media library to force scanning the device, so that files can become visible back?

Btw, Amaze's method to copy instead of moving, i.e. creating new (!) files destroys the files' datetime stamps. They are all reset to date and time of the move operation. Luckily as images, this data is also part of the filename, but the loss of the OS timestamps makes "funny" sort orders...

Android's SAF restricts coders not to use direct I/O but working with streams instead (content:// URIs), so one way another we will end up in the destroy-create path anyway, towards Android 10. However the timestamp issue... we may need to look into it.

[hman2]

Thanks. But the strange thing is: Once I opened a file in the newly created folder, clicking on it in Amaze, only this very file (and its folder) became visible. All the other files in this new folder still remained invisible... After clicking on all of them, all became visible.

[hman2]

Also, Simple File Gallery itself will only see files so clicked on, all other files (and folders) remain invisible to File Gallery, so it's not a USB/MTP related issue.

[VishalNehra]

As far as I remember, we do trigger media library scan after every cut/copy operation, so this issue shouldn't be there. The issue might also be related to your device's file system. It isn't USB related issue for sure, and there is no issue with cut/copy function as such as you verified yourself file are present at the new location. I'll confirm if we're triggering media scanner. As for timestamp there is already an open issue #892 We'll try to prioritise it since it's been open for quite some time.

[hman2]

Thanks! But this "media scanner" function, shouldn't that scan the entire media? That would be my understanding of a "media scan": Reading the entire file system. The card got formatted in the phone prior to it's initial use, a year ago. So I suspect no problems reading the card. I transferred about 40 GBytes of MP3s on it with no problems at all (and NO speed/performance anomalies).

So how can it happen that only the very file clicked on in Amaze gets "discovered"? Even when -together with that file- a new directory gets "discovered", no other entries in that directory get "discovered", that puzzles me. I believe it must mean that the directory isn't read at all, just the path gets remembered. As I wrote, after clicking on an image in a new directory, Simple File Gallery does NOT see other contents of this directory. IMHO both Simple File Gallery and the MTP interface dwell on reading the same functions in Android, so this explains why Simple File Gallery and MTP show the same "discovery" scenario.

If so, why does Amaze see this other content when Amaze gets force killed and restarted? Btw, during my move operations, (AFTER I saw the behaviour from this report) Amaze crashed two times.

[VishalNehra]

It seems the issue is result of MTP protocol implementation, and it's required to manually add paths to MediaScannerConnection We seem to be handling this only for file rename operations Operations#506 and other operations such as move/copy just trigger a system wide media scan. I think we need to extend the function FileUtils#244 for move/copy operations as well. @hman2 please validate if renaming any file/image is correctly being reflected on your PC. @TranceLove sadly I can't reproduce this issue on my device / PC connection, the files are visible immediately. Request you to validate this from your side.

[TranceLove]

It seems the issue is result of MTP protocol implementation, and it's required to manually add paths to MediaScannerConnection

Agreed. And in fact it is possible to notify Android MediaScanner that it only needs to scan for one single file/folder.

@TranceLove sadly I can't reproduce this issue on my device / PC connection, the files are visible immediately. Request you to validate this from your side.

Yes, same here - tested with Oneplus 2 running AOSPExtended (7.1.2) and Fairphone 3 running LineageOS 16.0 GSI. That suggested why I hesitated to reply...

[hman2]

Yes, with a little delay (5 s) renaming is being shown on the PC, without the need to open the image by clicking on it (so I did not leave Amaze, SFG was not called). Thanks.

[VishalNehra]

Yep, thanks for validating @hman2 So now we know what to do here, let's expect a fix in upcoming release. Suggest you to signup for beta releases to help us validate the fix.

[hman2]

Will you implement this in the F-Droid version, too? Thanks Hman

[TranceLove]

Will you implement this in the F-Droid version, too? Thanks Hman

Big showstopper: #1593, #1684. Sorry about that, and team had been working on a replacement. Before the Cloud plugin is also F-Droid compliant you may need to wait, or simply use releases from Github here.

[hman2]

As I wrote earlier in the Cloudrail discussion, OwnCloud is a completely Open Source cloud solution including an Android client. And in my opinion, cloud support could be pushed out into a plugin (that might get blacklisted by F-Droid). Not everyone needs cloud support and maybe (haven't tried yet) Amaze doesn't need cloud support by its own, if the user has the OwnCloud app, too...

[VishalNehra]

Noted @hman2 . Let's not deviate from the original topic of discussion here (:

[VishalNehra]

Hi @hman2 Can you please check if the issue is fixed or not, you need to download apk from following link which has code from the PR https://github.com/TeamAmaze/AmazeFileManager/actions/runs/150698685

[hman2]

Hi, which one? There is one with "play" in the name and one with "fdroid" in the name. I thought you wouldn't update the F-Droid version, at least not now?

Thanks Hman

[VishalNehra]

You can try play one..

[hman2]

Downloaded and installed. It asked for the right to access the storage card, which I granted. When I wanted to cut & paste a file, an error dialog opened, that I had on former Amaze years ago that claims a specific order of tasks to perform to grant necessary rights, which I never managed to repeat faithfully, but somehow managed anyway.

When I go into Settings | Apps | Permissions I see "storage", when I extend this view "all permisssions" I see "Read SD card contents" and "Modify or delete SD card contents". This is Fairphone Open OS, but it should be an exact Android Nougat 7.1.2.

Why is Modify & delete not enough and what are the precise steps to grant the right needed (the screenshot Amaze shows to do this is really tiny).

Thanks

[hman2]

The screenshot given says Step 1, three dots menu. Step 2 selct "Show SD card", but that is not an option. The first option of the three dots menu is "Sort"...

[VishalNehra]

It's required by the framework.. we have to request for storage permissions on mounted storages separately It should be fairly simply, can you send screenshot of the screen that opens up after you press on grant permission option from Amaze

[hman2]

I know I have to grant permissions for the SD card separately, but I did. Amaze Debug asked for it right away upon the first start. I can see the contents of the SD card in Amaze. The error message comes only when I paste (clipboard icon). When I do the step before that, which is cut (scissors icon) everything is okay. I would expect such behaviour if Amaze would only be allowed to read, but not to write. Then this would be a logical behaviour.

But Settings | Apps from Android says that "read" AND "modify or delete" so Amaze "should" have all the rights needed? I know I had this "trouble" before, but I do not remember how I solved this, but eventually it worked.

On the error message, when I click on Open I see "Open from" and under that the Documents folder, then the internal memory, then the SD card, so it is already there. When I then click on the three dots menu, I get "Recent", which is empty...

[VishalNehra]

Ahh.. okay.. so when we press on cut we don't know there you'll be pasting the content.. that's why on cut we can't just show the popup to grant write permission. I BTW if it's not in Three dot menu, the. You have to navigate into the root of SD Card from navigation drawer on the framework screen. Just swipe to open navigation drawer and choose your SD Card. It'll open up the root of your SD card.. and there should be 'Select' option in bottom right.. press on it and it'll grant the required permissions.

[hman2]

Ah, that does the trick. Thank you. But in Settings | Apps I see no other rights than before ?!

Now Cut & Paste work on the SD card. And I can see the result over MTP, kind of. It is VERY slow. For 20 Seconds or so nothing happened at all, neither on the phone in Amaze nor over MTP. Then on the phone a file with a very weird file name popped up "AugendiagnoseDummyFile1" (partially in German, it means ocular diagnostics ?!). Only on the phone, nothing via MTP. Also curious, the file claimed to be produced January 1st, 1970 at 01:00 (I guess this is the second 0 of Unix...) and no thumbnail.

Then this strange file vanished again. Then the pasted file appeared, and five seconds or so later also over MTP, and with proper thumbnail. In total it took about half a minute for a file of just 7 MBytes!? And the file bears the timestamp of the cut&paste operation, not of file creation (I just took a quick photo with Open Camera, so file creation becomes part of the files name, so this can be easily compared).

But for the first time I did not have to launch Simple Gallery to see it over MTP...

[VishalNehra]

Great! Thanks for testing. As for the strange file appearing.. it seems to be an older implementation to check if we can write on the storage, then it gets deleted. Currently I'm not sure I can remove/improve it. As we're about to push for new release and this will probably require much more testing. I think it's better to give something than nothing. So for the slow operation, I'll check if it can improved.. but as far as I remember.. it's already triggering media scanner only on paths getting pasted.. so not sure if speed can be improved. What do you think? Is it okay to give this out for the time being?

BTW for storage permissions, it seems android system don't show anywhere what all permissions are given for mounted devices as of now.. because these are temporarily permissions and gets revoked as soon as you unmount them.

[VishalNehra]

Would request you to check few more operations like - copy/paste, delete, rename, extract / compress, encryption / decryption (not sure if it's supported for SD Cards).

[hman2]

Copy/Paste ok, Rename ok, upon deletion Amaze Debug crashed :-( But after restart, the file was gone. Retested, crashed again, after restart, file was gone. There is no option to compress? Extract looked as if would work, it created a new file AugendiagnoseDummyFile1 which again took 1/2 minute, but then nothing else happened. In the test-Zip I had one directory and about 30 JPEGs, so I supected the directory, created a new Zip with just one JPEG and transferred it on the phone. When I wanted to extract this, Amaze Debug crashed. Upon restart I saw that Amaze Debug had extracted both the directory and all JPEGs. And the contents of the second Zip were there, too. Obviously Amaze Debug creates directories named after the Zip file.

[hman2]

So far did not try encryption, it gives an error message that contents can never be decrypted if I ever change the swipe pattern to unlock the phone?! Is this another Android weirdness? I'll test this only if you can assure me that it will only affect the test file and not one single other file... Thanks.

[VishalNehra]

So many crashes 🤔 strange Possible for you to get the logcat?

BTW for encryption.. yes, it's android's limitation, the keystore where we store the encryption keys gets wiped on changing screen lock. So it becomes impossible to decrypt the file.

Anyways, the original file will be there if you encrypt. So you don't have to worry, maybe for testing you can move the encrypted file to some other directory, so that when you decrypt it doesn't overwrite your original file.

[hman2]

If you tell me where logfiles or crashdumps are located, I'll get them for you.

[VishalNehra]

Great! You can use any logcat application from Play store https://play.google.com/store/apps/details?id=com.dp.logcatapp&hl=en_IN After running it you need to grant adb permissions to the application. Step would be listed in that app only. Then you can reproduce the crash while the log application is running, it'll capture the crash stacktrace. Further you can all filter in logcat application for 'amaze'.

[hman2]

I installed Tortel/SysLog. So far I did not give it the right to READ_LOGS, as I would have to that via ADB. But even without it does log a lot, I only left out modem logs. Another problem: When saving the log through Amaze (I have seen this before) Amaze doesn't take that upon being called (I have seen this with files coming through K-9 mail client), Amaze starts, but does not ask for a file name to set... And while Amaze is installed, Androids own File is disabled. But I managed to store the file throught Total Commander (and then seeing that Total Commander, too, doesn't present new files right away over MTP). But I managed to get it onto the PC.

I do see some reports of signal 6 (SIG_ABRT) but I do not see a string with Amaze in logcat.log. How do I attach a file here? I see no attachment feature, and even though the log is brief (at least when I compare it my kernel log on the PC :-) it still has 45 KBytes for just half a minute...

[hman2]

2020-06-30_00.27.zip Just saw the grey text that said drag & drop to attach - I always looked for an icon to do this :-) Attached first shot at a log.

[VishalNehra]

I'm sorry but it seems Amaze logs aren't getting captured in it. It seems you have to give READ_LOGS permissions after all. Going forward we'll try to implement Newpipe like log capture service when it crashes, instead of depending on Google services to capture and publish.

[hman2]

Here is a log taken with adb shell pm grant com.tortel.syslog android.permission.READ_LOGS and with filtering removed. 2020-06-30_09.32.zip

[VishalNehra]

Strange, logs are still not there in this. Are you able to see Amaze's logs in the Tortel's console?

[hman2]

I've seen that there is a newer version of SysLog (2.4.1). I updated and re-ran the test. Now there is considerably more stuff to read in it, including a segfault, but I am not sure whether that was actually inside Amaze or in the McAfee Antivirus Suite, but Amaze was terminated with that. Hm, cannot upload "Something went really wrong, and we can't process that file". I don't think size is an issue, it's just 179 KByte.

[hman2]

Since uploading failed (looks like Github went away?) I copied and pasted an occurence of Amaze in the log: 06-29 20:28:21.314 E/AndroidRuntime(19814): FATAL EXCEPTION: AsyncTask #5 06-29 20:28:21.314 E/AndroidRuntime(19814): Process: com.amaze.filemanager.debug, PID: 19814 06-29 20:28:21.314 E/AndroidRuntime(19814): java.lang.RuntimeException: An error occurred while executing doInBackground() 06-29 20:28:21.314 E/AndroidRuntime(19814): at android.os.AsyncTask$3.done(AsyncTask.java:325) 06-29 20:28:21.314 E/AndroidRuntime(19814): at java.util.concurrent.FutureTask.finishCompletion(FutureTask.java:354) 06-29 20:28:21.314 E/AndroidRuntime(19814): at java.util.concurrent.FutureTask.setException(FutureTask.java:223) 06-29 20:28:21.314 E/AndroidRuntime(19814): at java.util.concurrent.FutureTask.run(FutureTask.java:242) 06-29 20:28:21.314 E/AndroidRuntime(19814): at android.os.AsyncTask$SerialExecutor$1.run(AsyncTask.java:243) 06-29 20:28:21.314 E/AndroidRuntime(19814): at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1133) 06-29 20:28:21.314 E/AndroidRuntime(19814): at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:607) 06-29 20:28:21.314 E/AndroidRuntime(19814): at java.lang.Thread.run(Thread.java:761) 06-29 20:28:21.314 E/AndroidRuntime(19814): Caused by: java.lang.IllegalStateException: This is not the first datapoint! 06-29 20:28:21.314 E/AndroidRuntime(19814): at com.amaze.filemanager.asynchronous.services.AbstractProgressiveService.addFirstDatapoint(AbstractProgressiveService.java:234) 06-29 20:28:21.314 E/AndroidRuntime(19814): at com.amaze.filemanager.asynchronous.services.ExtractService$DoWork$1.onStart(ExtractService.java:284) 06-29 20:28:21.314 E/AndroidRuntime(19814): at com.amaze.filemanager.filesystem.compressed.extractcontents.helpers.ZipExtractor.extractWithFilter(ZipExtractor.java:79) 06-29 20:28:21.314 E/AndroidRuntime(19814): at com.amaze.filemanager.filesystem.compressed.extractcontents.Extractor.extractEverything(Extractor.java:77) 06-29 20:28:21.314 E/AndroidRuntime(19814): at com.amaze.filemanager.asynchronous.services.ExtractService$DoWork.doInBackground(ExtractService.java:315) 06-29 20:28:21.314 E/AndroidRuntime(19814): at com.amaze.filemanager.asynchronous.services.ExtractService$DoWork.doInBackground(ExtractService.java:225) 06-29 20:28:21.314 E/AndroidRuntime(19814): at android.os.AsyncTask$2.call(AsyncTask.java:305) 06-29 20:28:21.314 E/AndroidRuntime(19814): at java.util.concurrent.FutureTask.run(FutureTask.java:237) 06-29 20:28:21.314 E/AndroidRuntime(19814): ... 4 more

[hman2]

New try to upload the log(s). 2020-06-30_12.40.zip

[VishalNehra]

This works! Thanks (: Can you please let me know what all operations you tried during this log test?

[hman2]

In this test I copied a file (to get one I can delete) and then deleted it. That's it, Amaze crashed, and I saved the log in SysLog.