Enable writing to external storage folders

[kabukky]

I understand the NZBGet daemon is limited when it comes to the directories it can write to.

On the external storage, it can only write to /storage/XXXX-XXXX/Android/data/net.nzbget.nzbget as far as I can see.

This is probably because the daemon runs as a linux process and therefore does not use the Android API to write files.

To circumvent this, I was thinking about moving the finished download using the DaemonService. In the DaemonService, I could use the Android API to move the files to a user-defined directory after the download is finished. Maybe we could even detect TV shows, movies, etc. and treat them differently (e. g. moving them into different directories).

The destination folders would probably need to be specified in the Android UI, not in the Web UI of the daemon.

What do you think? What would be the best way to be notified of a finished download? I was thinking about watching the history log file (via https://developer.android.com/reference/android/os/FileObserver).

[hugbug]

Google currently doesn't provide native API for requesting storage write access. This is unfortunate considering Google efforts in improving NDK. Maybe such API will appear in the future.

Since I'm not using NZBGet on Android I can't tell how bad the limitation of current situation is. In my short tests I was able to use total commander to find downloaded files and open them in Android apps (pdf and videos). I think on a phone or tablet it is usable, for occasional downloads.

Android TV devices is probably a different thing (never had a chance to try one). If there is some kind of media library and media center software I suppose you would like to have downloaded videos appear there without looking deep through folders, especially without touch interface the navigation is probably not so quick.

One missing feature in NZBGet on Android currently is post-processing scripts. There are no easy way to execute python scripts. On other platforms the pp-scripts are used (among other things) to organise downloaded files. Currently without pp-scripts you can assume that all downloaded files remain in the destination folder. The NZBGet Android app can move them then. However it seems to be possible to compile Python and ship it within NZBGet package. If that would work someday it would be good if the solution with moving files would be compatible with pp-scripts. That's just some thoughts.

What would be the best way to be notified of a finished download?

The app can communicate with daemon via API method history. Either polling it every X seconds or by monitoring NZBGet disk state directory (option QueueDir) for changes to file history - if that's more resource friendly than constant API calls.

On each check the app could read the content of destination directory (option DestDir) and determine status of each download from API history. Maybe also doing some cleanup - delete directories for which there are no entries in history and queue.

Since the app can now communicate with daemon the app could show some useful infos on its home screen such as daemon version, download status and speed etc. Mimicking the whole webui would probably be a wasted work but a few infos could make the app nicer. Talking about UI rework again :).

Not forget about categories which can have their own DestDir defined. Maybe instead of reading content of global DestDir the app should read the history and process history entries which it hasn't processed yet.

How would it determine which were already processed? NZBGet allows to save custom properties for queue and history entries - using API method editqueue with Command=GroupSetParameter. The properties are available via API methods queue and history. Despite the name method editqueue works on history items too. After moving files the app would set a custom property of the history item and when the app checks the history next time it will see which items have the property set.

Moving can be a long process, especially if the device can work with external HHDs. The app probably needs some kind of progress information in its UI.

[kabukky]

Thanks for the detailed instructions, I'll look into the API. Since it uses HTTP Basic Auth, the user would need to enter the username/password into the Android UI as well. Not ideal, but probably good enough.

Beginning with Android 4.4, apps need to use the storage access framework to write to external storage (directories outside the private path - /storage/XXXX-XXXX/Android/data/net.nzbget.nzbget in this case). For that, the user needs to select the directory using a picker we can present like this:

        Intent intent = new Intent(Intent.ACTION_OPEN_DOCUMENT_TREE);
        intent.addFlags(
                Intent.FLAG_GRANT_READ_URI_PERMISSION
                        | Intent.FLAG_GRANT_WRITE_URI_PERMISSION
                        | Intent.FLAG_GRANT_PERSISTABLE_URI_PERMISSION
                        | Intent.FLAG_GRANT_PREFIX_URI_PERMISSION);
        startActivityForResult(intent, 0);

After that, the app has permanent write access to the picked directory. That should enable us to move downloaded files there.

Using PP scripts would be great, but moving the files to external storage must still be done with Java code as far as I know, since the storage access framework provides us with a DocumentFile that must be used to write data.

[hugbug]

No need for username/password entering by user. By default the daemon is installed with AuthorizedIP=127.0.0.1. And even if it wouldn't the app can read username/password from daemon configuration file, which is saved in the app data directory.

[kabukky]

That is good news!

I found a way to give the file descriptor of the DocumentFile from Java to the NDK: https://stackoverflow.com/questions/30593964/how-to-access-android-lollipop-documentfile-files-via-ndk/31677287

But I don't know much about the NDK and the way you build NZBGet with it to know if this is practical to you. Could you work with that?

In theory, we could let the user choose the root of the external storage, give that file descriptor to the NDK and there use it to write files anywhere on the external storage.

Java would probably pass that file descriptor every time the daemon starts, but that should be no problem.

[hugbug]

To my understanding every file must be opened via DocumentFile. Opening only one file via DocumentFile will not allow other files to be opened with standard functions fopen.

The only way the daemon could access files on storage would be if the app would pass file descriptors for all required files (all opened via DocumentFile). This is not possible because it's not known which files must be opened, the daemon creates files during download.

What could work is some kind of communication channel between daemon and the app where the daemon would ask the app to open certain file. That would require to change a lot of code in NZBGet. At the end that would not work for unrar anyway (unless we are going to rewrite unrar too).

[kabukky]

That would indeed be impractical. I was hoping that a file descriptor for the root directory of the external storage would be enough. I was imagining that you could write into any subdirectory of root using that file descriptor. But as I said, I'm not familiar with the NDK.

[kabukky]

I found some time yesterday to implement history watching and communicating with the API. Getting the config and history from the API works great. I'm having trouble setting my parameter using the editqueue method: I'm POSTing the following payload to http://localhost:6789/jsonrpc:

{"method":"editqueue","nocache":1544185334596,"params":["GroupSetParameter","MovedByAndroidDaemon=true",[2]]}

But I am getting this response:

{"version":"1.1","result":false}

I was expecting "result":true. Accordingly, MovedByAndroidDaemon doesn't seem to be added to the entry.

Can you see any fault in the way I'm doing it?

/Edit: I also tried the deprecated method just to be safe:

{"method":"editqueue","nocache":1544186627174,"params":["GroupSetParameter",0,"MovedByAndroidDaemon=true",[1]]}

Same result.

/Edit 2: I'm using the NZBID field from the history response to fill the last array in the params field btw.

[hugbug]

My bad. Please try HistorySetParameter.

[kabukky]

That works, thanks :)

[kabukky]

Two more questions:

Do I need to include the nocache parameter? The documentation doesn't mention it.

How would I best determine if NZBGet is finished with the download? Looking at the history API: As far as I can see, Status could have different values, depending on what should be done with the download. E. g. SUCCESS/PAR, SUCCESS/UNPACK, SUCCESS/ALL. Could I simply check if Status begins with SUCCESS/, or does the history entry move through the above stages (SUCCESS/PAR, SUCCESS/UNPACK, SUCCESS/ALL) as it is being repaired, unpacked, post processed?

[hugbug]

nocache isn't part of API, it's used in webui to prevent browser from caching results.

If item is in history then nzbget has finished the processing. Active or queued items are in queue (not yet in history) which is returned by listgroups.

[hugbug]

For history items Status is final. SUCCESS is probably indeed the only status of interest for the purpose of moving completed (and successful) downloads. Normally status WARNING/SCRIPT is also almost success but it's unlikely to happen on Android where support for scripts is problematic.

[hugbug]

Amazing work.

On your Nvidia Shield is /storage/XXXX-XXXX/Android/data/net.nzbget.nzbget on the same physical drive as the destination directories for final moving?

If they are on separate disks the moving can take a while. Maybe some kind of activity indication in the app would be useful.

[kabukky]

Thanks! On my Shield the src and dest dirs are indeed on the same drive. Unfortunately, since we need to use the DocumentFile, we can't just rename to move the dir if they are on the same drive. I'll look into a progress indicator. Where should it be displayed? In the notification, or in a "Status" activity inside the app?

/Edit: We could additionally add a warning to the StorageActivity about moving files potentially taking a long time.

[hugbug]

Where should it be displayed?

I thought about inside the app but the notification is also good. Since you use the app more than me it's better if you decide what's best :-)

About notifications: would it be useful if the app would show notifications about finished downloads? For downloads which are moved by the app - after moving. For downloads which are not moved - after they appear in the history.

[hugbug]

Thank you very much. Closing this as resolved.