Rclone android builds

[ncw]

I'm considering dropping the Android builds for rclone from the main rclone repository as they aren't maintained.

I'm not sure if you use these? I think probably not but this is a heads up if you do!

I still want rclone to build on Android though.

Alternatively would you like to maintain or contribute some android building code to the rclone repo?

[x0b]

Thanks for the heads up. I do build all versions from source though, and the same is true for Termux.

The magisk module people do use that build though, but it does not look automated, i.e. removing the build from rclone main does not suddenly break it for everyone.

Alternatively would you like to maintain or contribute some android building code to the rclone repo?

I could port the rcx build script, yeah. I had to figure this out for F-Droid and GitHub actions anyway. Questions:

• Would you add it to the main build.yml of rclone?
• All architectures? It should take around 6 minutes to build the 4 variants (armv7, arm64, x86, x64)

[ncw]

Thanks for the heads up. I do build all versions from source though, and the same is true for Termux.

That is good to know.

The magisk module people do use that build though, but it does not look automated, i.e. removing the build from rclone main does not suddenly break it for everyone.

Good.

Alternatively would you like to maintain or contribute some android building code to the rclone repo?

I could port the rcx build script, yeah. I had to figure this out for F-Droid and GitHub actions anyway.

Great :-)

Questions:

• Would you add it to the main build.yml of rclone?

Yes that would be my preference. It could go in a different file if that is easier but often things change all together in the same build.yml.

• All architectures? It should take around 6 minutes to build the 4 variants (armv7, arm64, x86, x64)

All architectures sounds perfect - the longest action for rclone takes about 10-15 mins so 6 minutes of build won't slow things down.

PS I've already abandoned the IOS build (it stopped building with xgo). I don't think it is useful in any way as you can't run binaries on IOS other than to show that rclone does build for IOS.

[x0b]

@ncw I've created a PR: https://github.com/rclone/rclone/pull/5167.

Re iOS: Agreed. BTW: I added a gomobile bind API to your librclone branch (https://github.com/rclone/rclone/issues/4891), so iOS might might come back as shared library.

It works surprisingly well, but I know just enough Go to be dangerous: https://gist.github.com/x0b/50a0f8ebb4a75051c7aed1e60e0a295d

[ncw]

@ncw I've created a PR: rclone/rclone#5167.

Thank you. Will review there.

Re iOS: Agreed. BTW: I added a gomobile bind API to your librclone branch (rclone/rclone#4891), so iOS might might come back as shared library.

Yes that is a good idea

It works surprisingly well, but I know just enough Go to be dangerous: https://gist.github.com/x0b/50a0f8ebb4a75051c7aed1e60e0a295d

I'm not sure I understand quite what your change does? Can't you just call the RcloneRPC method? Or is that not right for the gomobile?

[x0b]

I'm not sure I understand quite what your change does? Can't you just call the RcloneRPC method? Or is that not right for the gomobile?

No, gomobile only works with a specific set of go types. If a function has a compatible type signature, gomobile can automatically generate all the required scaffolding to call this method from Java (on Android) or Swift (on iOS).

Let me explain. I created this method signature in librclone:

func MobileCallRPC(method string, input string) (output string, status error) {
   // ...
}

Now, after running gomobile bind¹, I receive a .aar file (or a framework file, if I'm building for iOS). Then, I can just call the RC interface directly from Java:

try {
   String response = Api.mobileCallRPC("core/version", "{}");
   // parse JSON, etc.
} catch (Exception e) {
   // Any go errors arrive here
}

And to answer the question: No, at least Java can not call arbitrary C functions.

---

1 | Low-level details

Gomobile has automatically generated the following Java type signature (and all of the plumbing code, to prevent garbage collection of objects that are still in use on the other side).

public static native String mobileCallRPC(String var0, String var1) throws Exception;

If we look at the generated DWARF information, we can actually find a symbol Java_api_Api_mobileCallRPC for the JVM to call, various proxy objects, and of course our original github.com/rclone/rclone/librclone.MobileCallRPC.

[ncw]

I'm not sure I understand quite what your change does? Can't you just call the RcloneRPC method? Or is that not right for the gomobile?

No, gomobile only works with a specific set of go types. If a function has a compatible type signature, gomobile can automatically generate all the required scaffolding to call this method from Java (on Android) or Swift (on iOS).

Let me explain. I created this method signature in librclone:

func MobileCallRPC(method string, input string) (output string, status error) {
   // ...
}

Now, after running gomobile bind1, I receive a .aar file (or a framework file, if I'm building for iOS). Then, I can just call the RC interface directly from Java:

try {
   String response = Api.mobileCallRPC("core/version", "{}");
   // parse JSON, etc.
} catch (Exception e) {
   // Any go errors arrive here
}

I see - thank you for your detailed explanation.

I've pushed an updated librclone branch with a new RcloneMobileRPC function.

Does that work for you and gobind/gomobile?

If I wanted to run gobind to check how would I do that?

Thanks

[x0b]

Thank you for considering/working on this and sorry for the late response. I encountered a lot of errors, and it took me a while to notice that using a new folder for your branch might not be a bad idea 🤦‍♂️. Anyway, I tried your changes and noticed two things:

• The package name cannot be main. Any other package name works, but main results in:

  gomobile: binding 'main' package (github.com/rclone/rclone/librclone) is not supported 

• The method signature should be *`func RcloneMobileRPC(method string, input string) (result RcloneMobileRPCResult)`** - it seems that structs need to be declared as pointer. Otherwise, gomobile will only output (to the resulting java class, not on the console)

  // skipped function RcloneMobileRPC with unsupported parameter or return types

---

If I wanted to run gobind to check how would I do that?

For the tests, I ran gomobile bind -v -target=android in the librclone directory. This requires having set up

• JDK 8 (with environment variable JAVA_HOME pointing to the installation directory),
• Android SDK and NDK (with environment variables ANDROID_HOME pointing to the Android SDK root and environment variable ANDROID_NDK_HOME pointing to a specific NDK, e.g. $ANDROID_HOME/ndk/21.1.6352462 ) and
• gomobile.

I also just tried just running gobind -lang java -outdir gen_java. The resulting file is gen_java/java/<package>/<Package>.java. You may be able to skip installing the Android Components with this one. However, this doesn't show the complete picture - for example, gobind has no problem with a main package, but gomobile does. Though it did show the pointer problem, so it may be useful for testing function signatures.

[ncw]

Not being able to use main as the package is quite annoying!

I've had a go at refactoring the library so that gomobile has its own subdirectory under librclone. I've changed the function signature as you suggested too (I've also renamed the functions).

See branch librclone and gomobile.go.

I tried running gomobile bind -v and it seemed to work!

If it works for you then I'd like to merge what we've got so far into master.

I'd appreciate advice on some more stuff to write here: https://github.com/rclone/rclone/tree/librclone/librclone#gomobile I don't really know what to write since I know so little about it!

Do you think you could do a pull request which checks the gomobile build works in rclone's android build? Probably just seeing whether gomobile bind -v works would be enough but maybe you have a better idea. In that way we can make sure the gomobile stuff keeps working.

[x0b]

Built a simple test, looks good 🎉:

The resulting package name is pretty ugly though, Java usually pretty strictly follows reverse domain, e.g. org.rclone.mobile or something. This is not a real problem since there can only ever be a single gomobile library in an application, so no collisions.

I'd appreciate advice on some more stuff to write here: https://github.com/rclone/rclone/tree/librclone/librclone#gomobile I don't really know what to write since I know so little about it!

I'd add:

• links to https://pkg.go.dev/golang.org/x/mobile/cmd/gomobile and https://github.com/golang/go/wiki/Mobile

• An explanation what the command does, and how to use the library: The command generates an Android library (aar) that can be imported into an Android application project. Librclone will be contained within libgojni.so and loaded automatically.

  Example usage

  // imports
  import gomobile.Gomobile;
  import gomobile.RcloneRPCResult;
  
  // initialize rclone
  Gomobile.rcloneInitialize();
  
  // call RC method and log response
  RcloneRPCResult response = Gomobile.rcloneRPC("core/version", "{}");
  Log.i("rclone", "response status: " + response.getStatus());
  Log.i("rclone", "output: " + response.getOutput());  
  
  // Clean up when finished. WARNING: Async jobs must currently be cancelled manually.
  Gomobile.rcloneFinalize();               

• A caveat note that the library currently does not include any further model, serialization or job management, i.e. users must built something like RcloneRcd manually.

• An additional note that iOS has not been tested (but should probably work).

Do you think you could do a pull request which checks the gomobile build works rclone's android build? Probably just seeing whether gomobile bind -v works would be enough but maybe you have a better idea. In that way we can make sure the gomobile stuff keeps working.

Do you mean as a build target in the PR/repo build.yml workflow? I don't see why not. Since we already have all android ABI architectures as test builds, is a a single arm target enough?

We could also try an ios build, though I'm not sure if GitHub is happy to do that for every rclone PR since that would need to run on actual apple hardware.

edit: I forgot to link https://github.com/golang/go/issues/16876 and https://docs.google.com/document/d/1y9hStonl9wpj-5VM-xWrSTuEJFUAxGOXOhxvAs7GZHE/edit - these are some of the original proposals for the Go <=> Java binding, and contains details that aren't in the regular docs.

[ncw]

Built a simple test, looks good :

Excellent!

The resulting package name is pretty ugly though, Java usually pretty strictly follows reverse domain, e.g. org.rclone.mobile or something. This is not a real problem since there can only ever be a single gomobile library in an application, so no collisions.

Is there something we can do about this? I'm thinking probably not.

I'd appreciate advice on some more stuff to write here: https://github.com/rclone/rclone/tree/librclone/librclone#gomobile I don't really know what to write since I know so little about it! [snip]

I'll wedge those into the docs - thank you :-)

Do you think you could do a pull request which checks the gomobile build works rclone's android build? Probably just seeing whether gomobile bind -v works would be enough but maybe you have a better idea. In that way we can make sure the gomobile stuff keeps working.

Do you mean as a build target in the PR/repo build.yml workflow?

Yes

I don't see why not. Since we already have all android ABI architectures as test builds, is a a single arm target enough?

Yes I think a single ARM build is fine.

We could also try an ios build, though I'm not sure if GitHub is happy to do that for every rclone PR since that would need to run on actual apple hardware.

We do have macOS build, but I think we should leave that for the moment just to keep things simple.

I'll update the docs and merge the librclone branch tomorrow.

[x0b]

The resulting package name is pretty ugly though, Java usually pretty strictly follows reverse domain, e.g. org.rclone.mobile or something. This is not a real problem since there can only ever be a single gomobile library in an application, so no collisions.

Is there something we can do about this? I'm thinking probably not.

Actually, there is. New build command with -javapkg:

gomobile bind -v -target=android -javapkg=org.rclone github.com/rclone/rclone/librclone/gomobile

New example:

   // imports
   import org.rclone.gomobile.Gomobile;
   import org.rclone.gomobile.RcloneRPCResult;

   // initialize rclone
   Gomobile.rcloneInitialize();

   // call RC method and log response. 
   RcloneRPCResult response = Gomobile.rcloneRPC("core/version", "{}");
   Log.i("rclone", "response status: " + response.getStatus());
   Log.i("rclone", "output: " + response.getOutput());  

   // Clean up when finished. WARNING: Async jobs must currently be cancelled manually.
   Gomobile.rcloneFinalize();               

[ncw]

Thanks for that!

I've put that into the librclone README and merged it to master!

Let the games begin :-)