Generate redistributable binaries for foreign languages

[zduny]

• [x] Add bindings generation to CI
• [x] Investigate what redistributable format is used for library/package distribution for each of foreign languages (Kotlin, -Swift, Python, Ruby)
• [x] Figure out how to pack generated bindings into a single "redistributable" package/library (separately for each foreign language).
• [x] Add redistributable generation howto to README.md
• [x] Add redistributable generation to CI
• [x] Investigate what package managers/repositories are used by foreign languages (Kotlin, Swift, Python, Ruby)
• [x] Add automatic (decide on appropriate trigger) library/package publication to CI

[eloylp]

Alright, taking over this. Initially i thought in doing a bash script for generating the bindings for the different languages. But after thinking about it, i think i am more for creating a little CLI in Rust that wraps all the repo workflows, exposing the minimum information to users. Nothing against bash scripts. I think in general they act as good glue code, but when the code starts getting more sophisticated, i consider difficult to do error handling or retries as examples. We are also going to have access to all libraries in the ecosystem.

I see 2 main goals here:

1. The first one, is to produce the bindings for each target language.
2. With the outcome of 1 , see how to pack and distirbute the bindings to each language registry.

Lets start with 1 first and see how it goes !

[eloylp]

We have some progress on this. We have a CLI prototype at 2b4f5f3ddbcb1eba1ad54019bfb0ae3d86de48fd which is able to generate the bindings + the shared library putting all of that together in one place. So by executing:

cargo run -p uniffi-zcash-cli generate

We obtain:

By the way, i still need to check some of the output paths and create a small library on each language to see errors and more specific needs for each language. Also, check ergonomics and how to improve them.

[eloylp]

More progress on this, lets check for each language how the investigation about using the bindings is going. Before start, just a little remembering that all the outputs listed here can be obtained by making use of our custom CLI:

cargo run -p uniffi-zcash-cli generate 

Python

Our current CLI generates the following files in the bindings/python folder of the project:

|── python
│   ├── libuniffi_zcash.so
│   └── zcash.py

One can simply add the following app example main.py to that folder and execute it:

from zcash import *

def get_amount():
    return ZcashAmount(100)

if __name__ == "__main__":

    assert get_amount().value() == 100
    print("OK!")

$ python main.py
OK!

Ruby

Our current CLI generates the following files in the bindings/ruby folder of the project:

├── ruby
│   ├── libuniffi_zcash.so
│   └── zcash.rb

One can simply add the following app example to that folder and execute it:

require "./zcash.rb"

def get_amount
    Zcash::ZcashAmount.new(100)
end

raise "Error !" unless get_amount().value() == 100

print "OK!\n"

$ ruby main.rb
OK!

For this language, the following packages were needed:

$ sudo apt-get install ruby-devel
$ gem install ffi rubocop

Apart from the above, the following configuration was added to the uniffi.toml file (Mozilla UniFFI bindgen config) of the project, so the ruby bindings can locally find the shared library:

[bindings.ruby]
cdylib_name = "./libuniffi_zcash.so"

Kotlin

Our current CLI generates the following files in the bindings/kotlin folder of the project:

├── kotlin
│   └── uniffi
│       └── zcash
│           ├── jna.jar
│           ├── libuniffi_zcash.so
│           └── zcash.kt

We could make use of the bindings by, as an example, adding the following app example in a main.kt file :

import uniffi.zcash.*

fun main() {
    assert(getAmount().value() == 100.toLong())
    println("OK!")
}

fun getAmount(): ZcashAmount {
    return ZcashAmount(100)
}

Then we can compile and execute it by:

## Compile
$ kotlinc -cp jna.jar test.kt main.kt -include-runtime -d app.jar

## Execute
$ kotlin -classpath ./app.jar:./jna.jar:. MainKt

OK!

Note that we need to include the Java Native Access library in the classpath. Currently we are providing the jar file, but the user could download the original version here. We also include the libuniffi_zcash.so file to the classpath by including the local folder . . Of course, this would not be the most common use case. Other project building tools like gradle are expected to be used.

Swift

Our current CLI generates the following files in the bindings/swift folder of the project:

├── swift
│   ├── libuniffi_zcash.dylib
│   ├── libuniffi_zcash.so
│   ├── zcashFFI.h
│   ├── zcashFFI.modulemap
│   ├── zcash.swift
│   └── zcash.swiftmodule

Swift needs a swift module (zcash.swiftmodule ) in order to be able to import our bindings in swift applications. In order to generate such file, we automated the official docs steps in our CLI. Now we are trying to discover how to import that swift module in a sample application. A work in progress.

Anyway, there is more official documentation regarding swift we could use:

• An example ios app
• How to use this output in a xcode environment

Other considerations, questions and next steps

Lets keep in mind that the libuniffi_zcash.so file currently weights around 60 MB. This is due to the Sapling crypto trusted setup params, which are currently included in the shared lib for facilitating life to the user. The alternative would be to look in local paths for files that contains such parameters. I think until this supposes a problem, we can live with it.

As next steps, i think good ones would be:

• Add to CI the binding generation of our custom CLI. This CLI assumes certain fixed paths inside the project, so invoking it in the CI will serve us as early check in case we break something.

• Maybe try to provide a running swift example like in the other languages.

• Would it be a good idea to automatically add this little code examples to the current, each language CLI output ? (as seen above)

• Evaluate how feasible is publishing this generated artifacts on each respective language registry.

[eloylp]

Alright ! we have a CLI tool that automatically generates all bindings for us by invoking the generate subcommand. Now we are in a good position to try uploading such outcome to the relevant package registries. I think a good strategy is to try things manually first, for all the languages and then, add a new publish subcommand to the CLI.

So lets start streaming here the investigation per each language ... :fast_forward:

[eloylp]

Artifacts upload with Python

For python, we need to transform first the current bindings folder structure in something like this:

.
├── README.md
├── setup.py
└── zcash
    ├── __init__.py
    ├── libuniffi_zcash.so
    └── zcash.py

Lets do a walk-through through each file :point_up:

First, lets note we created a subfolder called zcash that will represent our zcash module, moving there all the previously generated source files libuniffi_zcash.so and zcash.py.

In Python, the module discovery system works by placing an __init__.py file inside each module. Such file also works as initialisation step. Here is the current content of such file:

from .zcash import *

Then, we also need a setup.py file, that will instruct the python tooling how to deal with this package. It will also allow us to specify that we have a shared library we also want to upload (libuniffi_zcash.so) along with the source code. Here is the content of the file, left some comments on each field:

import setuptools

with open("README.md", "r") as fh:
    long_description = fh.read()

setuptools.setup(
    name="test-uniffi-zcash",  ## We are just testing right now in TestPy package registry
    version="0.0.1",   ## This probably needs to be parameterised in the CLI 
    author="test",
    description="Zcash test Package",
    long_description=long_description,
    long_description_content_type="text/markdown",
    packages=["zcash"],
    classifiers=[
        "Intended Audience :: Developers",
        "Topic :: Software Development :: Libraries",
        "Programming Language :: Python",
        "Programming Language :: C"
    ],                                     
    package_data={"zcash": ["libuniffi_zcash.so"]}  ## Include the shared library !! 50MB actually :D
)

Before uploading the package, lets ensure we have all the tools up to date:

$ python -m pip install –-user –-upgrade setuptools wheel

And also install twine, a tool that helps with the upload :

$ python -m pip install --user --upgrade twine

Now we are ready to publish our package. Lets do it indicating we want to use the python test package index this time and the previously generated (by python tools) dist folder:

$ python -m twine upload --repository testpypi dist/*

Enter your username: eloylp_eiger
Enter your password: 
Uploading test_uniffi_zcash-0.0.1-py3-none-any.whl
100% ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 54.3/54.3 MB • 00:15 • 3.9 MB/s
WARNING  Received "503: Service Unavailable"                                                                              
         Package upload appears to have failed. Retry 1 of 5.                                                             
Uploading test_uniffi_zcash-0.0.1-py3-none-any.whl
100% ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 54.3/54.3 MB • 00:08 • 6.2 MB/s
Uploading test-uniffi-zcash-0.0.1.tar.gz
100% ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 54.3/54.3 MB • 00:08 • 6.3 MB/s

View at:
https://test.pypi.org/project/test-uniffi-zcash/0.0.1/

Whoa ! we can observe that twine also manages retries, which is nice :heart_eyes: . Our package is already published:

Lets just make an observation regarding upload process. We probably need to use API keys instead on interactive security challenge user/pass .

Lets now install the package from pypi:

$ pip install -i https://test.pypi.org/simple/ test-uniffi-zcash==0.0.1

We can observe how all the stuff, shared library plus bindings code are downloaded (~50MB):

Now, the final test, lets execute some code on the REPL ! :snake:

$ python
>>> from zcash import *
>>> amount = ZcashAmount(100)
>>> amount.value()
100

Yay ! :partying_face: we have our first, python librustzcash version. Lets continue with the rest of languages.

[eloylp]

Artifacts upload with Ruby

In ruby we need to re-organise our previous outcome to accomplish the following dir structure:

├── lib
│   ├── libuniffi_zcash.so
│   └── zcash.rb
├── README.md
└── zcash.gemspec

The lib folder will hold the previous output, which is the shared lib plus the bindings. Its worth mentioning that we needed to change the uniffi.toml configuration to specify the path in which the shared lib should be looked up:

[bindings.ruby]
cdylib_name = "lib/libuniffi_zcash.so"

Then we have the zcash.gemspec , that defines our gem and looks like this:

Gem::Specification.new do |s|
  s.name        = "zcash"
  s.version     = "0.0.1"
  s.summary     = "The librustzcash ruby FFI binding"
  s.description = "A library for interacting with the librustzcash lib, a privacy oriented cryptocurrency"
  s.authors     = ["test"]
  s.email       = "test@test.com"
  s.add_runtime_dependency 'ffi', '1.15.5'                        # Here we require the ffi gem.
  s.files       = ["lib/zcash.rb", "lib/libuniffi_zcash.so"]       # Adding the shared library and the bindings.
  s.homepage    = "https://github.com/eigerco/uniffi-zcash-lib"
  s.license       = "MIT"
end

Its really great the way gems work. Once we have defined the gemspec, we can just build our gem with:

$ gem build zcash.gemspec

Successfully built RubyGem
  Name: zcash
  Version: 0.0.1
  File: zcash-0.0.1.gem

The above will generate the zcash-0.0.1.gem gem file we can just install by:

$ gem install ./zcash-0.0.1.gem 

Time to test in the REPL ! :diamonds:

$ irb
irb(main):001:0> require "zcash"
=> true
irb(main):002:0> amount = Zcash::ZcashAmount.new(100)
=> 
#<Zcash::ZcashAmount:0x00007f37c8016bf0                        
...                                                            
irb(main):003:0> amount.value()
=> 100

Yay ! it works. Lets now check to upload the gem to a local gem server. The docs specify a great way. First we need to install the software:

gem install geminabox
gem install puma
mkdir data  ## a directory for storing gems by the server.

Then create a config file config.ru :

require "rubygems"
require "geminabox"

Geminabox.data = "./data"
run Geminabox::Server

Lets now bring up this guy by:

$ rackup
Puma starting in single mode...
* Puma version: 6.2.2 (ruby 3.1.4-p223) ("Speaking of Now")
*  Min threads: 0
*  Max threads: 5
*  Environment: development
*          PID: 525969
* Listening on http://127.0.0.1:9292
* Listening on http://[::1]:9292
Use Ctrl-C to stop

Great we have it running. Lets now just add it to our gem sources by:

$ gem sources -a http://localhost:9292/

Now we can install a fresh zcash gem from our gem server by:

$ gem install zcash -v "0.0.1"

Fetching ffi-1.15.5.gem
Building native extensions. This could take a while...
Successfully installed ffi-1.15.5
Successfully installed zcash-0.0.1
Parsing documentation for ffi-1.15.5
Installing ri documentation for ffi-1.15.5
Parsing documentation for zcash-0.0.1
Installing ri documentation for zcash-0.0.1
Done installing documentation for ffi, zcash after 4 seconds
2 gems installed

Awesome, we can also observe how the ffi gem is also installed. Lets repeat the REPL exercise once again for doing the last check:

$ irb
irb(main):001:0> require "zcash"
=> true
irb(main):002:0> amount = Zcash::ZcashAmount.new(100)
=> 
#<Zcash::ZcashAmount:0x00007f37c8016bf0                        
...                                                            
irb(main):003:0> amount.value()
=> 100

Alright ! we would be ready to go with ruby. Lets just take into account, we are going to need some kind of API authentication when dealing with the rubygems.org gem server.

Thats it, i would like to mention the great work this docs are. Practically i did not need to go to other sources to accomplish this investigation ! Fantastic.

[eloylp]

A random thought: Even if some of the artifact upload tools (as we observed) provides retries, i think we should implement our owns. I do not think all of them will implement retries.

[eloylp]

Artifacts upload with Kotlin

We used the gradle init command to create a Kotlin library project scaffolding. We adapted it, and the final result is the following one:

.
├── gradle
│   └── wrapper
│       ├── gradle-wrapper.jar
│       └── gradle-wrapper.properties
├── gradlew
├── gradlew.bat
├── lib
│   ├── build.gradle.kts
│   ├── libs
│   │   └── libuniffi_zcash.so
│   └── src
│       └── main
│           └── kotlin
│               └── zcash
│                   └── zcash.kt
└── settings.gradle.kts

Most of the files there are generated by the gradlew init command, except:

• lib --> libs --> libuniffi_zcash.so (Our shared lib)
• zcash.kt (The bindings)

Lets now see the relevant modifications done in settings.gradle.kts:

rootProject.name = "zcash"
include("lib")

And also, the most important part, the build.gradle.kts, left some comments on the relevant parts:

plugins {
    id("org.jetbrains.kotlin.jvm") version "1.8.10"

    `java-library`
    // Added the maven-publish plugin
    `maven-publish`
}

repositories {
   // Use Maven Central for resolving dependencies.
   mavenCentral()   
   // Use local repository for testing
   mavenLocal()
}

dependencies {
  // Add the JNA. No need to manually include the jar: https://github.com/java-native-access/jna
  implementation("net.java.dev.jna:jna:5.8.0")
}

java {
    toolchain {
        languageVersion.set(JavaLanguageVersion.of(17))
    }
}

// Inlude the .so files in the jar. Our shared library is there.
val libsDir = File("libs")
tasks.withType<Jar> {
    from(libsDir) { include("**/*.so") }
}

publishing {
   publications {
    create<MavenPublication>("mavenJava") {
            from(components["java"])
            pom {
        // Artifact coordinates and info. To be setup for production.
                artifactId = "zcash"
        groupId = "uniffi"
        version = "0.0.1"
                description.set("The librustzcash Kotlin FFI binding")
                url.set("https://github.com/eigerco/uniffi-zcash-lib")
                licenses {
                    license {
                        name.set("The MIT License")
                        url.set("https://github.com/eigerco/uniffi-zcash-lib/blob/main/LICENSE")
                    }
                }
           }
    }
   }
   repositories {
     // Below auth config should be adjusted/used for production publishing.
     maven {
        url = uri("https://example.com/repository/maven")
        credentials {
            username = "token" // Use "token" as the username for API token authentication
            password = System.getenv("MAVEN_REPO_API_TOKEN")
        }
     }
  }
}

With the above configuration, we managed to include the shared library along with all the code in the same jar file.

Maven works with a local repository (under the path $HOME/.m2) to which we can publish our library for testing purposes. From the root of the library working dir:

$ gradle publishToMavenLocal

BUILD SUCCESSFUL in 7s
6 actionable tasks: 6 executed

Now , we could create another project with gradle init (this time, application type) and try to import and use our library. After creation, we need to include the dependencies in the application build.gradle.kts:

plugins {
    id("org.jetbrains.kotlin.jvm") version "1.8.10"

    application
}

repositories {
    // Use Maven Central for resolving dependencies.
    mavenCentral()
    // Use Maven local in order to find our previously built library.  <-----------------------------------------
    mavenLocal() 
}

dependencies {
    testImplementation("org.jetbrains.kotlin:kotlin-test-junit5")

    testImplementation("org.junit.jupiter:junit-jupiter-engine:5.9.1")

    implementation("com.google.guava:guava:31.1-jre")

    implementation("uniffi:zcash:0.0.1")  // <-------------------------- Here, requiring our zcash library.
}

// Apply a specific Java toolchain to ease working on different environments.
java {
    toolchain {
        languageVersion.set(JavaLanguageVersion.of(17))
    }
}

application {
    mainClass.set("test.kotlin.AppKt")
}

tasks.named<Test>("test") {
    useJUnitPlatform()
}

Lets now modify the main function in App.kt to use our zcash code:

package test.kotlin
import uniffi.zcash.*

fun main() {
    val amount = ZcashAmount(100)
    println(amount.value())
}

Now we can just run it y going to the root of the application project and executing:

$ gradle run

> Task :app:run
100

BUILD SUCCESSFUL in 776ms
4 actionable tasks: 1 executed, 3 up-to-date

So that's it. We have tested our application. I think we can assume this will work the same way if pushed to a remote maven repo. The only difference would be the network. I wasn't able to find information about maximum upload size in maven, we will see.

Again, all the package metadata should be properly configured before shipping this on production. Like licenses, package names, initial version, auth data etc ...

[eloylp]

Artifacts upload with Swift

Swift comes with another different perspective regarding packages. Each package can have any number of modules. The folder structure is used in a meaningful way by the Swift Package Manager (SPM).

Our strategy will be to create a Zcash package, that will hold 2 modules:

• The zcashFFI module, that will contain our Rust shared library.
• The Zcash module, that will import the zcashFFI layer and will be imported by the final application.

The formal command to start creating library packages in swift is:

swift package init --type library

Then just follow the instructions for creating a Zcash package. The final directory structure must be the following one:

.
├── Package.swift ## Created by us.
├── Sources
│   ├── Zcash
│   │   └── zcash.swift ## part of UniFFI output
│   └── zcashFFI
│       ├── libuniffi_zcash.so ## part of UniFFI output.
│       ├── module.modulemap   ## part of UniFFI output, but adapted to our needs.
│       └── uniffi_zcash.h     ## part of UniFFI output.
└── Tests
    └── ZcashTests
        └── ZcashTests.swift ## Created by swift, adapted by us.

The Package.swift file defines the package itself and all the inter-dependencies among the internal modules:

// swift-tools-version: 5.8
// The swift-tools-version declares the minimum version of Swift required to build this package.

import PackageDescription

let package = Package(
    name: "Zcash",
    products: [
        // Products define the executables and libraries a package produces, making them visible to other packages.
        .library(
            name: "Zcash",
            type: .static,
            targets: ["Zcash"]
        )
    ],
    targets: [
        // Targets are the basic building blocks of a package, defining a module or a test suite.
        // Targets can depend on other targets in this package and products from dependencies.
        .systemLibrary(name: "zcashFFI"),
        .target(name: "Zcash", dependencies: ["zcashFFI"]),
        .testTarget(
            name: "ZcashTests",
            dependencies: ["Zcash"]
        )
    ]
)

The next interesting file is the module.modulemap file. This file comes from LLVM clang frontend. It helps us to map our libuniffi_zcash.so shared library to a swift module. That's it ! swift makes this very easy. It will automatically map all symbols to swift native code that could be called by an application. But of course, we are not going to use such raw nomenclature, as we will wrap such thing with a higher level layer, that is, our zcash.swift UniFFI generated file.

Here is the content of the module.modulemap file:

module zcashFFI [system] {
    header "uniffi_zcash.h"
    link "uniffi_zcash"
    export *
}

We are basically saying "Use this local uniffi_zcash.h header file and link the libuniffi_zcash.so shared library, exporting all the symbols at the scope of this zcashFFI module".

Then, in order to test this library, we added the ZcashTests.swift file:

import XCTest
@testable import Zcash

final class ZcashTests: XCTestCase {
    func testAmount() throws {
        let amount = try! ZcashAmount(amount: 200)
        assert(200 == amount.value())
    }
}

Executing the above test will properly exercise all the elements and give us the confidence that the entire generation chain is working properly (this probably should be done in all languages, and make it part of the CI). In order to execute this test, lets move to our library root, and execute:

$ LD_LIBRARY_PATH=$(pwd)/Sources/zcashFFI swift test -Xlinker -L$(pwd)/Sources/zcashFFI

That's good. We have our library. But how to use it ? Swift is very friendly with Git. That means we can import other Git hosted libraries as direct dependencies in our projects. So lets init a git repo in the root of our package:

$ git init
$ git add .
$ git commit -m "initial commit"
$ git tag 0.0.1

This Package could be easily pushed to a git server and later imported in any application. For demonstration, we are going to create an example application that will import our package by following a filesystem URL to our git repo. Lets start creating our App executable package by:

swift package init --type executable

Now we have the following structure:

.
├── Package.swift
└── Sources
    └── main.swift

The Package.swift should look like this one:

// swift-tools-version: 5.8
// The swift-tools-version declares the minimum version of Swift required to build this package.

import PackageDescription

let package = Package(
    name: "App",
    dependencies: [
        .package(url: "../Zcash", from: "0.0.1") // Specify our local Git depedency.
    ],
    targets: [
        // Targets are the basic building blocks of a package, defining a module or a test suite.
        // Targets can depend on other targets in this package and products from dependencies.
        .executableTarget(
            name: "App",
            dependencies: ["Zcash"], // Import our Zcash module.
            path: "Sources"
        )
    ]
)

An of course, our main.swift file:

// The Swift Programming Language
// https://docs.swift.org/swift-book

import Zcash

let amount = try! ZcashAmount(amount: 100)

print(amount.value())

Now, lets run our application by executing this command from the root of our App package:

$ LD_LIBRARY_PATH=$(pwd)/../Zcash/Sources/zcashFFI swift run -Xlinker -L$(pwd)/../Zcash/Sources/zcashFFI
Building for debugging...
Build complete! (0.16s)
100

For building the production binary, the following command can be executed:

$ LD_LIBRARY_PATH=$(pwd)/../Zcash/Sources/zcashFFI swift build -c release -Xlinker -L$(pwd)/../Zcash/Sources/zcashFFI

Note we are passing the -c release, that will optimize our application. The above will leave the binary in the relative path .build/release/App. Then we can execute it by:

$ cd .build/release
$ LD_LIBRARY_PATH=$(pwd)/../../../../Zcash/Sources/zcashFFI ./App
100

If want to avoid external runtime dependencies and have a statically linked binary, we could try to follow this steps. I am currently getting linker errors, but probably due to my setup. Will edit this comment once i fix it.

Regarding publishing to a real package registry, here we can find all the information here.

It was a bit hard to research the swift part. So leaving here the relevant links that helped me during the entire process:

• https://github.com/apple/swift-package-manager/blob/main/Documentation/PackageRegistryUsage.md
• https://www.hackingwithswift.com/articles/87/how-to-wrap-a-c-library-in-swift
• https://theswiftdev.com/how-to-use-c-libraries-in-swift/
• https://theswiftdev.com/building-static-and-dynamic-swift-libraries-using-the-swift-compiler/
• https://forums.swift.org/t/wrap-a-local-c-so-shared-library/42259
• https://github.com/apple/swift-package-manager/blob/263171977ebcd47f4aaca1202cff5a96c5158a64/Documentation/Usage.md#publish-a-package
• https://forums.swift.org/t/static-linking-on-linux-in-swift-5-3-1/41989
• https://github.com/apple/swift-package-manager/blob/main/Documentation/PackageRegistryUsage.md#publishing-to-registry

[eloylp]

We gained a lot of knowledge during this investigation about packaging solutions for:

• Python
• Ruby
• Kotlin
• Swift

Such knowledge should serve us to continue with the next step: Implement such steps in our CLI for an automated release on the relevant package registries. Later on, we will use this CLI from our CI. Some considerations regarding implementation:

• For the last language we investigated, we added a little test in the library package. That helped to test if the entire generation chain (UniFFI --> our final package) was successful. We think we should do the same for the rest of the languages and execute this tests in CI, just before publishing the packages.

• For the case of swift, we still don't know how to pack the .so file in the library itself and instruct the linker (via the Package.swift) for proper static linking. We also think this also depends on the end user setup. But certainly this shared library will always change with most of the changes in the Rust code, so its an extra point to always statically link it i think. We continue investigating in parallel this.

• Regarding the CLI implementation, we are thinking on saving the different investigated project templates on a folder, that will be used later on by the CLI to configure projects "on the flight" before publishing. We could parametrize such templates with common elements like version, name, description , etc ... so they get defined as arguments of the CLI once, and propagated to all templates.

• As we commented previously, having retries for the publish operation, would be a big plus and probably save manual interventions. We are uploading a weighty package, so the upload time is bigger than the usual.

Happy to hear feedback from you for any of the points ! Lets continue with the plan in the meanwhile.

[eloylp]

More progress on this. An intensive work is being done on this PR regarding the CLI. Most of it is ready at this moment. Lets do a walk-through:

Current CLI architecture

The CLI was splitted in the following sub-commands, which need to respect the following order of execution:

1. bindgen - This command calls all the needed UniFFI machinery for generating the language bindings. It invokes the UniFFI tools under the hood, passing our desired values by default. The outcome of this command is the lib/bindings git ignored folder, which contains the bindings.

2. release - It accepts a version argument among others. This command does not push the artifacts yet. It only prepares them by using a little, in house project template system. This system has predefined projects structures for the different languages, which later are parametrized with a text template engine. It also copies the needed files from the previous command outcome at lib/bindings. The outcome of this command is placed at the lib/packages git ignored folder. It contains the packages ready to be published. This packages are also automatically tested against little sample applications. This sample applications just import the artifact as an user would do, checking the entire import chain/dynamic library loading is not broken.

3. publish - This is the last step and where the development/testing is completely focused right now. The aim is to just execute a single publish command per each language, for finally pushing the previous generated packages at lib/packages. It will implement a little retry logic in order to reduce friction with intermittent problems in the CI. We want to avoid partial releases. i.e One language package registry fails, but others are OK.

Why this CLI architecture ?

Its a layered, multi-stage architecture that will bring the following benefits:

• Giving developers (users) access to the outcome of the three stages, are able to explore other alternative ways to import the zcash code on their applications.
• The ecosystem is changing quickly. Having the three stages clearly separated will help on future debugging scenarios.
• We isolate the external systems call over network to the last step, being able to concentrate the retry logic only in one place and being able to cancel the release if any of the previous steps (like package testing) did not work.
• The bindgen and release commands never publish anything. We can add them as last PR CI checks after normal testing, so ensuring everything is ok on each PR.

Why so much insistence on including the so files in packages ?

• A change in the bindings part always means a new, different shared library. The bindings and the shared library are completely coupled in terms of development cycle. So it makes sense to always be released together.
• Reduce moving parts from an user perspective. Try to just need to import the dependency as it was created on their native language.

Considerations, rough edges during development

Some hardening was done in the CLI on last commits. There are still some "uncomfortable" pieces:

• For the release preparatory step, we need to clone/commit/tag during the swift package preparation (swift works with git as backend for packages). The problem is that when committing in swift package (which is inside the git ignored folder) , if there are uncommitted changes in the parent repo (our repo), sometimes they also get committed ! Some solutions that are on the radar:

  • To prevent executing the release command if there are uncommitted changes in the repo. This should be the normal situtation BTW.
  • To move the automatic swift package generation outside the parent repo. Maybe /tmp ?

• There is a hack in the ruby generated code to be able to load the in-gem shipped shared library. This is due to limitations on the ffi_lib call, as it only accepts absolute paths or libs present in the LD_LIBRARY_PATH. We tried to not touch the UniFFi generated as much as we can but this time we think it worths it. We are more or less ok as we have the automatic sample test applications covering everything.

• Initially we are not going to publish in official, remote registries for testing. Only python is providing a staging test package index for testing AFAIK. We prepared a docker-compose file that brings up mocks of the relevant package registries for testing against them in a more comfortable, faster way. This could brings some caveats, as we could be avoiding testing against each official package registry thresholds, like max upload size of artifacts. Most of the times when this thresholds are exceeded, they can be increased in a per artifact way by contacting with registry admins.

• Currently there is no way to say "Hey CLI, please, just do the release/publish for one language" . This is something interesting in case a specific language registry is down for enough time to exceed the retry thresholds. To be able to resume the release process. The current CLI code would allow this with a very little refactor.

Current, next final steps

• Complete the publish step testing with the docker-compose registries.
• Document the CLI on its own readme.
  • Move some explanations written here about concepts.
  • How to point the CLI to real registries via environment vars with auth, etc ...
  • In general do a good handover for the next devs are going to work with it.

[eloylp]

Okay, we have some final conclusions for this issue:

• The CLI was merged at https://github.com/eigerco/uniffi-zcash-lib/pull/122, fixing all the issues and refactorings mentioned in the previous comment.
• We are also providing the CLI docs at https://github.com/eigerco/uniffi-zcash-lib/pull/124
• Also, waht we think its a good starting point regarding CLI - CI integration at https://github.com/eigerco/uniffi-zcash-lib/pull/123

I think all the goals listed on this issue have been achieved. Moving to its final review ! :fast_forward:

@MeerKatDev

[MeerKatDev]

I think this is done @eloylp we can close this

[eloylp]

I think this is done @eloylp we can close this

@MeerKatDev thanks ! just need approvals for this one first: https://github.com/eigerco/uniffi-zcash-lib/pull/124