Iterate on our publishing strategy for iOS components

[rfk]

We have a couple of different threads going about how to build and publish our components for iOS, so I'm filing an issue to help centralized the discussion.

This repo currently publishes a single component that targets iOS, the FxA iOS SDK. Applications can include this dependency via carthage by including this in their Cartfile:

github "mozilla/application-services" "0.2.2"

Which will download a pre-built binary of FxAClient.framework. IIUC this binary is built and published via travis.

We're soon going to run into issues when we want to publish more than one component for iOS (such as for logins storage) for at least two reasons:

• Carthage expects to build only a single project per git repository, as discussed here.
• Our multi-component story involves making megazord composite builds that combine several different rust crates into a single shared lib.

Let's use this issue to hash out what our build-and-publish pipeline for iOS should look like in a multi-component world.

[eoger]

It's probably too early for that as it is not ready for prime time but nevertheless we should keep an eye on the official Apple Swift Package Manager for the future.

[rfk]

My naive first impression after a light read of the Carthage docs, is that we could arrange to publish a pre-built binary distribution for each of our composite build targets, with accompanying frameworks for the swift wrapper code of each of the included components.

[garvankeeley]

Prebuilt binaries are great for convenience for a popular package, but I am not sure they are needed at an early stage of this project. Firefox iOS won't use them, we build our own for the certainty of knowing the exact build environment, avoiding binary incompat issues, and IIRC we get debugging symbols this way also.

The number of dynamic libraries used in an app is also a problem, as they impact cold boot performance. Apple's guideline has been to stick to 6-8 dylibs for boot performance, and Firefox iOS is more than double that. Our solution would be to do a static build or just link to the Carthage sources directly from our project. In the latter case (which we currently do in the project for some items), Carthage becomes a glorified git-external system, the advantage being we continue to use a single dependency management system (avoiding the confusion of having some things in Carthage, some things in git-external).

Overall, my hope is that we are able to KISS on this aspect of the project while it is in the inception stage, having two expert-level consumers (Lockbox and Firefox iOS) who can handle some pain points.

I imagine the first few months of development on Firefox iOS to be:

• application-services gets an update to any one of the swift components, someone tells Firefox iOS to update,
• Firefox iOS bumps the sha (or tag if we are getting fancy) for the application-services dependency in the cartfile. Next time devs build they run carthage. The Firefox iOS CI machines run carthage on every build.
• application-services is built locally either by carthage performing a local build, or its .xcodeproj files are a subproject of the Firefox iOS xcode project (which provides source-level inclusion)

[rfk]

Would you want/expect to also build the Rust dependencies, or to pull in a precompiled lib with them in it and just build the Swift wrapper as part of the application build process?

[garvankeeley]

Initially we will be ok with the path of least resistance for app-services team. If we need to compile the Rust deps, we would add that to our setup shell script.

What is the approx build time for that?

[eoger]

424 now allows us to publish multiple pre-built binaries in a single Github release.

Using @rfk first comment's example, a developer using: github "mozilla/application-services" "0.X.X" can now pick and choose any framework(s) we maintain in their application.

They can also choose to build from source as @garvankeeley mentioned, which will build every possible framework in our repository (takes ~10 minutes on Travis CI). As there is no way to choose which schemes to build in a Cartfile (see https://github.com/Carthage/Carthage/issues/437), I believe we can't do better than this for now using Carthage.

[rfk]

I dug this issue out of the backlog, as we now have two active iOS consumers working on integrating our components - Firefox for iOS for bookmarks, and Lockbox for iOS for logins.

@justindarc @garvankeeley do the above comments about building from source reflect the approach you ended up taking for consuming things in Firefox for iOS?

@thomcc you mentioned in triage that you had some concerns about how to do megazording on iOS, please elaborate here when you have a moment.

[justindarc]

I would ideally like to have pre-built binaries for this. The real issue here is that Carthage is very simplistic and will not let us pick and choose which components we want to install.

However, just because Carthage builds or installs a pre-built binary, it doesn't mean your end project will automatically use it. The consumer of the components still needs to configure their Xcode project to pull in the .framework they intend on using.

Therefore, I suggest we build a single ApplicationServices.framework that builds a megazord with all the components for Firefox for iOS. Lockbox can continue using the FxAClient.framework and Logins.framework.

[garvankeeley]

Binary Frameworks

@rfk I foresee Firefox iOS building from sources for the next few months, as we will be staying tightly integrated with the HEAD and not waiting for pre-built binaries (as I assume these aren't being generated automatically).

That said, once things start to stabilize, pre-built binaries are the way to go.

The type of binary supplied is an important consideration. Due to the size of the Megazord as a dylib, and the amount of linking time this will cost on startup, and the fact Firefox iOS suffers from slow startup time due to dylib load, we should be looking to statically link this. A pre-built static archive is what we would want. This will also strip unused code and reduce the binary.

Megazord vs multiple frameworks

IIUC the negative to the Megazord is the size of unused stuff (for Lockbox in this case), as for Firefox iOS a Megazord is fine as we will use the entire thing. I think a static library is the correct solution for reducing the binary size, both for Lockbox and Firefox. (As well as optimizing the startup time). For avoiding the Megazord, getting carthage to build a specific build configuration is something I can help with, so perhaps I should be engaging myself in more discussions on that topic.

[garvankeeley]

I am going to do a quick† PR for a Megazord lib for iOS. For one, it will help me understand the quirks that I may not be considering, but also, it will give us a concrete discussion point.

† quick could be my first false assumption!

[rfk]

Thanks for the feedback everyone. To check if I understand the current plan, here's what I'm hearing:

• Short-term, @justindarc is working on building the rust components from source as part of firefox-ios buddybuild, because waiting for releases is too slow a feedback loop at this stage. We don't feel like we need to invest in doing snapshot builds to smooth this out, but might consider those in future in a similar way to android-components.

• Next, we will use @garvankeeley's work to build a single static "everything" megazord. We update our build artifacts bundle to publish this MegazordAllTheThings.framework alongside the existing individual frameworks such as FxAClient.framework and Logins.framework, and once the rate of API change has subsided, firefox-os will start consuming that pre-built framework. Lockbox will continue to use the two individual frameworks, or may choose to experiment with using the everything megazord and see if seems acceptable.

• We learn from that initial megazording experience, and investigate how to build and publish multiple megazords, e.g. Everything-megazord.framework and Lockbox-megazord.framework. We stop publishing individual component .frameworks and consumers have to choose the megazord build that best meets their needs, or compile from source.

• At some point in the distant future, we investigate how to interact with other iOS packaging tools such as the switch package manager.

In all cases, the consumer experience looks like:

• Add "application-services" github repo to your carthage config
• Let carthage pull down all the things, or build all the things from source
• Tweak XCode project settings to pull in the appropriate .framework(s) from the results of the carthage build.

Does that sound broad-strokes accurate?

[justindarc]

Yes to all of the above ^^

[garvankeeley]

sgtm. I decided to call the iOS megazord RustAppServices.framework because that seemed the most semantically meaningful indicator of what it contained, rather than including the 'megazord' name which is largely a packaging description. I would change the name rather than bikeshed this :), but until told otherwise I'll keep that name.

[rfk]

👍 I also steadfastly refuse to bikeshed the name at this point, it seems like something we can easily change later. (And also I purposely used awful names in my description above so that nobody would be inclined to take my opinion on the naming too seriously).

[joeyg]

Sounds good to me. The only thing I haven't tried is using Carthage and pointing to a branch or commit instead of a release. Not sure if that is expected to work.

[rfk]

The only thing I haven't tried is using Carthage and pointing to a branch or commit instead of a release. Not sure if that is expected to work.

IIUC this won't work, at least not yet, because we only build the artifacts on release branches.

[rfk]

We got to a good decision here, and our next steps are just to try out the things we've built and see how they work in practice. I'm going to re-title this issue and put it back in "backlog" for now.

[garvankeeley]

I am going to close this as the publishing strategy is decided for the first iteration of this product, and if it needs to be changed a new bug could be filed to specify what the new issue/direction/strategy is. Or re-open if one of the original outstanding issues wasn't addressed.

To summarize for others: Carthage built frameworks for Logins, FxA, and the composite (megazord) are all bundled together in the dependency package. And the client-side gets to choose which one they use (i.e. link against) in their application.

So, in the Cartfile, github "mozilla/application-services" "some.version.num" will pull in a MozillaAppServices.frameworks.zip, which will place all the frameworks in your project's Carthage/ dir, and you can choose which one to use.