search

AnarchyTools / atbuild

The Anarchy Tools build tool
Apache License 2.0
6 stars 1 forks source link

Add atbin support #89

Closed drewcrawford closed 8 years ago

drewcrawford commented 8 years ago

atbin is a proposed binary interchange format for atbuild and the broader AT ecosystem.

Motivation

atbuild has a weakly standardized input format: build.atpkg. But what does atbuild, um, actually build? What's the output format?

There is no weak standard here, or even a convention. It may build an executable, a static library, or a dynamic one, or even a framework; it may emit swiftmodule and swiftdoc files, or not. A modulemap may or may not be part of the build products and clients may or may not need it in their search paths.

The uncertainty here complicates interoperability. atpm should download binary libraries, but what actually lives in that tarball? Some random dylibs we found in .atllbuild/products?

How do we build libraries for "fat" archs (like iOS, with 4 sub-archs)? How would we write an atinstall tool that installs/updates atpm/atbuild (or makes homebrew packages out of them)?

atbin proposes to answer these questions, providing a simple, portable, hackable binary interchange format for all platforms and all AT projects.

Specification

An atbin is a folder that ends in .atbin. It contains, at least, a built.atpkg file.

built.atpkg is clojure syntax identical to the more familiar build.atpkg. You can include tasks or w/e in there, although why would want to, I'm not totally sure (this is Anarchy Tools though, knock yourself out.) However, the important bit is this:

(package
    :name "foo"
    :payload "libFoo.a"
    :platforms ["ios-x86_64" "ios-i386"]
    :type "static-library"
)

(Other fields could also be present, this is not a complete enumeration)

This .atbin folder will then contain:

You can, of course, build an .atbin by hand from existing binaries you found lying around your disk. And we may eventually ship an atbin packager for Xcode or SwiftPM projects.

However more practically, we introduce a new tool, packageatbin, which packages an atbin payload from atllbuild.

:package {
   :tool "packageatbin"

   ;; Generate a mypayload.atbin
   :name "mypayload"

   ;; When used with the new --platform ios, will build a fat binary for all iOS platforms.
   ;; Alternatively specific platforms can be listed here
   :platforms ["all"]

   ;; The atllbuild task to package.
   ;; Special logic will re-run this task for each platform and merge the resulting output.
   :atllbuild-task "myatllbuildtask"
}

Applications

The obvious application is as an interchange format for prebuilt atllbuild dependencies. Presently, atllbuild can link with the output of any dependent atllbuild task, but if a library wasn't produced by a dependent task as part of the current build (but was say produced on a different computer a month ago) there's no "obvious" way to link to it. #13, https://github.com/AnarchyTools/atpm/issues/1. This PR does not actually include any of that tooling, but it would be a straightforward extension of this work.

An second application is the building of fat files. Currently, there is no mechanism to build a "fat" library or binary in atbuild, or even to specify that we want one, which for iOS is critical #29. Under this PR, we can do it.

A third application is a distribution format for executables. If an .atbin contains an executable, atpm (or hypothetical atinstall) could install/update/administrate that executable similar to homebrew or apt, and keep all your buildtools (or other random crap) up to date. We would need to extend this with version fields and whatnot, but again, it's straightforward.

An fourth application, and my real motivation, is as an intermediate binary representation. An atbin can be "downcast" with another tool to a platform-native format like deb, bottle, or Framework. This would allow us to cut debs, rpms, and framework releases with appropriate AT tools.

Alternatives considered

One alternative is to adopt an existing "standard", like Framework, for this purpose. Indeed, atbuild currently produces frameworks on OSX.

There are some complexities of extending frameworks to this case. For one, the Framework implementation is warty and involves a lot of symlinks and things like codesigning. We don't currently maintain that code for Linux hosts, nor is the standard especially sensible for Linux, as it relies on plists and choices basically unpopular on that platform.

For another, frameworks are not really built to house static library or executable payloads, which are important to atbuild. There are air- quote "obvious" ways to extend to nontraditional payloads, but IMO this is more confusing than it is helpful. An explicit step to "cast down" your atbin to a framework lets us check that your framework will actually make sense to the likes of Xcode.

For a third, it's unclear what platform some random Framework is built for, and what architectures it supports. You can find out by scripting out to platform-specific tools, but it's not portable.

Another alternative is to support multiple payloads/libraries in a single atbin, "one atbin to rule them all". However I don't see what we accomplish there that we don't accomplish with multiple atbins, except specification complexity. So let's not do that, at least not initially.

Core status

packageatbin is included in core primarily because it needs tight, source-level integration with atllbuild. In addition to peeking at the atllbuild options it needs to run the atllbuild task several times in order to produce fat binaries, which means it has to work around the usual dependency pruning logic. For that reason it can't be sensibly implemented via the current custom tool API.

dunkelstern commented 8 years ago

Good idea, but atbuild vs atbuilt? If you scan a directory fast you may probably confuse them.

Perhaps deploy.atpkg or package.atpkg would be better names.

owensd commented 8 years ago

I've only done a brief look, but seems sounds.

dunkelstern commented 8 years ago

Do fat libraries also work for linux or is only 64 bits x86 supported there?

drewcrawford commented 8 years ago

We only support x64 on Linux.

I believe Swift only supports x64 on Linux. Obviously I'm in favor of adding whatever architectures someone needs. I need fat iOS, so here we are.