Update libthemis version

[gaelph]

Hi !

Our application requires libthemis to be installed via Homebrew. It seems the binaries available there are stuck at version 0.13.0, which does not support Apple Silicon Macs. Some of our users have such Macs and building from source is not a viable solution for most of them.

Could it be possible to update the version available on Homebrew to something more current (as Apple Silicon is supported since 0.13.4) ?

Thank you very much !

[ilammy]

I'm afraid we don't provide any binaries – i.e., bottles – for Homebrew distribution (and never really did). As far as building from source goes, I believe 0.13.0 should still build fine on Apple Silicon CPUs (I can't confirm it now, but fairly sure). We never go to setting up bottle building process because there wasn't much demand for it to offset the cost of doing it properly (we'll need to have both Intel and Apple Silicon hardware available, ideally with all macOS versions installed).

Are there any specific issues that you run into when building for Apple Silicon?

Note that changes in 0.13.1 and later versions affect mostly the Carthage, CocoaPods, SPM distributions of ObjCThemis and SwiftThemis which are unrelated to Homebrew distribution of Themis Core. SPM et al. needed special attention as they are built from an Xcode project intended to integrate well with Xcode projects for apps written in Swift and Objective-C. Homebrew distribution is a way to install Themis Core into the system which is necessary to use Themis with languages other than Swift or Objective-C when building standalone apps in Xcode.

[gaelph]

I realize I could have added some more context : We are building a CLI tool in Golang to share sensitive information between members of a team. We are relying on Themis for key generation and its Secret Messages feature.

A developer on our team has an M1 Mac and could not simply install libthemis from Homebrew and start working: they had to build libthemis from source on their machine instead – it builds and works fine indeed. Still, it is not possible to cross compile our tool from an Intel Mac targeting Apple Silicon due to missing arm64 symbols in libsoter.dylib and libthemis.dylib, when they are installed form the Homebrew formula.

I noticed this trying to build our tool with CGO_ENABLED=1 GOOS=darwin GOARCH=arm64 while I was setting up our CI/CD pipeline, having the following error:

CGO_ENABLED=1 GOOS=darwin GOARCH=arm64 go build -o $TARGET_BIN
# github.com/cossacklabs/themis/gothemis/keys
ld: warning: ld: warning: ignoring file /usr/local/lib/libsoter.dylib, building for macOS-arm6
4 but attempting to link with file built for macOS-x86_64ignoring file /usr/local/lib/libthemi
s.dylib, building for macOS-arm64 but attempting to link with file built for macOS-x86_64

Undefined symbols for architecture arm64:
  "_themis_gen_ec_key_pair", referenced from:
      __cgo_3ee3ff74cff8_Cfunc_gen_keys in _x002.o
      __cgo_3ee3ff74cff8_Cfunc_get_key_size in _x002.o
  "_themis_gen_rsa_key_pair", referenced from:
      __cgo_3ee3ff74cff8_Cfunc_gen_keys in _x002.o
      __cgo_3ee3ff74cff8_Cfunc_get_key_size in _x002.o
  "_themis_gen_sym_key", referenced from:
      __cgo_3ee3ff74cff8_Cfunc_gen_sym_key in _x003.o
      __cgo_3ee3ff74cff8_Cfunc_get_sym_key_size in _x003.o
ld: symbol(s) not found for architecture arm64
clang: error: linker command failed with exit code 1 (use -v to see invocation)

I understood from this issue, that Apple Silicon support was merged in version 0.13.4. Seeing the Homebrew formula at 0.13.0 prompt my initial request.

[vixentael]

thank you for details @gaelph. Looking at it.

[ilammy]

Thanks for the context!

We are building a CLI tool in Golang to share sensitive information between members of a team. We are relying on Themis for key generation and its Secret Messages feature.

Indeed, then GoThemis is the right choice, and it requires Themis Core to be available somehow, with Homebrew being a natural way to try installing it.

It's nice to see tools being built Themis :)

A developer on our team has an M1 Mac and could not simply install libthemis from Homebrew and start working: they had to build libthemis from source on their machine instead – it builds and works fine indeed.

For one, I can confirm that I have reports from other people that Homebrew on M1 machines seems to install libthemis for x86_64. No clue why at the moment. We're looking into it.

I'm not really sure where the problem is, since Homebrew is supposed to build for the host machine if it can't find a bottle.

Just a random guess, does installing with brew install --build-from-source libthemis on Apple Silicon change anything? IIRC, Homebrew has changed the prefix they use, so the library should be somewhere in /opt/homebrew/libthemis. Could you please try it out?

Still, it is not possible to cross compile our tool from an Intel Mac targeting Apple Silicon due to missing arm64 symbols in libsoter.dylib and libthemis.dylib, when they are installed form the Homebrew formula.

Cross-compilation with Homebrew-installed version is going to be tough. The thing is, Homebrew does not normally build fat binaries with multiple architecture slices, so on Intel machines any libraries installed with Homebrew will contain only x86_64 code. It seems to be fundamentally based around building stuff on the host machine for the host machine.

That said, if you forgo Homebrew, I think it could be possible to cobble something together that will work, if you really want to build universal fat binaries with you Go toolchain.

Themis has some support for cross-compilation. It seems to be broken in the current release, but it was quick to unbreak (https://github.com/cossacklabs/themis/pull/849). With that and some elbow grease it should be possible to cross-compile Themis libraries for ARM64. But it will be up to you to combine that stuff into fat binaries.

[ilammy]

I understood from this issue, that Apple Silicon support was merged in version 0.13.4. Seeing the Homebrew formula at 0.13.0 prompt my initial request.

Unfortunately, you have been misled :( Apple Silicon support was added—for ObjCThemis and SwiftThemis which are a different thing. For other languages on macOS we're basically relying on Homebrew to get Apple Silicon support right, and it seems that we or they didn't get it right.

[gaelph]

Thank you for your help. It confirms what I was starting to understand: we probably are better off building everything on the user's machine. That is what we've done and it seems to work on both x86 and arm64. The formula clones the libthemis repo, builds it, and our tool is linked against that. It might be problematic for users who already have libthemis installed, though, but Homebrew is good enough to warn them about it.

Just a random guess, does installing with brew install --build-from-source libthemis on Apple Silicon change anything? IIRC, Homebrew has changed the prefix they use, so the library should be somewhere in /opt/homebrew/libthemis. Could you please try it out?

I asked my teammate to run that command, but, although the installation process ended successfully, building our project failed with the following output:

go: downloading golang.org/x/sys v0.0.0-20210616094352-59db8d763f22
# github.com/cossacklabs/themis/gothemis/keys
ld: library not found for -lthemis
clang: error: linker command failed with exit code 1 (use -v to see invocation)

Thank you both for your help, it greatly help clarifying things.