Shared library versioning

[infinity0]

Any plans to adopt a versioning scheme? It's necessary to be able to deploy this as a system library, e.g. as part of a Debian package (which I can do).

You have to set fields like soname/soversion, and it's probably easier to do that by using CMake or GNU autotools, rather than crafting the makefiles by hand as currently.

[herumi]

I have no plan to do so now, but I have already made a CMakefileLists.txt for mcl then I think that it is not difficult for bn(=mcl + bls) to use CMake. I'm not familiar with soname/soversion, then I'm happy for your help.

cf. Now I'm developing mcl to support BLS12-381(this BLS means type of pairing), then the api of bls(this is signature) may be changed.

[infinity0]

I sent a pull request to herumi/mcl#21 which adds a shared-library versioning scheme, please take a look. It includes some comments explaining how the versioning scheme is supposed to work and when you're supposed to bump it. Notably the version can be separate from the "package release version" so you can call it "mcl 1.2" but the shared library version/soversion might be something entirely different like 5.1.6/5.

As for cybozulib and xbyak, are they likely to be directly-depended on by other packages apart from mcl (and not indirectly through mcl)? Packaging 3 Debian packages is of course more work than packaging one. Would it be acceptable for me to bundle these headers as part of the mcl package and distribute them that way?

[infinity0]

Would it be acceptable for me to bundle these headers as part of the mcl package and distribute them that way?

For example, how often do they change in a non-backwards-compatible way, and how does this synchronise with similar changes in mcl.

[herumi]

1. CMakeLists.txt is a subset of Makefile. Makefile is necessary to build some asm files and evaluate many benchmark and tests, so I dont want remove Makefile.

How about

mkdir build
cd build
cmake -f .. && make

2. The suffix of shared name is optional.

make SHARE_BASENAME_SUF=

generates lib/libmcl.so . The reason to assign different suffix is to use not libmcl.so but libmcl.a for some other application. The linker seems to use libmcl.so instead of libmcl.a if there are in same directory. cf. drop _dy suffix from names of shared libraries I would appreciate it if you tell me the way. But I want not to change the name for backward compatibility if possible.

3. Would it be acceptable for me to bundle these headers as part of the mcl package and distribute them that way?

I see. To copy those headers to mcl/include is at the time to make a debian package? or should mcl have those files?

[infinity0]

Regarding Makefile, running cmake in another directory is fine for me. An alternative method is to rename Makefile to Makefile.dev and remove the duplicate functionality that CMakeLists.txt provides, then you only have to maintain one copy of the building logic. I think it may be cleaner, do you agree?

make SHARE_BASENAME_SUF=

OK, I will need to add a similar thing to CMakeLists.txt then.

The linker seems to use libmcl.so instead of libmcl.a if there are in same directory. cf. drop _dy suffix from names of shared libraries I would appreciate it if you tell me the way. But I want not to change the name for backward compatibility.

As mentioned on that bug report it is standard to use the -Wl,-Bstatic flag to gcc/clang for this, which is supported by ld.bfd ("traditional" ld) and ld.gold (newer), both part of GNU binutils, as well as ld.lld from LLVM. I am not sure why the other bug reporter said it's "only GNU", maybe it is because ld.lld is quite new and perhaps didn't support it until recently. I'm not sure about the situation on Windows, but it looks like you're not building a shared library there anyways.

Note that if calling ld(1) directly then the flag is only -Bstatic. Some versions of the lld man page say --Bstatic but I just tested -Bstatic and it also works.

I don't think any tool supports statically linking against some libraries libX.a but dynamically linking against other libraries libY.so, hopefully that's not what you're asking for. -Bstatic will cause everything to be linked in statically.

Most distributions wouldn't pass -Bstatic though, since system package managers can install the transitive dependencies automatically, e.g. installing libbn.so will automatically install libmcl.so if you didn't give -Bstatic.

Would it be acceptable for me to bundle these headers as part of the mcl package and distribute them that way?

I see. To copy those headers to mcl/include is at the time to make a debian package? or should mcl have those files?

The situation is slightly awkward because (on a quick glance) it looks like cybozulib and xbyak may affect the ABI-compatibility of mcl. In other words if I build the same version of mcl but with different versions of cybozulib+xbyak, the resulting libmcl.so files may be incompatible with each other - if I link against one when compiling, I cannot load the other during runtime, e.g. on a different system.

If this is true, or if you don't know if it's true, then it would be best to make mcl "have" cybozulib and xbyak i.e. import (copy) these two into mcl's source directory, require these exact versions, and bump the SOVERSION of mcl every time new versions are re-imported. This would be "safe" according to ABI compatibility but results in duplicate code, which is not nice. (Of course you only need to do the bumping when making a public release.)

If it is false i.e. updating cybozulib and xbyak can never affect ABI compatibility - which is possible if mcl's public ABI does not expose any details of xbyax or cybozulib, then it would be safe to package these things separately in 3 packages, which is advantageous to avoid duplication. Since they are header libraries however, libmcl.so would still need to be continually recompiled to bring in any beneficial updates to the other libraries, but the plus side is that anything that dynamically links to libmcl.so would not need to be recompiled.

(Debian package builds cannot access the network during runtime so I would have to package each 3 separately in this case, I cannot package only mcl and download the header files during the build. If we copy cybozulib and xbyak into the mcl source code, we avoid this issue.)

I'm still not sure of the best way myself here so it would be good if you could give some more information on the above issues I mentioned.

[herumi]

Thank you for valuable information of a linker. I'll check it later. By the way, Do you need a debian package of mcl or bls? If you need only libbls384.so, then it has C-api interface bls/bls.h and mcl/bn.h and mlc/curve_type.h. So if you make a script to copy all source for libbls384.so and CMakeLists.txt to build a debian package of it, then there is no need to worry about ABI. If there is no problem, then I'll make a script.

[infinity0]

I'm trying to get bls/mcl into the official Debian distros so that anyone can do apt-get install bls without needing any extra configuration, this makes deployment easier in much wider scenarios. This involves following Debian policy, which defines ABI compatibility for shared libaries, and various other standards.

For dfinity since we already have our own custom nixbuilds, I don't think we gain much extra by just being able to build a .deb out of the existing libbls384.so if it's not in the official Debian distros. However I understand the extra requirement about ABI compatibility is a burden so if you're not ready for it then maybe we should leave this until later.

[infinity0]

(To be clear, for Debian it is not exactly that one must keep ABI compatibility, just be aware when it breaks, what dependencies can cause this, and bump the relevant version numbers accordingly. It is less strict than Qt for example.)

[herumi]

Xbyak does not affect api because it it used only in building a library. cybozulib affects not C api but C++ api. mcl is an experimental project to develop new algorithm and cryptosystem so api is unstable, then how about making ony bls package?

By the way, I'm sorry to have a very basic question. What is the merit of using cmake instead of Makefile? debian package distributes compiled binary so I don't care for how to build it.

[infinity0]

ABI is a bit stricter than API, but if the API changes then very likely the ABI also changes. The KDE project has some details about this. For the purposes of Debian it is not exactly "Do vs Don't" but only that if you do one of the "Don't"s then the SOVERSION should be bumped.

ABI is roughly, if we build libmcl.so separately from the same source code including dependency headers, and then I build libbls384.so on my system dynamically linked against my libmcl.so and then I send only libbls384.so to you, it still works together with libmcl.so on your system.

BTW when I wrote:

I don't think any tool supports statically linking against some libraries libX.a but dynamically linking against other libraries libY.so, hopefully that's not what you're asking for. -Bstatic will cause everything to be linked in statically.

I was wrong here, you can combine -Bstatic and -Bdynamic to link against some libraries statically vs dynamically, so for example using the test code from here:

$ gcc test.c -lgmp && ldd -r a.out && ./a.out 
    linux-vdso.so.1 (0x00007ffedd7fd000)
    libgmp.so.10 => /usr/lib/x86_64-linux-gnu/libgmp.so.10 (0x00007f042db21000)
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f042d767000)
    /lib64/ld-linux-x86-64.so.2 (0x00007f042dfa7000)
14472
11293
[..]

# Link in only gmp statically.
$ gcc test.c -Wl,-Bstatic -lgmp -Wl,-Bdynamic && ldd -r a.out && ./a.out 
    linux-vdso.so.1 (0x00007ffcce4d0000)
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fd2a50cf000)
    /lib64/ld-linux-x86-64.so.2 (0x00007fd2a56b7000)
14472
11293
[..]

Note that it's important to give the options in the correct order on the command line, each library must come before other libraries that it depends on. For example gcc -Wl,-Bstatic -lgmp -Wl,-Bdynamic test.c fails with the above example.

[infinity0]

mcl is an experimental project to develop new algorithm and cryptosystem so api is unstable, then how about making ony bls package?

In general Debian doesn't like to duplicate code so if bls uses a lot of functionally from mcl, I will have to argue with some people in order to put it in, in the form you're suggesting (bls with mcl statically linked into it). It is a possibility but I'll have to think about it.

Note that for Debian, it's OK for a library to be unstable as long as you bump the relevant version numbers when the ABI changes. I can understand if you don't want to spend effort on that however, at this time.

What is the merit of using cmake instead of Makefile? debian package distributes compiled binary so I don't care for how to build it.

Mainly, it simplifies installing things into standard system directories that distributions expect, and it defines a standard interface to set things like CFLAGS. For example if Debian detects a cmake build system then the build scripts will automatically pass in Debian's build flags like -g -O2 -fstack-protector-strong -Wformat -Werror=format-security etc and I don't have to think about it. Otherwise I have to look through your Makefile and figure out what variables to set, in order to set the build flags that Debian policy wants. Different distributions might also install libraries in different locations (e.g. /usr/lib vs /usr/lib/${triplet}) and cmake also provides a standard interface for distributions to select this.

GNU autotools also provides a similar standard interface for distributions to tweak these things, I also don't care exactly how to build, as long as the buildsystem has this sort of standard interface.

[herumi]

I test a linker option -Wl,-Bstatic ... -Wl,-Bdynamic and verified that it runs correctly. Thank you.

[herumi]

I'm sorry to late response and thank you for detailed description. I understand there are many things to consider. Please give me a time.