Roadmap for Botan 2.0

[webmaster128]

There are some notes about the Roadmap of Botan 2.0 and @randombit told me, Botan 2 should come by the end of the year.

From other projects I learned that major version jumps are make for breaking things, not for adding things. The latter can be done in the current version as well.

So I am asking myself as well as Jack and all the other users of Botan: which braking changes need to be done and how urgent are they? How important is timing? Is Botan 2.0 for usage only or for marketing as well?

[randombit]

Labeling something as 2.0 means that it will be supported in an API compatible manner, and wherever possible also remaining ABI compatible (this matters greatly to distributions), for some reasonably long duration, say 18 to 48 months to provide a rough idea.

I agree that breaking things is the more important aspect of this development period. Many things can be done without breaking API such as new algorithms like BLAKE2b or scrypt, integration with OS services (CommonCrypto, CryptoAPI, /dev/crypto), wholly new protocols/formats (OTR, PGP), as can most optimizations and additional side channel countermeasures.

My basic plan is to continue focusing on API breaking changes until they are either done or I've convinced myself to put them off to a later development branch (for example redesigning the filter/pipe system to not require memory copies has entered this category ... maybe in 2.2). At whatever point I believe API changes are complete, declare a first release candidate and begin focusing on test improvements, build/release infrastructure and documentation, with 3 to 5 time-based release candidates before 2.0

I would also like to arrange for at least one third-party audit prior to the 2.0.0 release, resources permitting.

The major API breaking projects left in my todo-for-2.0 list are

• TLS is still missing a number of features I deem critical to a production stack. These include OCSP stapling, key pinning, and X.509 name constraints. I anticipate incompatible API changes to support some of these changes. Supporting the extended master secret and finite field extensions will also be important for 2.0, but shouldn't break API.
• ECC parameter refactoring. Certain optimizations and side channel countermeasures are impossible due to the current design which splits the p/a/b parameters out from the rest of the group information. And currently all of these params are copied around everywhere as BigInts, even though the values are fixed and public, which I'd also like to fix.
• Remove the intermediate pubkey algorithm classes (IF_Scheme_PublicKey, DL_Scheme_PrivateKey, etc) which are kind of useless and can be easily replaced by a small library of helper functions. This is basically a refactoring but does break API as these are nominally public classes.
• I want the ffi layer, which is intended to provide a long-term stable C89 API/ABI especially intended for language wrappers, to be pretty solid. It doesn't have to be 'finished' (what ever is), but I'd like to be confident that what is included in 2.0 is going to work well and not require too many breaking changes. Ideally TLS would be also available from ffi/python for 2.0.0 but if that ends up going into a point release later on that's ok. Ideally before any API freeze ffi would be validated using some other language very unlike Python/ctypes (Rust or OCaml maybe?) to ensure I haven't done anything that happens to be easy for ctypes to handle but hard for other languages.
• The command line tool has its origins as a set of example programs that were included in the docs, and they have inconsistent behavior and are often not flexible enough to be usable in practice. I'd like the botan command to be at least usable enough to replace similar uses of openssl for common tasks like creating a cert, encrypting a file, testing a TLS connection, etc. Alternately the CLI interface can be noted as still in flux for 2.0 and incompatible changes made as needed through 2.0.x. So this one is not necessarily a release blocker, it just depends on what we're willing to commit to supporting.

[webmaster128]

Great, thanks Jack! Has there been such and API freeze for 1.11? Or is Botan 1.11 the working draft between Botan 1.10 and Botan 2.0?

I am asking myself, how we can preserve ABI compatibility given the limited resources. Can that be automatically tested somehow? I am asking because even huge projects with great amounts of resources any paying enterprise customers like Qt make mistakes with that.

[randombit]

I would characterize it as approaching the freeze but not there yet. All the really broad API modifications (eg switching to std::vector with a custom allocator instead of a custom container, or switching cipher modes to process without making a copy) are in. But if an API can be dramatically improved in usability I'm willing to break it now to improve the life of future users, and have specific plans in mind in that regard in a couple of areas (mostly in TLS and the FFI layer).

I agree ABI compatibility is difficult, and definitely constrains how bugs and features can be addressed. It might be better to only guarantee API compatibility in 2.0.x, and when ABI breaks must occur document them, change the soname and move on. This would allow much more evolution to occur within the stable tree, which would make that work more immediately available to users. I'm not sure how productized it is but I know GCC has a system for ABI checking that they use with libstdc++, and likely other big C++ projects (Qt, Firefox, ...) have something similar. Even if we avoid guaranteeing ABI compat, being able to notify users to expect breakage would be useful; most of the time a change breaking ABI is obvious (adding a new virtual function or class member), but I accidentally broke ABI (at least) once in 1.10, which had quite minimal changes over its lifetime so an automated checker would be best for a more active 2.0.

[webmaster128]

Since we seem to agree in #128, I think removing the LibraryInitializer entirely should be on the list.

[cdesjardins]

Here is another sort of cleanup idea for rel 2.0, get rid of the configure.py time header file copy. I recommend having a directory structure like so:

botan/src/lib
botan/src/include

or

botan/src/lib
botan/include

where the include dir is basically the external interface for botan, which gets copied to the install directory at make install time, but otherwise doesn't move. This will eliminate the need to re-run configure and do a full rebuild every time one of the external header files is tweaked.

[webmaster128]

The reason why Botan needs to be re-build after every change is that dependencies are not set properly in the Makefile. Each .o file depends on one .cpp file only and not on the headers it includes. Having no list of header dependencies is the price we pay by no using a build system like cmake or qmake. This problem will not disappear by your suggestion.

If you want to have a stable file modification times based on the header change time, not on configure time, you can use --link-method=hardlink on UNIX and Windows.

Anyways, I do not know if and why we really need to copy headers at configure time. So your suggestion might still make sense.

[cdesjardins]

Indeed this doesn't fix the dependency issue, but I think it is a good first step towards making that problem solvable. Furthermore if you want the include dir to have some structure like the src dir, then now is the time to do it, and dependencies can be fixed later without breaking any external code.

[webmaster128]

One thing I did not understand entirely yet: Is there going to be a stable version of 1.11? Or is it 1.10 and then 2.0?

[mouse07410]

Well, I hope there would be a stable 1.11 release.

Sent from my iPad

On Jul 23, 2015, at 06:29, Simon Warta notifications@github.com<mailto:notifications@github.com> wrote:

One thing I did not understand entirely yet: Is there going to be a stable version of 1.11? Or is it 1.10 and then 2.0?

— Reply to this email directly or view it on GitHubhttps://github.com/randombit/botan/issues/141#issuecomment-124050908.

[randombit]

No, odd minor versions (1.7.x, 1.9.x, 1.11.x, 2.1.x) indicate a branch which is not API stable. What is currently 1.11 will eventually be stabilized on a 2.0 release branch, at which point master becomes 2.1.x and progress continues with backports as needed onto the release branch. This branch scheme is described in the roadmap but I don't know if anybody reads that, maybe it should move to the readme.

[webmaster128]

Ah now it makes sense, thanks for clarifying. I am sure I read the document before I started this thread but probably not carefully enough. Maybe a small table with the the last versions (e.g. 1.9–2.1) helps more than moving the paragraph around. I don't think we need it on the because is too much detail for someone to get started.

[randombit]

We're here! :)