Signing Erlang modules

[ophirzk]

Hi, 

Does Erlang support any form of code signing (basic or EV certification)? I would like to sign Erlang modules and validate their authenticity before execution with trusted Root CAs, Revocation List, Timestamping, etc... 

Is that possible?

[KennethL]

There is currently no support for code signing of .beam files if that is what you mean. Why do you think that is needed if you are fetching singed packages containing .beam files?

I found this written in Elixir. Not that I am convinced that this is necessary, but maybe it is in the direction you are asking for? https://hexdocs.pm/code_signing/CodeSigning.html

[max-au]

EEF Security WG was looking at this problem about 2 years ago. We could not find a viable solution, as signature checks must be happening in the VM itself (which must also be signed, otherwise there is no chain of trust). Several components are missing:

• ability to work with keys/certificates on the VM level (and at that point, OTP did not even support OS-supplied CA Root store)
• compiler would need to have an extra pass (signing pass), and it would need to somehow know the developer private key (to sign bytecode)
• lots of tooling won't be compatible with this (e.g. cover, or anything else that uses AST stored in the debug chunk)

[KennethL]

I am first interested in defining a use case where code signing of Erlang modules (.beam files) makes a big difference when it comes to security.

For Windows we happens to provide a pre-built installer with a development system. The installer can be seen as an executable ,and it is also signed. So when running the installer you get a warning if it is not signed, thus you get no warning here. Installing Erlang/OTP means that all the files you are installing are coming from a trusted source. With our installer there are a number of .exe and .dll files installed and these are not signed. When any of these executables are run you get no warning despite that fact that they are not signed. So what extra security would it give to sign these executables? As I understand it the warning for an executable that is not signed is only triggered if an individual .exe file is downloaded and executed via the browser. But we don't offer individual download of these executables.

Anyway the executables could be signed by us during the build process (but does it add extra security?).

Next step is the request to sign .beam files. If they are not installed individually but only via packages which are code signed then what extra security would that give? To support this would like @max-au writes above require a new chunk in the beam format with place for the certificate and that the loader and all tools can handle that. I assume this would add substantially to the startup time of a system.

And note that there is nothing in Windows that care about if the .beam files are signed or not. On other OS:es (Linux and MacOS), I don't know if the code signing is standardized and checked like on Windows. It is also so that we (the OTP team) are not providing binaries on the other platforms so the responsibility for code signing must be on someone else.

So if there are ideas and PRs regarding how code signing can be done in practice and why it is important please bring it forward here. We have:

• Signed installer (already there)
• Signed executables from the installer?
• Signed .beam files from the installer?
• What about applications fetched from Hex PM and built with rebar3?

[max-au]

Audit is the primary use-case. In many companies, all code running in production must be digitally signed. It is used to ensure that production code complies with various rules and regulations. Usually it's also ensured that toolchain used to build the code is also signed/secure (that is, no developer can deploy the code in production, only a robot can, and the robot deploys only the code that has been committed to the source control repository). This approach ensures that code at least was reviewed (another regulation/requirement that companies has to follow).

Initially I wanted to do a "poor man's signature", replacing (or extending) md5 attribute that already present in the compiled module, with cryptographic signature. That does not need and extra chunk, and it can also leverage existing infrastructure handling "significant" chunks (also used by hot code upgrade to check whether the loaded module has the same code as the module on disk). But signature verification has to happen before loading any *.beam files, and it needs crypto.

For open source work, in addition to signed installers, and signed *.beam files (which larger companies would anyway build on their own, using their own certified build toolchain), there is also a need to teach compiler to sign the .beam files (using a private key from the engineer operating the compiler).

hex.pm distributes applications as source files, so they will be automatically covered by code signing in the compiler. Rebar3 should also receive a config parameter support for code signing (to pass it to the compiler).

[KennethL]

I would like some comparisions with how this is done in other languages. I googled around and saw some info about Java where they sign .jar files which sounds reasonable. If we move that approach to Erlang we should sign a few archives instead of individual .beam files. I think that makes more sense. The Erlang runtime should then refuse to load modules if they don't reside in a signed archive. The collection of modules into 1 or a few archives can be done when making a release.

[okeuday]

@KennethL For hot-code loading, a module represented by a .beam file would provide the code change. So, for Erlang it seems better to pursue being able to sign modules by signing .beam files. I don't see archives as beneficial, only a zip archive that helps to conceal its contents. However, if archives were signed, any code change involving them wouldn't be atomic.

[KennethL]

@okeuday I know that a module is the smallest unit for hot upgrade, but I think you have to sacrifice something and hot code upgrade is actually not used much at all in produktion. My experience is that most systems are restarting the whole E-node when upgrading. This since the hot upgrade is not supported by other environments so then it is hard to use it when you have combinations of different languages/processes in the same system. Possibly you are also running in containers, Kubernetes or similar and then the upgrade is not as fine granular as an Erlang module. Signing individual .beam files when you have hundreds of them would just be to much or to inefficient, Maybe it can be supported but better not used. Better to collect many modules together which is kind of what Java does.

[okeuday]

@KennethL I understand hot-code loading isn't typically used in production because it requires extra effort, knowledge and considerations, so there is an extra cost when using it (it could be considered an extra labor cost). However, hot-code loading is a fundamental part of Erlang's design to pursue high-reliability, so it should be wrong to not support code signing at the module-level. The goal would be to ensure people are capable of using hot-code loading in production if they are able to (so helping technology improve in the future is the goal).

Everyone generally wants to have systems be more reliable, but typically systems suffer due to the economics involved because labor costs are always minimized to the lowest level possible, as part of competition. Virtualization helps systems suffer when they want accurate time-keeping and complete CPU capacity. So, there is a reduction of quality that is accepted with extra levels of virtualization because it is justified with a reduction in cost (a combination of labor and infrastructure costs). For example, hard real-time systems will always want to avoid the extra complexity of virtualization to pursue their high-reliability (typically to avoid loss of human life and/or the loss of money).

I have trouble seeing archives as beneficial for Erlang. The compression of archives is not required normally to avoid storage consumption. If archives are avoided, that always allows systems to be simpler with more transparency about the source code being executed.

I do believe a namespace concept would be beneficial for Erlang to help manage larger amounts of source code. I understand the Erlang packages concept had problems in the past and was removed, but that looks like the feature in Java that Erlang is currently missing (e.g., tool support for having namespace prefixes on Erlang modules in a standard way).

[josevalim]

I have trouble seeing archives as beneficial for Erlang.

Archives as today has little benefits but they could most likely be reworked to provide features around security, runtime performance, module loading performance, etc. There is a lot you can gain if you assume modules will always be loaded together. :)

[max-au]

I know that a module is the smallest unit for hot upgrade, but I think you have to sacrifice something and hot code upgrade is actually not used much at all in produktion

Actually, the most common production use for hot code load is to load a single module without restarting ERTS (so called "canary", temporary production deployment on a single machine to manually verify regressions). Another common case is firefighting, loading a single (or a small amount of modules) in some/all machines of a cluster to mitigate an incident without going through a full restart cycle.

Archives could be a pretty neat concept (especially tied to applications, that'd be quite analogous to Rust crates and JARs), but I think code signing needs to be implemented at a module level. I don't think there could be any difference in signature verification performance. Because regardless of the packaging mechanism, code loader would need to get some hash of the "significant chunks", omitting various attributes and other non-code items (e.g. documentation chunks).

[ophirzk]

@KennethL Is there a plan to start supporting code\module signing in Erlang?

[VaronisContributor]

Signed executables are important for EDRs. There are policies that require all loaded assemblies to be signed, no matter what was the installation file that created them, since this relation is difficult to validate. What if a virus modified the file after installation?

Do you have milestones for different types of signing support integration? MS1: Signed executables MS2: Signed .beam files etc...

Thank you!

[ophirzk]

@varonisarchitect The executable was already signed: https://github.com/erlang/otp/issues/6885

This thread is about signing Erlang modules.