Review current code sigining methods and identify gaps for improvement

godfreykutumela commented 2 years ago

Goal:

As a Security Specialist

`I want to review existing mojaloop code signing measures to identify gaps and opportunities to improve code integrity assurance.

Acceptance Criteria:

[ ] TBD

Complexity: Medium

Uncertainty: Medium - May not have the dedicated resources to execute on this.

Tasks:

[ ] Review and identify gaps in the current mojaloop code signing measures
[ ] Research and design an improved code-signing process and solution
[ ] Present the solutions to the security workstreams for recommendations and approval
[ ] Implement the recommended improvements

Done

[ ] Acceptance Criteria pass
[ ] Designs are up-to date
[ ] Unit Tests pass
[ ] Integration Tests pass
[ ] Code Style & Coverage meets standards
[ ] Changes made to config (default.json) are broadcast to team and follow-up tasks added to update helm charts and other deployment config.
[ ] TBD

Pull Requests:

[ ] TBD

Follow-up:

N/A

Dependencies:

N/A

Accountability:

Owner: @godfreykutumela @pedrosousabarreto @lewisdaly
QA/Review: @godfreykutumela

godfreykutumela commented 2 years ago

Solution option leveraging a Github integrated service - https://wiki.debian.org/Creating%20signed%20GitHub%20releases

godfreykutumela commented 2 years ago

@pedrosousabarreto and @bukasaaime Can we update this issue with the output of your research and I will set up a session for second week in Feb to review the options and adopt a solution.

pedrosousabarreto commented 2 years ago

Ok, the initial link I sent seems to solve this nicely, only caveat is that we must create a OpenPGP key. Creating it is easy enough, not sure about how we store it. Once we have it, we will be able to cryptographically sign our releases so they can be verified by the downloader.

bukasaaime commented 2 years ago

Good day @pedrosousabarreto , would you re-share the link please.

millerabel commented 2 years ago

Once you have a working prototype process, please work with foundation staff to establish a key custodian strategy.

Some Q’s we should ponder…

How will the use of the signing key be authorized?

How will staff insure the foundation assets and reputation (what process and technical tools should they have)? E.g. Dual control, AWS HSM key vault, … other…

Can the runtime system (not just the CD pipeline) verify the image signatures and refuse to start servers with invalid signatures? How might this facility be defeated? Can in-built Linux (or Docker) code security features be leveraged (SELinux, https://wiki.archlinux.org/title/SELinux)?

How should key rotation and signature deprecation or revocation be handled?

— Miller

On Feb 2, 2022, at 4:35 AM, Pedro Barreto @.***> wrote:

Ok, the initial link I sent seems to solve this nicely, only caveat is that we must create a OpenPGP key. Creating it is easy enough, not sure about how we store it. Once we have it, we will be able to cryptographically sign our releases so they can be verified by the downloader.

— Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android. You are receiving this because you are subscribed to this thread.

bukasaaime commented 2 years ago

We can make use of public Key Servers (keyserver) such as as (1) keyserver.ubuntu.com (2) pgp.mit.edu or (2) keyserver.pgp.com in conjunction with Git CLI commands to generate pgp keys and push them to these servers for Public Keys distribution.

The default pgp Key Server hkp://hkps.pool.sks-keyservers.net does not seem to handle the load therefore not recommended.

We can then avail a link for the code verifier(s) to receive the Public Key to be used to verify the Mojaloop code signature from the Key Servers. The Git CLI commands can be used for that as well.

Seen that the Mojaloop code is to be customized or forked by the implementer, the original code signature will cease to work since the code would have been modified at that point.

millerabel commented 2 years ago

This is a good plan. Please have a look into the license and terms for the MIT public server.

An extra check can be made to flag changes in the public key fingerprint (by holding the last known good value locally) before applying the public key in verification. If the public key changes, we ask “why?” Overcoming that flag should require a manual op by key custodian auth in the CI/CD pipeline.

As for source distribution to implementers, I agree the commit hashes should be the limit of our distribution promise. Once the open source code “leaves the building,” we can’t guarantee anything further. Anyone hosting locally would have to establish their own veracity over what the they modify and deploy.

Which of our images can be used “without modification” and without rebuilding or re-signing? Or must everything be rebuilt by each implementer?

On Feb 4, 2022, at 12:00 AM, bukasaaime @.***> wrote:

We can make use of public Key Servers (keyserver) such as as (1) keyserver.ubuntu.com (2) pgp.mit.edu or (2) keyserver.pgp.com in conjunction with Git CLI commands to generate pgp keys and push them to these servers for Public Keys distribution.

The default pgp Key Server hkp://hkps.pool.sks-keyservers.net does not seem to handle the load therefore not recommended.

We can then avail a link for the code verifier(s) to receive the Public Key to be used to verify the Mojaloop code signature from the Key Servers. The Git CLI commands can be used for that as well.

Seen that the Mojaloop code is to be customized or forked by the implementer, the original code signature will cease to work since the code would have been modified at that point.

— Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android. You are receiving this because you commented.

godfreykutumela commented 2 years ago

Hi @millerabel @bukasaaime @pedrosousabarreto

I completely agree that if we go the key singing route then we will need to have end to end key management process in place for this to securely manage the keys from cradle to craves.

The second point, can we consider only go as far as verifying the downloaded package to ensure nothing was tampered with during the downloading process and also if the implementers that do not change the package and happens to experience problems or find any vulnerabilities then we can this signing to verify that the package comes from us directly.

To enforce other restrictions suggested above like verifying the image signatures and refusing to start servers with invalid signatures might be a big ask with fewer benefits in my view and it might also generate a lot of support tickets back to the core team that is already stretched too thin!

millerabel commented 2 years ago

Yes, these are fair points. Directionally, we ought to aim for secure validation of images, in a production environment. Our doing that in our build and test environment allows us to iterate on what’s required for others to do this in their own environments. But that may be too aspirational for now.

Do others agree we can aim lower for now, verifying signatures during download process, but postponing more complete integration of image verification into runtime?

We will still need some key management process and facilities, both for signing and verification, but it should be doable within the AWS and Azure build and test environments.

Can we leverage GitHub signing support (commit signing and tag signing) together with Docker trust?

Since we release open source, the built images are an artifact. It’s important, for sure, that images we release represent the tagged released code, and that they are runnable. But ultimately it’s the collection of tagged source and dependencies that represents the “release.” Yes?

On Feb 7, 2022, at 4:36 AM, Godfrey Kutumela @.***> wrote:

Hi @millerabel @bukasaaime @pedrosousabarreto

I completely agree that if we go the key singing route then we will need to have end to end key management process in place for this to securely manage the keys from cradle to craves.

The second point, can we consider only go as far as verifying the downloaded package to ensure nothing was tampered with during the downloading process and also if the implementers that do not change the package and happens to experience problems or find any vulnerabilities then we can this signing to verify that the package comes from us directly.

To enforce other restrictions suggested above like verifying the image signatures and refusing to start servers with invalid signatures might be a big ask with fewer benefits in my view and it might also generate a lot of support tickets back to the core team that is already stretched too thin!

— @pedrosousabarreto and @lewisdaly tagging you to review Miller's proposition and provide your input. We will start working on solution options for this leveraging Github for code signing and AWS/Azure for key management services. I am going to setup a discussion meeting with Head of DevSecOps at Github to get his view and input on this.

bukasaaime commented 2 years ago

I have reviewed Miller's proposals. Yes, we can start with

(1) a simple process of signing a build or bundle, providing a corresponding hash value or checksum and a signature.

In order to guard against corrupted downloads/installations, the developer will then need to verify the signature of the release bundles against the public KEYS that Mojaloop would have provided trough a third party.

(2) As a second phase, we can implement a continuous code signing through the CICD pipeline process that will include Key Management, continuous code signing at all stages of de deployment through the pipeline and audit trails.

If this approach is acceptable we put a proposal forward and start this body of work.

bukasaaime commented 2 years ago

Good day Miller and Godfrey,

Please see below a summary of the requirements as per Miller's remarks for Code signing:

A key Management process for the Code Signing Key.
Leverage on existing Github, AWS or Azure.
Release image should be constituted by tagged build and dependencies.
Runtime verification of Mojaloop code when installed (long term).

Can we confirm ? Can we start addressing the above ?

Thanks and regards,

Aime Bukasa

millerabel commented 2 years ago

This looks right to me, thank you Aime for distilling the requirements. I will defer to Godfrey and DevOps folks on next steps. Will you folks confer?

On Mar 6, 2022, at 9:21 AM, bukasaaime @.***> wrote:

Good day Miller and Godfrey,

Please see below a summary of the requirements as per Miller's remarks for Code signing:

A key Management process for the Code Signing Key. Leverage on existing Github, AWS or Azure. Release image should be constituted by tagged build and dependencies. Runtime verification of Mojaloop code when installed (long term). Can we confirm ? Can we start addressing the above ?

Thanks and regards,

Aime Bukasa

— Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android. You are receiving this because you were mentioned.

bukasaaime commented 2 years ago

Thank you Miller,

Regards,

Aime Bukasa

On 2022-03-06 20:50, Miller Abel wrote:

This looks right to me, thank you Aime for distilling the requirements. I will defer to Godfrey and DevOps folks on next steps. Will you folks confer?

On Mar 6, 2022, at 9:21 AM, bukasaaime @.***> wrote:

Good day Miller and Godfrey,

Please see below a summary of the requirements as per Miller's remarks for Code signing:

A key Management process for the Code Signing Key. Leverage on existing Github, AWS or Azure. Release image should be constituted by tagged build and dependencies. Runtime verification of Mojaloop code when installed (long term). Can we confirm ? Can we start addressing the above ?

Thanks and regards,

Aime Bukasa

-- Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android. You are receiving this because you were mentioned.

-- Reply to this email directly, view it on GitHub [1], or unsubscribe [2]. Triage notifications on the go with GitHub Mobile for iOS [3] or Android [4]. You are receiving this because you were mentioned.Message ID: @.***>

-- Aime Bukasa

Chief IT Architect, Friedcorp 89 cc |

Office: +27 12 654 0270 |

Cel: +27 82 625 6220 | +27 78 618 0154 | +243 810 715 532

www.friedcorp89cc.com [5] |

Links:

[1] https://github.com/mojaloop/project/issues/2634#issuecomment-1060017854 [2] https://github.com/notifications/unsubscribe-auth/AH5DXL7GLIFPN5PIMP35HK3U6T46BANCNFSM5KNO56WQ [3] https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 [4] https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub [5] http://www.friedcorp89cc.com

godfreykutumela commented 2 years ago

We recommend a phased approach in implementing the above code signing requirements

Phase I – Use Github code signing and Key Management process .

Add the process in Mojaloop official documentation.
Identify official repository.
Document the commit process.
Commits/images need to be signed.
Singing commits – identify / document the process if one exist. If not, create the process.
Github uses PGP (Pretty Good Private).
Pgp key management process :
• Pgp key definition (size, expiry, passphrase) • Pgp Key ceremony.
• Identify pgp Key Custodian (identity).
• Pgp Key generation.
• Password.
• Public key distribution into the public Key server (MIT) • Pgp key revocation. • Pgp key renewal • Pgp key protection. Backup and restore. • Document CL commands
Configure pgp key in Github.
Document the code verification process in Github for developer. • Pgp installation • Search for public key. • Importing the public key. • Verifying the key. • Key ring • Document CL commands

Phase II – CICD (TBD) – AWS / Azure / GCP

Phase III – Runtime verification (TBD).

@lewisdaly @elnyry-sam-k @pedrosousabarreto @mdebarros any review feedback on this approach? if not let us plan to test this on one or two repos @bukasaaime

bukasaaime commented 2 years ago

(1) Who would normally commit the official Mojaloop repository, different commits and tags. (2) from what machine are these done. This will enable us to identify the keys custodian.

bukasaaime commented 2 years ago

This will enable us to start documenting the key management process.

godfreykutumela commented 2 years ago

Assume all community developers will be able to commit on an adhoc basis.

elnyry-sam-k commented 2 years ago

Thanks @bukasaaime and @godfreykutumela; On the question of the commits - Yes, any contributor can merge a Pull Request once it is approved; but direct committing is not allowed (though it may not have been enforced in branch rules on all the repos). Most repos have codeowners defined who need to approve the PR only after which the PRs can be merged..

here's the forking workflow used: https://docs.mojaloop.io/community/standards/creating-new-features.html

bukasaaime commented 2 years ago

Thank you @elnyry-sam-k. Once all the Pull requests are merged, is that when the helm repo release is packaged into a .tgz or .zip file for distribution to users ? If yes, that is the file that we will need to sign and I will need to review and document the process as far as GPG Key Management process is concerned.

bukasaaime commented 2 years ago

Please see Code signing proposal for your comment. The Solution is based on Helm Provenance and Integrity.

Helm has provenance tools which help chart users verify the integrity and origin of a package. Using industry-standard tools based on PKI, GnuPG, and well-respected package managers, Helm can generate and verify signature files.

Integrity is established by comparing a chart to a provenance record. Provenance records are stored in provenance files, which are stored alongside a packaged chart.

Kebase.io / GnuGPG will be used for GPG lifecycle management (instead of MIT public key server).

Regards,

bukasaaime commented 2 years ago

keybase.io

godfreykutumela commented 2 years ago

Hi @pedrosousabarreto @mdebarros @elnyry-sam-k @vijayg10 @lewisdaly @MichaelJBRichards Please review and comment on the above-proposed architecture for code signing so that we can move on to test this on one or two old helm charts before enforcing this in the latest charts. @millerabel we decided to first apply the code signing solution on the helm charts but it could be done at the repo level which has more work that requires changes to the current code delivery flows hence we are avoiding that route!

millerabel commented 2 years ago

This looks like an excellent approach. Thanks Aime and Godfrey.

I’m curious why you chose not to use the MIT PK repo. It would be good to document our choice to use Kebase.io so in future we can recall how we got here.

What happens if the .prov file is missing or corrupt? or if the signature does not match? Is the system partially imaged (and so in an inconsistent state requiring manual correction) or is the image change rolled back?

Can this solution ensure that all Helm charts are consistent (by some kind of cross-chart provenance check)? Similar to how yarn.lock or shrinkwrap works for NodeJS.

On Mar 24, 2022, at 11:18 AM, bukasaaime @.***> wrote:

Please see Code signing proposal for your comment. The Solution is based on Helm Provenance and Integrity.

Helm has provenance tools which help chart users verify the integrity and origin of a package. Using industry-standard tools based on PKI, GnuPG, and well-respected package managers, Helm can generate and verify signature files.

Integrity is established by comparing a chart to a provenance record. Provenance records are stored in provenance files, which are stored alongside a packaged chart.

Kebase.io / GnuGPG will be used for GPG lifecycle management (instead of MIT public key server).

Regards,

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.

bukasaaime commented 2 years ago

The use of Keybase.io is optional but recommended by the Helm package management documentation. We can also use the MIT PGP Public Key Server, but it is a basic system that only handles Public Keys (import and search).

Keybase.io is a system (CLI, web and desktop) with a lot more features. Handles custodians (signer) information for instance, establishes a Chain of trust by establishing the authority of a signer i.e., if the verifier trusts the signer of the chart, the verifier can trust the public key of the said signer.

Keybase.io manages the GPG keypair (both private and public). Even if the signer loses the machine where the GPG keyring or key store) , where the signer generates or imports GPG keys, generates and verify signatures, the signer can recreate the environment on another machine.

It is to note that the Helm itself is not part of the chain of trust.

Chart repositories serve as a centralized collection of Helm charts. Chart repositories must make it possible to serve provenance files over HTTP via a specific request and must make them available at the same URI path as the chart.

The provenance file is generated at the same time as the chart is packaged and signed, by the same CLI command, in the same process.

During chart installation (and provenance verification), if the provenance file does not match the packaged chart, the installation will fail. This means that the package has been tempered with after it was signed, or the provenance file is not the correct one (mismatch or corrupt). This is in fact how packaged chart integrity is proven.

bukasaaime commented 2 years ago

From all the discussions, I have update the code signing process. We need to implement and test as a first phase which includes the following:

sign packed Mojaloop chart with GPG keys (MIT or Keybase.io)
use provenance to verify packaged chart and inject custom values during installation .

Thanks, and regards

bukasaaime commented 2 years ago

Adding @mdebarros comments.

godfreykutumela commented 1 year ago

https://docs.docker.com/engine/security/trust/ thanks for the link @pedrosousabarreto. @bukasaaime lets put a pause on the current design using helm and focus on signing imaging at the point of production!

millerabel commented 1 year ago

Great explanation, thank you @bukasaaime. We should add this to the rationale section of the documentation for code signing.

On Mar 25, 2022, at 8:19 AM, bukasaaime @.***> wrote: