[RFC] - Signed Package Validation

alikon commented 6 years ago

Incipit

from #19748 , #17632, #17619

Package Validation

17632 and associated PRs somewhat address this, but it also somewhat falls in the "too little too late" category. If someone gains access to publish fake releases, they would publish the right checksums for their fake packages and Joomla would take it without issue. So you need signed releases as another step on top of (or instead of) checksum validation. https://core.trac.wordpress.org/ticket/39309 proposed that for WordPress, but they clearly aren't interested in it. I link it here though >because the patches provided by Scott in that item are not too far off from what we would have to add >to our core code (and in effect how you do it is all spelled out in the patch, you just have to convert it to >fit the Joomla API). Truth be told that is probably a bigger change to release management workflow >than end user workflow, but it is still an item to address.

Sign the package

the developer/mantainer sign the package with something like:

// Generate keys
$kp = ParagonIE_Sodium_Compat::crypto_sign_keypair(); 
// secret
$sk = ParagonIE_Sodium_Compat::crypto_sign_secretkey($kp); 
// public
$pk = ParagonIE_Sodium_Compat::crypto_sign_publickey($kp); 
// Sign the package
$message =file_get_contents( 'com_patchtester.tar.bz2' );
$signature = ParagonIE_Sodium_Compat::crypto_sign_detached($message, $sk); 
// Verify
if (ParagonIE_Sodium_Compat::crypto_sign_verify_detached($signature, $message, $pk))
{ 
    echo 'OK', PHP_EOL; 
    echo 'SecretKey:' . bin2hex($sk), PHP_EOL;
    echo 'PublicKey:' . bin2hex($pk) , PHP_EOL;
    echo 'Signature:' . bin2hex($signature), PHP_EOL; 
} 
else 
{ 
   throw new Exception('Invalid signature'); 
}

TAG the manifest package

add two new tags to the package manifest

publickey value from the previous step
signature value from the previous step

screenshot from 2018-02-27 07-39-01

Check the package signature on update

from #17619 https://github.com/joomla/joomla-cms/pull/17619/files#diff-da01c5bbbd8dc2c831d18d095a3e2f16R371 change the function bodyisChecksumValid($packagefile, $updateServerManifest)as:

$update = new \JUpdate;
$update->loadFromXml($updateServerManifest);

if (!$update->get('publickey', false)->_data)
{
    return self::HASH_NOT_PROVIDED;
}

$pk= trim($update->get('publickey', false)->_data);
$publicKey = \ParagonIE_Sodium_Compat::hex2bin($pk);

if (!$update->get('signature', false)->_data)
{
    return self::HASH_NOT_PROVIDED;
}

$sign = trim($update->get('signature', false)->_data);
$signature = \ParagonIE_Sodium_Compat::hex2bin($sign);
$packagefile = file_get_contents($packagefile);

if (\ParagonIE_Sodium_Compat::crypto_sign_verify_detached($signature, $packagefile, $publicKey))
{ 
    return self::HASH_VALIDATED;
}
return self::HASH_NOT_VALIDATED;

alikon commented 6 years ago

@mbabker, @zero-24 any feedback?

zero-24 commented 6 years ago

sorry. I have looked into that and did wrote some dummy code to test. But I did not found enough time yet to test it. :(

https://github.com/joomla/joomla-cms/compare/staging...zero-24:signedUpdates?expand=1

As it looks like the pk / sk generation took longer than I expected.

alikon commented 6 years ago

As it looks like the pk / sk generation took longer than I expected.

luckily we should run that code only once for each release :smiley:

PhilETaylor commented 6 years ago

Here we go again..... //cc @nikosdion

nikosdion commented 6 years ago

Last year I took three months of my time to do serious research on the subject, think about the pitfalls and come up with a solution. I did a presentation at J and Beyond about securing updates. I recommend that y'all watch it.

An obvious problem I see in your update XML file is that it's pointless as it's dictating which public key to use. We are trying to protect our users against someone spoofing the XML update source for example doing one of the following:

stealing the Joomla CDN credentials
subverting the Joomla CDN credentials, e.g. convincing the CDN company they're talking to a legit representative of OSM
launching a man in the middle attack (very easy if you're a state actor or a corporate actor within the boundaries of your network, more difficult but not impossible otherwise)
generally speaking, the update XML delivered to the user is NOT the one the Joomla project / OSM wanted to deliver
generally speaking, the Joomla! Update component is pointed to an unauthorized ZIP package

The only way to protect the users in this case is to check a signature. The signature must be calculated against a known public key with an expiration date. If you include the signature date in the signature itself you can fix the problem of expiring certificates. This is how e.g. Authenticode works (and why you can still install old software with expired, but not revoked, certificates).

Please note that the signature MUST be calculated by the update client AND included in the package itself. Anything else is pointless. For example, this naive XML I see on this RFC is laughably insecure. If I am running a MITM I will point your users to my public key and the signature calculated with my key for the illegitimate, malicious package. You're pwned.

Here's the thing about cryptography and security. When you do it right it does offer a benefit to the user. When you implement it shoddily it's worse than not having it at all: a false sense of security leads to complacency and that's worse than no security at all!

Please do watch my presentation. While I was writing it I had to reevaluate all my preconceptions and, let me tell you, some of that was utterly wrong. I've done the hard work, please watch the result of that hard work and hire a cryptographer before writing a single line of code. I don't even trust myself enough to write such a feature. I don't trust any of you either. Don't take offense in that, none of us is qualified to write cryptography code unsupervised.

mbabker commented 6 years ago

I've done the hard work, please watch the result of that hard work and hire a cryptographer before writing a single line of code. I don't even trust myself enough to write such a feature. I don't trust any of you either. Don't take offense in that, none of us is qualified to write cryptography code unsupervised.

I'm not qualified either. TBH if I ever got around to writing a patch for this it would be very highly based on the one @paragonie-scott proposed for WordPress (https://core.trac.wordpress.org/attachment/ticket/39309/39309_-_Updated_Patch_3.patch is the latest version of that effort). So it's a matter of understanding what exactly is going on there to be able to adapt it.

nikosdion commented 6 years ago

This wouldn't quite work since it's based on the assumption that your content is delivered through WordPress' update infrastructure. They send the signature as an HTTP header.

What I proposed is more versatile. You can attach any signature you want to the file itself by taking advantage of the way the ZIP format works, i.e. ZIP comments are appended to the end of the archive file.

The takeaway from Scott's code should be a. the signature methodology (Ed25519) and b. the trusted public keys are delivered with the software, not separate from it.

I am not qualified to comment on the best signature method. I trust Scott :)

The second takeaway is the most important. Don't trust the wire to deliver your public key. Including the trusted public keys in the software itself creates a chain of custody: only a signed update or code you trust (and run on your server) can change the list of keys.

BTW, this is pretty much what I had presented last year minus the actual signature methodology (hey, I wrote that presentation 8 months before the WP patch and still got very close!).

alikon commented 6 years ago

I know that I'm not qualified and I not claim to be, and i'm aware that i'm putting "all the eggs in the same xml basket", but was done for the purpose to easily the start of a discussion on how to quick deliver an easy solution to raise a little bit the security level of joomla's updates

p.s. i swear, i'll watch your presentation again....(iirc i was there when you did it :flushed:)

alikon commented 6 years ago

after watching that presentation https://www.youtube.com/watch?v=NltBeMG22WI&t=5s

it's quite clear that we need to sign the update manifest xml file too in order to simplify things let's concentrate for now on this issue

how to do this ?

we can use a trusted identification using a certificate authority (CA)

or

we can do something like Linux distro like (ubuntu/debian) are doing it https://help.ubuntu.com/community/SecureApt

i would follow the linux approach

Basic Concepts

A hash algorithm is a function which can be run on a file in order to produce a fixed length representation of that file. This fixed length representation is called a digest. It is impossible to reproduce a file from its digest.

Public key cryptography is based on pairs of keys, a public key and a private key. The public key is given out to the world; the private key must be kept secret. We use a private key to sign a file. When a private key is used to sign a file, then anyone who has the public key can check that the file was signed by that key.

A digital signature is created by creating a hash of the file and then encrypting the digest with the private key

Steps

key pairs generation
public key acquisition/distribution
digital signature
verification process

1) key pairs generation

// Generate keys
$kp = ParagonIE_Sodium_Compat::crypto_sign_keypair(); 
// secret
$sk = ParagonIE_Sodium_Compat::crypto_sign_secretkey($kp); 
// public
$pk = ParagonIE_Sodium_Compat::crypto_sign_publickey($kp);

2) public key acquisition/distribution

the public key is acquired when the extension is installed the public key is stored in a new field of the #__extension table

3) digital signature

create an hash/digest (with sha384 for example) of the update manifest file
sign with the private key the hash/digest
add the tag signature to the update manifest file with the signed digest
publish the update manifest file to the update server

4) verification process

read the signature tag from the update manifest file
remove the signature tag from the update manifest
calculate the digest from modified update manifest
get the extension public key from the #__extension table
verify the the digest with the public key

nikosdion commented 6 years ago

This is better but not quite there. I have a few objections regarding operational integrity, the choice of hash and signing algorithms and the use of the database as the all important Source of Truth regarding certificates.

It's going to be a long reply. Grab some coffee.

Do not create key pairs using Sodium on a networked computer. Instead, use GPG on an airgapped comptuer. Any computer that is connected to the network should be considered compromised. The key generation should be conducted in person, e.g. at JaB18 or whenever the Joomla officials meet physically.

Important: whenever I am talking about "keys" I actually mean "certificates". A certificate encapsulates the key in an encrypted and/or signed envelope. It helps us make sure that it wasn't an arbitrary third party who infested our user's site with a rogue key. It also makes revocation possible, even though I don't touch the subject below. I also want to point out that we should absolutely ask a cryptographer before implementing our own public key infrastructure with reckless abandon. Cryptography is notoriously prone to esoteric oversights the average senior developer with thirty years of experience might miss. That's where cryptographers come into the picture. Did I mention we need cryptographers to do this? Yup, we definitely need cryptographers to make this happen in a sensible manner.

The SECRET keys for the primary key pair never, ever leaves the airgapped computer. For reasons of operational integrity one or more password protected backups can be made and distributed in USB keys sealed in tamper evident envelopes to responsible people in the Joomla organization. A completely different set of responsible people will be given tamper evident envelopes with the password. The persons (always three, one operator and two observers) who took part in the key generation are ineligible for having custody of either part of the key. After moving the secret keys of the primary key pair to backup USB sticks and having them sealed they are deleted from the airgapped computer. Then the computer itself is placed in a tamper evident envelope, then inside a locked case. Thus, nobody has the ability to create signing keys.

Do not use the primary key pair to sign releases! This makes revocation and chain of trust nigh impossible. Instead, use the primary pair as a certificate authority to create and sign intermediate certificates. Each intermediate certificate should only be used for a brief period of time (18 months or one release cycle such as 3.9, whichever comes first).

Intermediate (signing) keys are obviously generated only in person. One operator, one secret key holder and one password holder -in the presence of the rest of the Joomla leadership as witnesses- operate the airgapped computer to restore the secret key, create a new set of intermediate keys, deliver them to the release leader and a backup member of the team (more on that later) and then remove them from the airgapped computer. The process of creating key backups and password slips is then repeated.

The PUBLIC keys for the primary key pair (certification authority) and the intermediate (signing certificate) will be made publicly available.

The SECRET key for the intermediate authority will be stored in a PIN-protected smartcard such as a YubiKey 4 at the very minimum. Better yet, use a smartcard reader with a hardware PIN entry keyboard. They key is transferred to the smartcard by an operator and the PIN is set by the release leader without the operator looking. Two witnesses will confirm the operation. Same thing with a backup release leader (to make sure we don't have a bus factor of one).

In no circumstances are the private keys stored in plaintext or transmitted over any network, including a direct Ethernet connection. The airgapped computer never connects to any network whatsoever.

The release leader needs to use GPG with the smartcard extension to sign both the release package and the XML update file.

The big question is how do you sign something and attach the signature to it. Glad you asked, it was in the presentation :)

Signing the XML Update is trickier. We need to read the XML update file and normalize it. This is fairly easy using PHP's SimpleXML to read the file, extract the update record we are signing, remove the signature information and then export it as an XML document string. We hash then sign this string. The signature information (hash, signature and the ID of the public key we used) is appended to the update entry and the whole lot is uploaded to the Joomla CDN.

Verifying the XML update integrity is a similar process. Get the XML update file from the CDN. Extract the update record. Parse it with SimpleXML, remove the signature information, export as XML document then hash it. Compare the hash with the one on the signature information block. Verify the signature against the public key. If we don't have the public key in our code we download that certificate from the CDN and we check its validity (signature against the main public key we distribute with Joomla and its expiration date which must be before the release date of the update, as recorded in the update information). If anything fails the update is considered compromised and we don't use it.

Signing the ZIP file is straightforward. Create the ZIP file normally and make sure it has no ZIP comment. Then you need to generate a hash of the file and sign it. This is the part I cannot help (I am not a cryptographer): which signature method to use. In any case, we can then create an XML document with the signature method, the hash of the GPG key used and the actual signature. You can then append that XML document as a ZIP file comment.

Verifying the signature is also straightforward. After downloading the ZIP file read the ZIP file comment and parse the XML document. If parsing fails the signature is invalid and the package is rejected. Check if our code (NOT the database!!!) has a copy of the signing key defined in the XML manifest. If not, download the public signing key by that signature from the Joomla! CDN. Then check if the public key certificate we downloaded is signed by the Joomla! certification authority (whose public key we distribute with Joomla! itself) and if the expiration date is less than the release date defined in the XML update stream. If any of these conditions fail the package signature is invalid. Then calculate the hash of the ZIP file without the ZIP comment. Check that it matches the one on the XML manifest of the package and that the signature on the manifest is valid. Again, any condition failing means the package is tampered with.

KEY POINTS

We only distribute public keys to users
Public keys must be present IN THE CODE (not the database, the database CAN be tampered very easily!!!) and MUST be validated against the main (root CA) public key of Joomla before being trusted.
Secret keys are never handled unencrypted except for a brief period of time on an airgapped computer which is not used for any other reason.
Secret keys for signing are always stored on smart cards and protected by a PIN, i.e. they never exist in any form which can be used by unauthorized personel.
Physical integrity of the signing process is of paramount importance. After all, we are responsible for millions of sites.
We need a cryptographer to weigh in on which signature method we should use. This is critically important and none of us is qualified to answer this question in a way which won't jeopardize the whole venture.
Never trust a networked computer.
Never trust people.

How should you proceed?

You can make POC (proof of concept) command line scripts for signing releases with GPG. You can also make POC PHP code for verifying the signatures.

Assuming the POC yields positive results (it should, Git is using a similar process to sign commits forever) you can proceed with the actual implementation of protocols and the purchase of the necessary hardware. The hardware costs less than $1000 which is peanuts compare to what the average Code Sprint costs.

Do you need help?

I can help with the technical aspects of encryption but I cannot help you with the all-important selection of a signing algorithm. We absolutely need to talk to a cryptographer. Cryptography seems deceptively simple but it's seriously complicated. The smallest oversight can lead to a massive security issue down the line. I have been there and I have positively screwed it up so now I can warn you :)

paragonie-scott commented 6 years ago

One nit: Don't use GPG on an airgapped computer, use Minisign on an airgapped computer.

nikosdion commented 6 years ago

Thank you, @paragonie-scott !

In this case we should clarify that we would need a total of THREE airgapped computers (which are never, ever connected to the Internet): one for generating the signatures and two given to the release leader and the backup release leader.

After building the release on the regular computer, the new release's files must be placed on removable storage and transferred to the airgapped computer for signing. The signed information is placed back to removable storage and then used on the regular computer for actually releasing them to the world. The downside is that the total cost for the project increases a bit. Then again, we can use dirt cheap older models since we don't need any horsepower from these machines, keeping the total cost manageable (also, these machines don't need to be replaced unless they are completely dead or stolen).

Regarding removable storage, I don't trust USB keys. I've seen and read things. Am I too paranoid for proposing that we burn CD-ROMs for transferring data between the airgapped computer and the regular computer? Also, am I completely insane for also suggesting that the airgapped computer is operated in a room without any powered up electronics?

And a final question. Should we still use GPG on the airgapped computer to make the certificates? If not, what is our alternative?

ggppdk commented 6 years ago

So a part of this work or all of it, will be usable / useful to 3rd party extensions too ?

PhilETaylor commented 6 years ago

So a part of this work or all of it, will be usable / useful to 3rd party extensions too ?

History/Evidence shows we can't trust 3rd party extension developers to even protect against the most basic SQL Injection of id params ... I'm pretty sure that cryptography is going to be well beyond (the majority of) them too. :-(

ggppdk commented 6 years ago

Regardless of how fast the adaptation / support of signed packages will happen by 3rd party developers

Joomla Installer should support using public keys (signed by known authority) from already installed extensions, and require a properly signed package during extension update if the public key exists

and later / much later ? , when the adaptation gets high enough, then require that all extensions have signed packages

I mean when this work is done for Joomla update packages, then there is no reason ? not to support this in Joomla installer for 3rd party extensions

nikosdion commented 6 years ago

That's pretty much the gist of my presentation. I also mentioned what is the caveat in this case: certificate signing. Joomla would need to act as a Certification Authority for third party developers in the same way, for example, Apple does. There's an opportunity for the project to finance JED development from the proceeds of certificate signing, something that I also mentioned in my presentation, something which is beneficial for all parties involved: JED / OSM / Joomla, developers, users.

But as Phil said, I'm not very optimistic that we'll live to see that day. This is definitely one of the times I wholeheartedly wish I am dead wrong.

brianteeman commented 6 years ago

As soon as the JED adds a flag to say an extension is signed then it will be rapidly adopted. We saw this when the JED added the uses joomla updater flag

PhilETaylor commented 6 years ago

Dare I say it, but the JED is only one source for extensions... :-)

There are still a LOT of extensions who don't implement the Joomla Update API, and a LOT of those are even listed in the JED still. Would be interesting (but out of scope of this issue) to know the percentage of JED listed extensions that do implement the Joomla Updates API.

mbabker commented 6 years ago

I mean when this work is done for Joomla update packages, then there is no reason ? not to support this in Joomla installer for 3rd party extensions

Because core searches for updates in the same way that extensions do, any architectural work done for core updates inherently affects extension updates (and vice-versa). So similar to the checksum code added to 3.9, it would be available to extension developers but requires them to make the required updates in their code/workflow.

alikon commented 6 years ago

i've made a quite simply poc for the signing extension update manifest part an the relative poc for the verification #20073 i've used ParagonIE_Sodium_Compat

joomla-cms-bot commented 6 years ago

Set to "closed" on behalf of @franz-wohlkoenig by The JTracker Application at issues.joomla.org/joomla-cms/19792

ghost commented 6 years ago

closed as having Pull Request #20073

alikon commented 6 years ago

no please don't close this one #20073 it's only a little piece of the picture if i'm doing it in the right way

joomla-cms-bot commented 6 years ago

Set to "open" on behalf of @franz-wohlkoenig by The JTracker Application at issues.joomla.org/joomla-cms/19792

ghost commented 6 years ago

Reopened as stated above.

_{This comment was created with the J!Tracker Application at issues.joomla.org/tracker/joomla-cms/19792.}

alikon commented 4 years ago

after 2 years of nothing let me sadly close

joomla / joomla-cms