Open binwiederhier opened 2 years ago
I looked at this a little today, and sadly had to rule out gpg
, because the library for Go is deprecated, and I don't want to build on deprecated technology. I also looked at age
a while ago and even opened an issue here (https://github.com/FiloSottile/age/discussions/404) to ask for support in other languages; and sadly there is not enough support.
I also played with openssl
to see if I can encrypt and protect the integrity (~ AEAD-style), but I wasn't successful.
I will try and play some more, but I believe I'll have to implement my own format. Something similar to what I have done here (https://syncany.readthedocs.io/en/latest/security.html#encrypting-new-files), though much simpler hopefully.
Here's the basic message structure: https://github.com/binwiederhier/ntfy/pull/354. This implements the exact same thing that Pushbullet does (https://docs.pushbullet.com/#encryption) and is even compatible with it. This was super easy. The trickier part is figuring out the UX of this. I'll have to think about that.
This can encrypt a message with a key, but a key is not a password. The password we'll still have to derive using pbkdf2.
The question is now:
Relatively easy would be:
Even though I have not updated this ticket, I have been working on it in the #354 branch. I've explored JWE as a solution for the crypto format, and implemented PoCs in PHP and Python and Go.
After realizing that that won't work for attachments, I have switched gears after a discussion with a co-worker and with @wunter8.
"ntfy2586v1" || tag (128 bits) || iv (96 bits) || ciphertext [aes-256-gcm]
PUT /mytopic HTTP/1.1
Encrypted: yes
<encrypted binary data, ntfy format>
PUT /mytopic HTTP/1.1
Encrypted: yes
Content-Type: multipart/form-data; boundary=--ntfy-boundaryXyASDsdA
----ntfy-boundaryXyASDsdA
Content-Disposition: form-data; name="message"
Content-Length: 1234
<encrypted binary data, ntfy format>
----ntfy-boundaryXyASDsdA
Content-Disposition: form-data; name="attachment"
Content-Length: 42343242
<encrypted binary data, ntfy format>
----ntfy-boundaryXyASDsdA--
Password:
KDF:
It might also be worth adding 'kid' to the JWEs so that can give users a chance to change or rotate their passwords. Changing the passwords has some choices because of historically encrypted messages. Since the user holds the password it would be up to them to have to re-encrypt every message.
Using a key-encryption-key (KEK) to encrypt the JWEs would provide better security. The KEK would be wrapped by the password and the client would store the wrapped KEK somewhere. It could be password wrapped, it could be key-vaulted, who knows. How to transport that KEK between endpoints? iOS has cloud-sync as an example. Or if it was wrapped by the password (pbkdf2) then could nfty itself do it? Rotation the password on the KEK is easier as there's just one. Still hard to rotate the KEK though.
I'll think some more on ergonomics of this.
I am still working on this. It's going slow (mostly due to personal commitments), but I have still managed to come up with a design that I think I like. So here it goes:
sha256(topic)
as a salt (e.g. sha256(mytopic)
){"alg":"dir","enc":A256GCM"}
) is supportedRequest (HTTP)
POST /sometopic HTTP/1.1
Title: some title
some message
Request (via curl)
curl -d "some message" -H "Title: some title" ntfy.sh/sometopic
Response
{
"id":"eWa5epsDHkQ4",
"time":1657759261,
"event":"message",
"topic":"sometopic",
"title":"some title",
"message":"some message"
}
Request (HTTP)
POST / HTTP/1.1
{
"topic":"sometopic",
"title":"some title",
"message":"some message"
}
Request (via curl)
curl ntfy.sh -d '{
"topic":"sometopic",
"title":"some title",
"message":"some message"
}'
Response
{
"id":"eWa5epsDHkQ4",
"time":1657759261,
"event":"message",
"topic":"sometopic",
"title":"some title",
"message":"some message"
}
Request (HTTP)
POST /sometopic HTTP/1.1
Title: some title
Message: some message
<attachment data>
Request (via curl)
curl -T attachment.jpg -H "Title: some title" -H "Message: some message" ntfy.sh/sometopic
Response
{
"id":"eWa5epsDHkQ4",
"time":1657759261,
"event":"message",
"topic":"sometopic",
"title":"some title",
"message":"some message",
"attachment": {
"name": "attachment.jpg",
"type": "image/jpeg",
"size": 715814,
"expires": 1657775583,
"url": "https://ntfy.sh/file/tmhElNL0MiKM.jpg"
}
}
Request (HTTP)
POST / HTTP/1.1
Content-Type: multipart/form-data; boundary=--ntfy-boundaryXyASDsdA
----ntfy-boundaryXyASDsdA
Content-Disposition: form-data; name="message"
{
"topic":"sometopic",
"title":"some title",
"message":"some message"
}
----ntfy-boundaryXyASDsdA
Content-Disposition: form-data; name="attachment"
<attachment data>
----ntfy-boundaryXyASDsdA--
Request (via curl)
curl ntfy.sh \
-F attachment=@attachment.jpg \
-F message='{
"topic":"sometopic",
"title":"some title",
"message":"some message"
}'
Response
{
"id":"eWa5epsDHkQ4",
"time":1657759261,
"event":"message",
"topic":"sometopic",
"title":"some title",
"message":"some message",
"attachment": {
"name": "attachment.jpg",
"type": "image/jpeg",
"size": 715814,
"expires": 1657775583,
"url": "https://ntfy.sh/file/tmhElNL0MiKM.jpg"
}
}
Request (HTTP)
POST /sometopic HTTP/1.1
Encoding: jwe
eyJhbGciOiJka.............-nH7ya1VQ_Y6ebT1w.2eyLaTUfc_rpKaZr4-5I1Q
Request (via curl)
curl ntfy.sh/sometopic \
-H "Encoding: jwe" \
-d "eyJhbGciOiJka.............-nH7ya1VQ_Y6ebT1w.2eyLaTUfc_rpKaZr4-5I1Q"
Response
{
"id":"eWa5epsDHkQ4",
"time":1657759261,
"event":"message",
"topic":"sometopic",
"message":"eyJhbGciOiJka.............-nH7ya1VQ_Y6ebT1w.2eyLaTUfc_rpKaZr4-5I1Q",
"encoding":"jwe"
}
Request (HTTP)
POST /sometopic HTTP/1.1
Encoding: jwe
Content-Type: multipart/form-data; boundary=--ntfy-boundaryXyASDsdA
----ntfy-boundaryXyASDsdA
Content-Disposition: form-data; name="message"
eyJhbGciOiJka.............-nH7ya1VQ_Y6ebT1w.2eyLaTUfc_rpKaZr4-5I1Q
----ntfy-boundaryXyASDsdA
Content-Disposition: form-data; name="attachment"
eyJhbGciOiJk......(this could be 15 MB of data) .........Rzs4Y5QLE2XD2_aw_SQ.y2hadrN5b2LEw7_PJHhbcA
----ntfy-boundaryXyASDsdA--
Request (via curl)
curl ntfy.sh/sometopic \
-H "Encoding: jwe" \
-F attachment=@encrypted-attachment.bin \
-F message=@encrypted-message.bin'
Response
{
"id":"eWa5epsDHkQ4",
"time":1657759261,
"event":"message",
"topic":"sometopic",
"message":"eyJhbGciOiJka.............-nH7ya1VQ_Y6ebT1w.2eyLaTUfc_rpKaZr4-5I1Q",
"encoding":"jwe",
"attachment": {
"name": "attachment.bin",
"type": "application/octet-stream",
"size": 715814,
"expires": 1657775583,
"url": "https://ntfy.sh/file/tmhElNL0MiKM.bin",
"encoding":"jwe", // <<<<<<<<<< Do we need this?
}
}
sha256(topic)
insteadNTFY_PASSWORD=... ntfy publish --encrypt mytopic my encrypted message
Friendly reminder: environment variables are only little less bad means to pass secrets than command line. Otherwise +1 (or more).
I want to throw in two considerations regarding the decision against "age" and "openpgp":
golang.org/x
tree is no longer maintained, but they even referenced the existing fork from ProtonMail as a successor for people who continue using OpenPGP. We use it in production, for example. (Just as ProtonMail does, I would assume.)age
does not have existing implementations in a lot of languages, rolling your own scheme doesn't either. If you start implementing a scheme, you might as well use one with a spec. The advantage over a custom / own scheme is obviously that it is already more battle tested and you have reference implementations to test against.@aksdb Thanks for the comments.
Re 1: Thanks for the correction. I was not aware that there was another active implementation.
Re 2: The scheme that I picked is not a roll-your-own. It's a subset of JWE. I picked very small subset, so that I could implement it in many different languages easily. I already did it in Python, PHP, Go, and (partially) JS. It is understandable that you think it's roll-your-own, because this ticket discussion is long and old, but this is the one I picked: https://github.com/binwiederhier/ntfy/issues/69#issuecomment-1183839284 -- which is JWE with AES-256-GCM in "dir" mode ({"alg":"dir","enc":A256GCM"}
).
Re 2 (age): I did approach the age devs in the GitHub discussions asking for different implementations, and while there is interest, it doesn't seem like it's been done or actively worked on/maintained. Most of the chosen scheme I could implement in 10 lines of code in most languages. It's just AES-256-GCM and a bunch of base64-ing. Simple and I've done that many many times. I'm not new to this :)
Sounds good, thanks for the clarification (and all the effort you put into this project).
Awesome project @binwiederhier! Great work. Password protection would be a really great feature that would make the tool a lot more powerful.
A small question (I have no crypto background): What would happen if two people used (e.g. by accident) the same topic with a different password? Would that be possible? I guess they wouldn't receive the other person's notification and only their own (because they can only decipher their own message)?! Would they get an error message of an attempted notification/wrong password?
Regarding key derivation: since this is a new design and you have a choice, it doesn't make much sense to use PBKDF2. Instead, a memory-hard hash function like Argon2 or scrypt is far preferable.
If you ultimately stick with PBKDF2, the proposed number of iterations (50,000) is too low to defend against brute-force attacks on modern hardware. 1Password uses 650,000 iterations now, and OWASP recommends at least 600,000 iterations of PBKDF2-HMAC-SHA256: https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html#pbkdf2
Regarding key derivation: since this is a new design and you have a choice, it doesn't make much sense to use PBKDF2. Instead, a memory-hard hash function like Argon2 or scrypt is far preferable.
I think this is a great recommendation, thank you. I think I picked something "low" for PBKDF2 since there are clients that will have to derive the password every time, which would make sending messages have a delay of 1s or something, or require storing the key somewhere. I'll experiment with the other key derivation functions though.
We now also have access tokens since 2.x, so it may be feasible to ditch the password altogether. I'll look at it when I eventually get to this ticket.
A small question (I have no crypto background): What would happen if two people used (e.g. by accident) the same topic with a different password? Would that be possible? I guess they wouldn't receive the other person's notification and only their own (because they can only decipher their own message)?! Would they get an error message of an attempted notification/wrong password?
@blewsky according to the "General Flow" in https://github.com/binwiederhier/ntfy/issues/69#issuecomment-1183839284 you would get a notification with (encrypted message)
This would only protect content, not the topic itself, you could have a combined encrypted and unencrypted topic even.
also, slightly interesting tidbit, there is nothing really stopping you from creating a custom client and using this with the current setup, the only thing you would miss is the native Encoding
header support
What is the relation here to the Web Push Payload Encryption?
I gave an application server https://ntfy.sh/mytopic
as the Web Push endpoint
instead of the browser's generated endpoint URL. This resulted in receiving encrypted binary blobs in the ntfy Android app.
Seems to me like the Android app could act like a browser and have its own keys (per subscription), while the CLI or whatever would act as the application server and have its own keys. Also using Web Push libraries would avoid rolling own crypto.
If I'm on the wrong track here I'm interested in learning more about what the difference is.
I'm happy to see E2E as a feature to ensure every content being pushed is only visible for the push source and the certain recipients, and not readable for either some vendor's push server (like APNs or Firebase) or temporary push server. Additionally, I think passphrase instead of key is better, because generate a passphrase requires zero knowledge and passphrase can be shared easily. Generate key requires more work, and exchange key content offline is harder.
Thus I think JWE or GPG is good enough. As for Web Push Payload Encryption, it is a pre-HTTPS-everywhere tech, more about preventing insecure push providers using HTTP to send your content, as it stated in the blog:
browser chooses which push provider will be used to actually deliver the payload,..., HTTPS can only guarantee that no one can snoop on the message in transit to the push service provider. Once they receive it, they are free to do what they like, including re-transmitting the payload to third-parties or maliciously altering it to something else. To protect against this we use encryption to ensure that push services can't read or tamper with the payloads in transit.
Seems like much of this could be overkill. A simpler way is to treat it like every other method (Signal, Matrix, etc). Provide a way to send a push that tells the client to check a predetermined URL via HTTPS using its bearer token. The user can then determine if they wish to do E2E, but this would be enough to keep Google's eyes out.
This is basically what I do for Matrix API from a shell script for pushes via Element/Matrix.
As for Web Push Payload Encryption, it is a pre-HTTPS-everywhere tech, more about preventing insecure push providers using HTTP to send your content, as it stated in the blog
My reading is that it does not state that Web Push encryption protects from insecure push providers using HTTP.
First they assume everyone uses HTTPS, but "HTTPS can only guarantee that no one can snoop on the message in transit to the push service provider."
Then they go on to explain that HTTPS isn't End-to-End Encryption, and E2EE is needed to prevent Push Providers from snooping on messages, which is why they added E2EE to Web Push.
Part of the reason I'm asking if Web Push Payload Encryption has been considered is because it is an existing solution widely used in production already.
Another reason is that while people claim "Web Push compatibility", there can not be any meaningful Web Push compatibility without clients that can decrypt the messages.
Ntfy is almost Web Push compatible. I've had third party services (application servers I do not own) send push messages into my test endpoint. The only missing piece is decryption.
HTTPS can only guarantee that no one can snoop on the message in transit to the TLS terminator. This could be Cloudflare, a corporate security proxy, an evil MITM proxy whose operator has the private key of some TLS certificate trusted by your device, etc. HTTPS is not enough by itself.
Any updates for this feature?
I’d like to suggest NaCl for consideration as an encryption/decryption library. In JavaScript TweetNaCl.js can be used, and for Android Lazysodium can be used. They support both symmetric and asymmetric encryption.
It should be possible to e2e encrypt messages, like this:
Or this
The message should look like this:
It is important to me that the solution is widely supported and can be easily implemented in all languages. Ideally something like
gpg
orage
, but I doubt such a format exists.