search

binwiederhier / ntfy

Send push notifications to your phone or desktop using PUT/POST
https://ntfy.sh
Apache License 2.0
18.58k stars 731 forks source link

End-to-end encryption (E2E) between clients (Android app, CLI, web app) #69

Open binwiederhier opened 2 years ago

binwiederhier commented 2 years ago

It should be possible to e2e encrypt messages, like this:

NTFY_PASSWORD=... ntfy publish --encrypt mytopic my encrypted message

Or this

echo my encrypted message | <something> | curl -T- -H "Encrypted: yes" ntfy.sh/mytopic

The message should look like this:

{
  "id": "..",
  "type": "message",
  "encryption": "aes-128-gcm",
  "message": "ZnNkZmRzZnNkZmZzZGtqZmRzamZsc2RmCg=="
}

It is important to me that the solution is widely supported and can be easily implemented in all languages. Ideally something like gpg or age, but I doubt such a format exists.

binwiederhier commented 2 years ago

I looked at this a little today, and sadly had to rule out gpg, because the library for Go is deprecated, and I don't want to build on deprecated technology. I also looked at age a while ago and even opened an issue here (https://github.com/FiloSottile/age/discussions/404) to ask for support in other languages; and sadly there is not enough support.

I also played with openssl to see if I can encrypt and protect the integrity (~ AEAD-style), but I wasn't successful.

I will try and play some more, but I believe I'll have to implement my own format. Something similar to what I have done here (https://syncany.readthedocs.io/en/latest/security.html#encrypting-new-files), though much simpler hopefully.

binwiederhier commented 2 years ago

Here's the basic message structure: https://github.com/binwiederhier/ntfy/pull/354. This implements the exact same thing that Pushbullet does (https://docs.pushbullet.com/#encryption) and is even compatible with it. This was super easy. The trickier part is figuring out the UX of this. I'll have to think about that.

binwiederhier commented 2 years ago

This can encrypt a message with a key, but a key is not a password. The password we'll still have to derive using pbkdf2.

The question is now:

Relatively easy would be:

binwiederhier commented 2 years ago

Even though I have not updated this ticket, I have been working on it in the #354 branch. I've explored JWE as a solution for the crypto format, and implemented PoCs in PHP and Python and Go.

After realizing that that won't work for attachments, I have switched gears after a discussion with a co-worker and with @wunter8.

Proposal

General flow

Key derivation

Encryption file format

Publishing (without attachment)

PUT /mytopic HTTP/1.1
Encrypted: yes

<encrypted binary data, ntfy format>

Publishing (with attachment)

PUT /mytopic HTTP/1.1
Encrypted: yes
Content-Type: multipart/form-data; boundary=--ntfy-boundaryXyASDsdA

----ntfy-boundaryXyASDsdA
Content-Disposition: form-data; name="message"
Content-Length: 1234

<encrypted binary data, ntfy format>

----ntfy-boundaryXyASDsdA
Content-Disposition: form-data; name="attachment"
Content-Length: 42343242

<encrypted binary data, ntfy format>

----ntfy-boundaryXyASDsdA--
goalieca commented 2 years ago

Password:

KDF:

goalieca commented 2 years ago

It might also be worth adding 'kid' to the JWEs so that can give users a chance to change or rotate their passwords. Changing the passwords has some choices because of historically encrypted messages. Since the user holds the password it would be up to them to have to re-encrypt every message.

Using a key-encryption-key (KEK) to encrypt the JWEs would provide better security. The KEK would be wrapped by the password and the client would store the wrapped KEK somewhere. It could be password wrapped, it could be key-vaulted, who knows. How to transport that KEK between endpoints? iOS has cloud-sync as an example. Or if it was wrapped by the password (pbkdf2) then could nfty itself do it? Rotation the password on the KEK is easier as there's just one. Still hard to rotate the KEK though.

I'll think some more on ergonomics of this.

binwiederhier commented 2 years ago

I am still working on this. It's going slow (mostly due to personal commitments), but I have still managed to come up with a design that I think I like. So here it goes:

Proposal 7/13

General flow

Key derivation

Encryption file format

Use cases

1. Unencrypted message (headers)

Request (HTTP)

POST /sometopic HTTP/1.1
Title: some title

some message

Request (via curl)

curl -d "some message" -H "Title: some title" ntfy.sh/sometopic

Response

{
  "id":"eWa5epsDHkQ4",
  "time":1657759261,
  "event":"message",
  "topic":"sometopic",
  "title":"some title",
  "message":"some message"
}

2. Unencrypted message (JSON)

Request (HTTP)

POST / HTTP/1.1

{
  "topic":"sometopic",
  "title":"some title",
  "message":"some message"
}

Request (via curl)

curl ntfy.sh -d '{
  "topic":"sometopic",
  "title":"some title",
  "message":"some message"
}'

Response

{
  "id":"eWa5epsDHkQ4",
  "time":1657759261,
  "event":"message",
  "topic":"sometopic",
  "title":"some title",
  "message":"some message"
}

3. Unencrypted message with attachment (headers + attachment)

Request (HTTP)

POST /sometopic HTTP/1.1
Title: some title
Message: some message

<attachment data>

Request (via curl)

curl -T attachment.jpg -H "Title: some title" -H "Message: some message" ntfy.sh/sometopic

Response

{
  "id":"eWa5epsDHkQ4",
  "time":1657759261,
  "event":"message",
  "topic":"sometopic",
  "title":"some title",
  "message":"some message",
  "attachment": {
    "name": "attachment.jpg",
    "type": "image/jpeg",
    "size": 715814,
    "expires": 1657775583,
    "url": "https://ntfy.sh/file/tmhElNL0MiKM.jpg"
  }
}

4. Unencrypted message with attachment (multipart: JSON + attachment) [** new]

Request (HTTP)

POST / HTTP/1.1
Content-Type: multipart/form-data; boundary=--ntfy-boundaryXyASDsdA

----ntfy-boundaryXyASDsdA
Content-Disposition: form-data; name="message"

{
  "topic":"sometopic",
  "title":"some title",
  "message":"some message"
}

----ntfy-boundaryXyASDsdA
Content-Disposition: form-data; name="attachment"

<attachment data>

----ntfy-boundaryXyASDsdA--

Request (via curl)

curl ntfy.sh \
  -F attachment=@attachment.jpg \
  -F message='{
    "topic":"sometopic",
    "title":"some title",
    "message":"some message"
  }'

Response

{
  "id":"eWa5epsDHkQ4",
  "time":1657759261,
  "event":"message",
  "topic":"sometopic",
  "title":"some title",
  "message":"some message",
  "attachment": {
    "name": "attachment.jpg",
    "type": "image/jpeg",
    "size": 715814,
    "expires": 1657775583,
    "url": "https://ntfy.sh/file/tmhElNL0MiKM.jpg"
  }
}

5. Encrypted message (JWE-encrypted JSON) [** new]

Request (HTTP)

POST /sometopic HTTP/1.1
Encoding: jwe

eyJhbGciOiJka.............-nH7ya1VQ_Y6ebT1w.2eyLaTUfc_rpKaZr4-5I1Q

Request (via curl)

curl ntfy.sh/sometopic \
  -H "Encoding: jwe" \
  -d "eyJhbGciOiJka.............-nH7ya1VQ_Y6ebT1w.2eyLaTUfc_rpKaZr4-5I1Q"

Response

{
  "id":"eWa5epsDHkQ4",
  "time":1657759261,
  "event":"message",
  "topic":"sometopic",
  "message":"eyJhbGciOiJka.............-nH7ya1VQ_Y6ebT1w.2eyLaTUfc_rpKaZr4-5I1Q",
  "encoding":"jwe"
}

6. Encrypted with attachment (multipart: JWE-encrypted JSON + JWE-encrypted attachment) [** new]

Request (HTTP)

POST /sometopic HTTP/1.1
Encoding: jwe
Content-Type: multipart/form-data; boundary=--ntfy-boundaryXyASDsdA

----ntfy-boundaryXyASDsdA
Content-Disposition: form-data; name="message"

eyJhbGciOiJka.............-nH7ya1VQ_Y6ebT1w.2eyLaTUfc_rpKaZr4-5I1Q

----ntfy-boundaryXyASDsdA
Content-Disposition: form-data; name="attachment"

eyJhbGciOiJk......(this could be 15 MB of data) .........Rzs4Y5QLE2XD2_aw_SQ.y2hadrN5b2LEw7_PJHhbcA

----ntfy-boundaryXyASDsdA--

Request (via curl)

curl ntfy.sh/sometopic \
  -H "Encoding: jwe" \
  -F attachment=@encrypted-attachment.bin \
  -F message=@encrypted-message.bin'

Response

{
  "id":"eWa5epsDHkQ4",
  "time":1657759261,
  "event":"message",
  "topic":"sometopic",
  "message":"eyJhbGciOiJka.............-nH7ya1VQ_Y6ebT1w.2eyLaTUfc_rpKaZr4-5I1Q",
  "encoding":"jwe",
  "attachment": {
    "name": "attachment.bin",
    "type": "application/octet-stream",
    "size": 715814,
    "expires": 1657775583,
    "url": "https://ntfy.sh/file/tmhElNL0MiKM.bin",
    "encoding":"jwe",  // <<<<<<<<<< Do we need this? 
  }
}

Issues/questions

  1. JWE has a 30% overhead due to base64; that's okay for messages, but a little sad for attachments
  2. ~The salt for PBKDF2 is the topic URL. In some cases, endpoints are different and clients will derive an "incorrect key". Any ideas for a better salt?~ We'll use sha256(topic) instead
  3. For use case 5+6, the topic+encoding has to be passed as header. That ok, right?
goalieca commented 2 years ago
  1. base64 does appear to compress well with gzip (https://blog.virtual-void.net/base64-vs-gzip/)
steelman commented 2 years ago 
NTFY_PASSWORD=... ntfy publish --encrypt mytopic my encrypted message

Friendly reminder: environment variables are only little less bad means to pass secrets than command line. Otherwise +1 (or more).

aksdb commented 2 years ago

I want to throw in two considerations regarding the decision against "age" and "openpgp":

  1. The Go OpenPGP implementation is still maintained. The one that was in the golang.org/x tree is no longer maintained, but they even referenced the existing fork from ProtonMail as a successor for people who continue using OpenPGP. We use it in production, for example. (Just as ProtonMail does, I would assume.)
  2. While age does not have existing implementations in a lot of languages, rolling your own scheme doesn't either. If you start implementing a scheme, you might as well use one with a spec. The advantage over a custom / own scheme is obviously that it is already more battle tested and you have reference implementations to test against.
binwiederhier commented 2 years ago

@aksdb Thanks for the comments.

Re 1: Thanks for the correction. I was not aware that there was another active implementation.

Re 2: The scheme that I picked is not a roll-your-own. It's a subset of JWE. I picked very small subset, so that I could implement it in many different languages easily. I already did it in Python, PHP, Go, and (partially) JS. It is understandable that you think it's roll-your-own, because this ticket discussion is long and old, but this is the one I picked: https://github.com/binwiederhier/ntfy/issues/69#issuecomment-1183839284 -- which is JWE with AES-256-GCM in "dir" mode ({"alg":"dir","enc":A256GCM"}).

Re 2 (age): I did approach the age devs in the GitHub discussions asking for different implementations, and while there is interest, it doesn't seem like it's been done or actively worked on/maintained. Most of the chosen scheme I could implement in 10 lines of code in most languages. It's just AES-256-GCM and a bunch of base64-ing. Simple and I've done that many many times. I'm not new to this :)

aksdb commented 2 years ago

Sounds good, thanks for the clarification (and all the effort you put into this project).

blewsky commented 2 years ago

Awesome project @binwiederhier! Great work. Password protection would be a really great feature that would make the tool a lot more powerful.

A small question (I have no crypto background): What would happen if two people used (e.g. by accident) the same topic with a different password? Would that be possible? I guess they wouldn't receive the other person's notification and only their own (because they can only decipher their own message)?! Would they get an error message of an attempted notification/wrong password?

Fysac commented 1 year ago

Regarding key derivation: since this is a new design and you have a choice, it doesn't make much sense to use PBKDF2. Instead, a memory-hard hash function like Argon2 or scrypt is far preferable.

If you ultimately stick with PBKDF2, the proposed number of iterations (50,000) is too low to defend against brute-force attacks on modern hardware. 1Password uses 650,000 iterations now, and OWASP recommends at least 600,000 iterations of PBKDF2-HMAC-SHA256: https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html#pbkdf2

binwiederhier commented 1 year ago

Regarding key derivation: since this is a new design and you have a choice, it doesn't make much sense to use PBKDF2. Instead, a memory-hard hash function like Argon2 or scrypt is far preferable.

I think this is a great recommendation, thank you. I think I picked something "low" for PBKDF2 since there are clients that will have to derive the password every time, which would make sending messages have a delay of 1s or something, or require storing the key somewhere. I'll experiment with the other key derivation functions though.

We now also have access tokens since 2.x, so it may be feasible to ditch the password altogether. I'll look at it when I eventually get to this ticket.

gardient commented 1 year ago

A small question (I have no crypto background): What would happen if two people used (e.g. by accident) the same topic with a different password? Would that be possible? I guess they wouldn't receive the other person's notification and only their own (because they can only decipher their own message)?! Would they get an error message of an attempted notification/wrong password?

@blewsky according to the "General Flow" in https://github.com/binwiederhier/ntfy/issues/69#issuecomment-1183839284 you would get a notification with (encrypted message)

This would only protect content, not the topic itself, you could have a combined encrypted and unencrypted topic even.

also, slightly interesting tidbit, there is nothing really stopping you from creating a custom client and using this with the current setup, the only thing you would miss is the native Encoding header support

MzHub commented 1 year ago

What is the relation here to the Web Push Payload Encryption?

I gave an application server https://ntfy.sh/mytopic as the Web Push endpoint instead of the browser's generated endpoint URL. This resulted in receiving encrypted binary blobs in the ntfy Android app.

Seems to me like the Android app could act like a browser and have its own keys (per subscription), while the CLI or whatever would act as the application server and have its own keys. Also using Web Push libraries would avoid rolling own crypto.

If I'm on the wrong track here I'm interested in learning more about what the difference is.

SilverBut commented 1 year ago

I'm happy to see E2E as a feature to ensure every content being pushed is only visible for the push source and the certain recipients, and not readable for either some vendor's push server (like APNs or Firebase) or temporary push server. Additionally, I think passphrase instead of key is better, because generate a passphrase requires zero knowledge and passphrase can be shared easily. Generate key requires more work, and exchange key content offline is harder.

Thus I think JWE or GPG is good enough. As for Web Push Payload Encryption, it is a pre-HTTPS-everywhere tech, more about preventing insecure push providers using HTTP to send your content, as it stated in the blog:

browser chooses which push provider will be used to actually deliver the payload,..., HTTPS can only guarantee that no one can snoop on the message in transit to the push service provider. Once they receive it, they are free to do what they like, including re-transmitting the payload to third-parties or maliciously altering it to something else. To protect against this we use encryption to ensure that push services can't read or tamper with the payloads in transit.

Fmstrat commented 1 year ago

Seems like much of this could be overkill. A simpler way is to treat it like every other method (Signal, Matrix, etc). Provide a way to send a push that tells the client to check a predetermined URL via HTTPS using its bearer token. The user can then determine if they wish to do E2E, but this would be enough to keep Google's eyes out.

This is basically what I do for Matrix API from a shell script for pushes via Element/Matrix.

MzHub commented 1 year ago

As for Web Push Payload Encryption, it is a pre-HTTPS-everywhere tech, more about preventing insecure push providers using HTTP to send your content, as it stated in the blog

My reading is that it does not state that Web Push encryption protects from insecure push providers using HTTP.

First they assume everyone uses HTTPS, but "HTTPS can only guarantee that no one can snoop on the message in transit to the push service provider."

Then they go on to explain that HTTPS isn't End-to-End Encryption, and E2EE is needed to prevent Push Providers from snooping on messages, which is why they added E2EE to Web Push.

Part of the reason I'm asking if Web Push Payload Encryption has been considered is because it is an existing solution widely used in production already.

Another reason is that while people claim "Web Push compatibility", there can not be any meaningful Web Push compatibility without clients that can decrypt the messages.

Ntfy is almost Web Push compatible. I've had third party services (application servers I do not own) send push messages into my test endpoint. The only missing piece is decryption.

CyberShadow commented 1 year ago

HTTPS can only guarantee that no one can snoop on the message in transit to the TLS terminator. This could be Cloudflare, a corporate security proxy, an evil MITM proxy whose operator has the private key of some TLS certificate trusted by your device, etc. HTTPS is not enough by itself.

cmj2002 commented 1 year ago

Any updates for this feature?

dtinth commented 4 months ago

I’d like to suggest NaCl for consideration as an encryption/decryption library. In JavaScript TweetNaCl.js can be used, and for Android Lazysodium can be used. They support both symmetric and asymmetric encryption.

7heo commented 1 month ago

It should be possible to e2e encrypt messages, like this:

NTFY_PASSWORD=... ntfy publish --encrypt mytopic my encrypted message

Or this

echo my encrypted message | <something> | curl -T- -H "Encrypted: yes" ntfy.sh/mytopic

The message should look like this:

{
  "id": "..",
  "type": "message",
  "encryption": "aes-128-gcm",
  "message": "ZnNkZmRzZnNkZmZzZGtqZmRzamZsc2RmCg=="
}

It is important to me that the solution is widely supported and can be easily implemented in all languages. Ideally something like gpg or age, but I doubt such a format exists.

I would like to suggest a two-folds solution for implementing E2E in ntfy; which I believe would greatly ease its implementation.

Sender side

Instead of going

NTFY_PASSWORD=... ntfy publish --encrypt mytopic my encrypted message

Or

echo my encrypted message | <something> | curl -T- -H "Encrypted: yes" ntfy.sh/mytopic

as suggested, why not using:

echo "my plaintext message" | gpg --encrypt --trust-model always --armor -r "$email_or_hash" \
  | curl -d@- ntfy.sh/mytopic

Notice the use of -d@ with curl instead of -T, which results in a POST and not a PUT (I don't know if PUT is currently implemented in ntfy) Never mind, PUT is clearly implemented, it is even in the repository's summary.

Of course, it is necessary to add a recipient flag (-r) per recipient.

This method would literally not require any change to the server code.

Receiver side

On the receiver's end, the message is obtained as usual, but in case the first line is -----BEGIN PGP MESSAGE-----, the message is processed by an appropriate app (OpenKeychain on Android, and PGPro on iOS), so that the resulting clear-text can be displayed; or, if it failed, either the app displays a warning (or error) or silently ignores it, depending on per-topic configuration.

It would be probably good to display an appropriate warning by default, when receiving a PGP encrypted message, if no PGP app is installed on the device.

bogorad commented 4 weeks ago

Why all this complexity - do as Pushbullet does: if you set a secret on one end, all other endpoints won't be able to read messages unless they know the same secret. Doesn't really matter which encryption library is used, nacl is as good as any. Just mandate a lengthy secret.

7heo commented 1 week ago

Why all this complexity

What you advertise as "complexity" (pgp) is only the standard way to do things, supports both asymmetric and symmetric encryption, is already portable because it is standard, is distributed by any stable distribution (Linux or otherwise), and is factually orders of magnitude simpler than your proposal. Just because you will end up "using docker anyway" doesn't mean it's okay to force heaps of dependencies and sometimes barely-reviewed software ripe for supply-chain attacks on everyone. Please, no gaslighting, I'm being thorough, not "complex". And pgp isn't a "library". Ops matter.

do as Pushbullet does: if you set a secret on one end, all other endpoints won't be able to read messages unless they know the same secret. Doesn't really matter which encryption library is used, nacl is as good as any. Just mandate a lengthy secret.

Don't roll your own crypto.

From Phil Zimmermann's (PGP creator) Introduction to Cryptography (Page 43, section "Beware of snake oil"):

When I was in college in the early 70s, I devised what I believed was a brilliant encryption scheme. A simple pseudorandom number stream was added to the plaintext stream to create ciphertext. This would seemingly thwart any frequency analysis of the ciphertext, and would be uncrackable even to the most resourceful government intelligence agencies. I felt so smug about my achievement.

Years later, I discovered this same scheme in several introductory cryptography texts and tutorial papers. How nice. Other cryptographers had thought of the same scheme. Unfortunately, the scheme was presented as a simple homework assignment on how to use elementary cryptanalytic techniques to trivially crack it. So much for my brilliant scheme.

From this humbling experience I learned how easy it is to fall into a false sense of security when devising an encryption algorithm. Most people don’t realize how fiendishly difficult it is to devise an encryption algorithm that can withstand a prolonged and determined attack by a resourceful opponent.

bogorad commented 1 week ago

From Phil Zimmermann's (PGP creator

I have profound respect for Zimmermann, I actually started studying cryptography by reading his paper on PGP, also used to be a paid user of his Silent Circle project.

But PGP is total garbage in respect to useability. It is needlessly complex and convoluted. No one sane uses it in the 21st century (unless they are forced to). That's the reason we have Signal messenger and other projects with great usability and security. Dragging GPG into the new world is just crazy.

7heo commented 1 week ago

But PGP is total garbage in respect to useability.

Good thing OpenKeyChain and PGPro are great apps that are well designed and UX focused.

Also, "Secure, Convenient, Doesn't need hundreds of millions USD to implement; pick any two."

MzHub commented 1 week ago

As stated earlier there already is a standard for end-to-end push message encryption, and it is RFC8291.

To me it seems it would be beneficial for both interoperability and security ("don't roll your own crypto") to implement the standard.

brian6932 commented 1 week ago

PGP sucks, and I'd hardly consider it standardized, on the other hand, SSH's actually standardized, and broadly used in modern programs. It has replaced PGP in all modern contexts.

bogorad commented 1 week ago

or age, X25519 is fairly common.

dtinth commented 1 week ago

Speaking about SSH, I recently took a look at age that was earlier suggested by @binwiederhier and found that:

  1. It supports using SSH keys as a public/private key format (but only ssh-rsa and ssh-ed25519).
  2. It can be trivially compiled into WebAssembly.

Personally, I prefer asymmetric encryption (public/private keys) over passwords, so that if there are many senders sending notifications to the same topic, different senders cannot decrypt each other's messages.

7heo commented 1 week ago

As stated earlier there already is a standard for end-to-end push message encryption, and it is RFC8291.

I was not aware that there was a standard for E2EE of push messages specifically. I am surprised, but also thankful: that is valuable information. However, that would imply specific tooling on the sender, would it not? That was precisely what my proposal aimed to avoid.

PGP sucks, and I'd hardly consider it standardized, on the other hand, SSH's actually standardized, and broadly used in modern programs. It has replaced PGP in all modern contexts.

Absolutely fair. I'd also recommend SSH instead of PGP, now that you mention it. I don't know what apps can be used to support it on a mobile terminal, though.

Personally, I prefer asymmetric encryption (public/private keys) over passwords, so that if there are many senders sending notifications to the same topic, different senders cannot decrypt each other's messages.

Exactly. The same goes for receivers.

steelman commented 6 days ago

@brian6932

PGP sucks, and I'd hardly consider it standardized

FYI RFC 9580

K4LCIFER commented 4 days ago

I'm a little confused about the existence of this issue. The Unified Push spec explicitly states that notification messages must be encrypted:

Push message: This is an array of bytes (ByteArray) sent by the application server to the push server. The distributor sends this message to the end user application. It MUST be the raw POST data received by the push server (or the rewrite proxy if present). The message MUST be an encrypted content that follows RFC8291. Its size is between 1 and 4096 bytes (inclusive).

Does Ntfy not adhere to the Unified Push spec?

aksdb commented 4 days ago

Does Ntfy not adhere to the Unified Push spec?

Does it claim anywhere it would? I think the docs are quite clear that it implements its own protocol.

dtinth commented 4 days ago

@K4LCIFER There are 3 parties. Sender → Ntfy → Receiver.

So it is encrypted, but not end-to-end from Sender to Receiver. Right now Ntfy as the intermediary has to un-encrypt the message from the Sender, and then re-encrypt it for the Receiver, so it is technically possible for Ntfy service to access to the message contents. E2EE makes it impossible for the intermediary to access the message contents.