Making NIP-04 less leaky

[vinliao]

Edit: I've built a working POC. See the latest comment.

This is an improvement upon NIP-04. It makes private messaging leaks less metadata.

Idea:

1. Sender announces their pubkey to recipient
2. Both recipient and sender now has shared key (ecdh)
3. They both post private message to "inbox address" with sha256(shared_key) on its tag
4. Inbox address is 64 characters of "0" (00000000...00000)
5. With the shared key, only both can create the private event, only both can decrypt
6. Nobody knows who is messaging who since there's no pubkey on the event

Example of such an event:

{
  id: '9141dc50144cc243acfe78dc7799768e2f3eef2f857d8adc4fb8600dee6e8e9b',
  pubkey: '0000000000000000000000000000000000000000000000000000000000000000',
  created_at: 1643777968,
  kind: 4,
  tags: [
    [
      'shared',
      'dbb06024d1ec94e1d0b76b2105410ecf85f01f0ecf6de563b5485288f6eff6c1'
    ]
  ],
  content: 'ubrKJp8e2XKQ7utVAY66BQ==?iv=sy0vaW9Au0jIRSFqSrCjnA==',
  sig: '78fdfffb8bc983af75a250cb13225b75a74e104a82881fa35d7efa36f48dcd5bedfe43f29b50c7c26a5d39b1bee7d4f23b0c3516b1149c6bc047198e5f9acf99'
}

Request to relay to get such an event: ["REQ", "foobar", {"#shared", sha256(shared_key)}]

Cons of this idea:

1. Relays don't recognize such an event, therefore I haven't fully tested this idea
2. Since the pubkeys aren't present on the event, the signature's legitimacy can't be verified
3. Announcing sender's pubkey to recipient leaks metadata

Please poke holes in this idea (if you find any); I'd like to know whether this idea has merit or it's just full of hot air.

The JS code below is an implementation of this idea.

import * as secp from "@noble/secp256k1";
import crypto from "crypto";

const senderPriv = "d5f9b88ae04e7adb2fc075515e39df546df56d88ccdb304a9a779af1563d79ba";
const senderPub = "ab1a33b0cf3d8f8896c433e6996744e48f1401e6fbc94aea6f84291074fb1b75";
const recipientPriv = "c10d9871f37f5d7dae09e93f2a381593b57697e02e15b446fbc99531b4623555";
const recipientPub = "7002538efd7175b2b5fafe4ee5a933242081c067a48ff019deca56eb13ef2186";
const inboxAddress = "0000000000000000000000000000000000000000000000000000000000000000";

function toHexString(byteArray) {
  return Array.prototype.map
    .call(byteArray, function (byte) {
      return ("0" + (byte & 0xff).toString(16)).slice(-2);
    })
    .join("");
}

async function broadcastKey() {
  // the purpose of this is to announce sender's pubkey to recipient
  // it's basically a normal kind `4` event

  const dummyMessage = "adsfasdf" // doesn't matter
  const unixTime = Math.floor(Date.now() / 1000);
  const data = [0, senderPub, unixTime, 4, [["p", recipientPub]], dummyMessage];

  const eventString = JSON.stringify(data);
  const eventByteArray = new TextEncoder().encode(eventString);
  const eventIdRaw = await secp.utils.sha256(eventByteArray);
  const eventId = toHexString(eventIdRaw);

  const signatureRaw = await secp.schnorr.sign(eventId, senderPriv);
  const signature = toHexString(signatureRaw);

  const sampleBroadcastEvent = {
    id: eventId,
    pubkey: senderPub,
    created_at: unixTime,
    kind: 4,
    tags: [["p", recipientPub]],
    content: dummyMessage,
    sig: signature
  }

  // sampleBroadcastEvent looks like this:
  /*
  {
    id: '714f104515c8980dc9a793cf15b0a8700f8be1e2436165217bfca6976975c1d4',
    pubkey: 'ab1a33b0cf3d8f8896c433e6996744e48f1401e6fbc94aea6f84291074fb1b75',
    created_at: 1643778625,
    kind: 4,
    tags: [
      [
        'p',
        '7002538efd7175b2b5fafe4ee5a933242081c067a48ff019deca56eb13ef2186'
      ]
    ],
    content: 'adsfasdf',
    sig: '03e9b5c375188ff912d536cb8e59bc45e643b0eca642838e8fa8f8692c1ce7bb051a22ff83dd2d17aefc0703ea046aaef9ec3859559a01cbf45226957a57e520'
  }
  */

  // push sampleBroadcastEvent to relay: `["EVENT", sampleBroadcastEvent]`
  // recipient can get the event: `["REQ", "foobar", {"#p": [recipientPub]}]`
  // when both have each other's pubkey, they can generate the shared key
  // when they both have the shared key, they can communicate privately through inbox address
}

async function generatePrivateEvent(priv, pub) {
  const unencryptedMessage = "supersecret"

  // `secp.getSharedSecret(senderPriv, "02" + recipientPub)`
  // and
  // `secp.getSharedSecret(recipientPriv, "02" + senderPub)`
  // produce the same value
  const sharedPointBytes = secp.getSharedSecret(priv, "02" + pub);
  const sharedPoint = toHexString(sharedPointBytes);
  const sharedX = sharedPoint.substr(2, 64)
  const sharedXByteArray = new TextEncoder().encode(sharedX);
  const sharedXByte = await secp.utils.sha256(sharedXByteArray);
  const sharedXSha = toHexString(sharedXByte);

  const iv = crypto.randomFillSync(new Uint8Array(16))
  const ivBase64 = Buffer.from(iv.buffer).toString('base64')
  const cipher = crypto.createCipheriv(
    'aes-256-cbc',
    Buffer.from(sharedX, 'hex'),
    iv
  )

  // to decrypt later on, use `crypto.createDecipheriv()`

  let encryptedMessage = cipher.update(JSON.stringify(unencryptedMessage), 'utf8', 'base64')
  encryptedMessage += cipher.final('base64')
  encryptedMessage += "?iv=" + ivBase64

  const unixTime = Math.floor(Date.now() / 1000);
  const data = [0, inboxAddress, unixTime, 4, [["shared", sharedXSha]], encryptedMessage];

  // event id is sha256 of data above
  // sig is schnorr sig of id
  const eventString = JSON.stringify(data);
  const eventByteArray = new TextEncoder().encode(eventString);
  const eventIdRaw = await secp.utils.sha256(eventByteArray);
  const eventId = toHexString(eventIdRaw);

  const signatureRaw = await secp.schnorr.sign(eventId, priv);
  const signature = toHexString(signatureRaw);

  return {
    id: eventId,
    pubkey: inboxAddress,
    created_at: unixTime,
    kind: 4,
    tags: [["shared", sharedXSha]],
    content: encryptedMessage,
    sig: signature
  }
}

(async () => {
  console.log(await generatePrivateEvent(senderPriv, recipientPub))
  console.log(await generatePrivateEvent(recipientPriv, senderPub))
})();

Edit: This issue has been edited to clarify.

[fiatjaf]

I like this idea, but for a different NIP instead of NIP04. And one can use random keys to sign instead of a blank key.

Maybe instead of using a shared key parties could instead send an initial message through NIP04 containing a random encryption key and then proceed using this new protocol using that?

[vinliao]

Or, instead of a blank key (inbox address), both parties use MuSig2 as the event's pubkey.

Ideally, no one can even surveil which pubkey talks to which pubkey. The problem here seems to be privately announcing sender's pubkey before both can talk privately using this new protocol.

Introducing "encryption key" to Nostr means future devs have to juggle between RSA and Schnorr, no?

I think loquaz and nostr.chat are suited to implement this idea, to experiment and see whether this new protocol actually works.

[fiatjaf]

By encryption key I meant the key that would be used to encrypt the messages using the same AES-CBC that NIP04 was using already, not RSA.

[vinliao]

Revision to the idea:

• Instead of using a blank address, a Schnorr keypair is derived from the ECDH key
• The shared Schnorr can be used to sign and can be used as the event's pubkey
• Remove the "shared" tag (unnecessary)

Result:

• Recipient's and sender's pubkey aren't present in the event
• Event can now have a valid signature (the blank address couldn't)
• Only both can pull, encrypt, sign, decrypt the private event

Below is the JS implementation of this revised idea. I've published an event to a relay, and it's actually accepted as a valid NIP-04 event.

async function generatePrivateEvent(priv, pub) {
  const unencryptedMessage = "supersecret"

  // `secp.getSharedSecret(senderPriv, "02" + recipientPub)`
  // and
  // `secp.getSharedSecret(recipientPriv, "02" + senderPub)`
  // produce the same value
  const sharedPointBytes = secp.getSharedSecret(priv, "02" + pub);
  const sharedPoint = toHexString(sharedPointBytes);
  const sharedX = sharedPoint.substr(2, 64)

  // not sure how safe this is, but sharedX is 32 bytes
  // which can be used as a privatekey, and which
  // a schnorr pubkey can be derived from
  const schnorrMutualKey = secp.schnorr.getPublicKey(sharedX)
  const schnorrMutualHex = toHexString(schnorrMutualKey)

  const iv = crypto.randomFillSync(new Uint8Array(16))
  const ivBase64 = Buffer.from(iv.buffer).toString('base64')
  const cipher = crypto.createCipheriv(
    'aes-256-cbc',
    Buffer.from(sharedX, 'hex'),
    iv
  )

  let encryptedMessage = cipher.update(JSON.stringify(unencryptedMessage), 'utf8', 'base64')
  encryptedMessage += cipher.final('base64')
  encryptedMessage += "?iv=" + ivBase64

  const unixTime = Math.floor(Date.now() / 1000);
  const data = [0, schnorrMutualHex, unixTime, 4, [], encryptedMessage];

  // event id is sha256 of data above
  // sig is schnorr sig of id
  const eventString = JSON.stringify(data);
  const eventByteArray = new TextEncoder().encode(eventString);
  const eventIdRaw = await secp.utils.sha256(eventByteArray);
  const eventId = toHexString(eventIdRaw);

  // sharedX is used as a private key here
  const signatureRaw = await secp.schnorr.sign(eventId, sharedX);
  const signature = toHexString(signatureRaw);

  return {
    id: eventId,
    pubkey: schnorrMutualHex,
    created_at: unixTime,
    kind: 4,
    tags: [],
    content: encryptedMessage,
    sig: signature
  }

  // example output
  /*
  {
    id: '24d6b6ba8b938c393ed13ee8fb1cfd639fc841f936dcbe28a039338340acda03',
    pubkey: '6646a4323f0e7234488579975d2dcaabfe87e8b086a91ddc3f08f4c9e978e82d',
    created_at: 1643932277,
    kind: 4,
    tags: [],
    content: 'lXDprQheAyTQ10AyKmYXcg==?iv=15FpmW00Nk+Fe+zdM+KCXA==',
    sig: '728b8dec8719cef760afd5a6b46ee02054441e9547c1c3af5c6fdd834b2c4ea7aeecb0b1c1584220c92278a1b987c1c8243586c5319fe5d76794e5c6341fc1c4'
  }
  */
}

async function decrypt(priv, pub) {
  const sharedPointBytes = secp.getSharedSecret(priv, "02" + pub);
  const sharedPoint = toHexString(sharedPointBytes);
  const sharedX = sharedPoint.substr(2, 64)

  const schnorrMutualKey = secp.schnorr.getPublicKey(sharedX)
  const schnorrMutualHex = toHexString(schnorrMutualKey)

  // request to relay: ["REQ", "foobar", [{"authors": schnorrMutualHex}]]
  // let's say we get back the data, then parse the encrypted message
  const encryptedMessage = "lXDprQheAyTQ10AyKmYXcg=="
  const iv = Buffer.from("15FpmW00Nk+Fe+zdM+KCXA==", "base64")

  const decipher = crypto.createDecipheriv('aes-256-cbc', Buffer.from(sharedX, 'hex'), iv)
  const message = Buffer.concat([decipher.update(encryptedMessage, 'base64'), decipher.final()]);
  console.log(message.toString()) // supersecret
}

I've sent a private message to @fiatjaf's known pubkey on Branle (3bf0c63fcb93463407af97a5e5ee64fa883d107ef9e558472c4eb9aaaefa459d) with a dummy pubkey (ab1a33b0cf3d8f8896c433e6996744e48f1401e6fbc94aea6f84291074fb1b75) through wss://relayer.fiatjaf.com. Here's what the event looks like:

{
  id: '66337672d578940cf97ec84a8d82e547564ec527a26ac275b039d52c7a7cfe4b',
  pubkey: 'c00a84bd0c368c32a339e7200995c419ebd4a9fdc0b4230f82271eb9ea13d394',
  created_at: 1643934337,
  kind: 4,
  tags: [],
  content: 'KO8Cqvw5RIuynVkXjJze5g==?iv=n4YGW9p0L/RoFbs0t3vBYA==',
  sig: 'f0e0b1f54920c6fa636b6151c4268e807e7f8f5457042a2e318d46171edc89b1956697d1e7cda011ad074252675c95db734104f19a243351b63c9275062bee86'
}

Neither pubkeys are displayed on the event. With this improvement, NIP-04 only leaks timestamp.

[vinliao]

I like this idea, but for a different NIP instead of NIP04.

If this idea actually works in improving privacy, I agree that this should be a new NIP and a new kind.

[vinliao]

I've implemented a half-baked POC to test this idea, and it's pretty... bad - it's a super-mega-complex stuff added on top of NIP-04 with marginal benefit. Not worth the headache.

[fiatjaf]

Hahaha, thank you for your honesty and for actually trying to implement. I was indeed afraid it would be too complex, but I thought that was just me being lazy. Since you, the creator of the idea, say it's bad super-mega-complex, then I'll follow you.

[HamishMacEwan]

Please poke holes in this idea (if you find any); I'd like to know whether this idea has merit or it's just full of hot air.

Wouldn't it be easier, on the client side, for the sender to create two disposable profiles, and NIP-04 a message to the receiver from the first profile with the second profile (and other information if desired, such as a private relay) as encrypted payload.

Receiver then creates a disposable profile and begins the conversation with the second profile from the sender (aka initiator).

It appears to have the advantage of being able to be implemented (🤞) using existing primitives, but could be formalised in a new NIP with a new kind.

(I see similarities to the concept you describe here https://github.com/fiatjaf/nostr/issues/70#issue-1126915577)

[vinliao]

It seems to me that using two disposable profiles makes impersonation easy. I think the alias system described on #70 is good enough to obfuscate NIP-04 pubkeys, and it's also extensible enough for other purposes.

[HamishMacEwan]

It seems to me that using two disposable profiles makes impersonation easy.

Three disposable profiles are involved, and how do you see any third party gaining access to the private keys of any of them (required for impersonation), or linking them to either of the main profiles?

[vinliao]

I can create two pubkeys, send one to you (encrypted), claiming to be some popular guy, telling you to trust me that I'm that popular guy, then tell you to chat with me through the pubkey I've encrypted to you. That counts as impersonation to me.

[HamishMacEwan]

telling you to trust me that I'm that popular guy

This popular guy's public key is not well known?

Telling me to do something, particularly to trust, isn't the same as me doing it.

This may be an attempt at impersonation but it isn't easy to succeed with this approach.

[vinliao]

Thanks to the comment above by @fiatjaf and @HamishMacEwan - I took some inspiration from their ideas. The ideas thrown around in this issue have merit. I've built a working POC: https://github.com/vinliao/clust. NIP-04 is more private now.

It's still clunky as hell, but it works. I'm gonna iron out a few UX details before sharing it to the Telegram group. Let me know if the explanation in the readme is lacking, or if there's a bug in the CLI app.

[Giszmo]

So what is the state of this? Please update the original description as we all agree that events with publicly known private keys are not cool. Heck, random others could even delete them as they appeared.

So do I understand right the last iteration was:

1. Alice derives kind-4 shared secret with Bob's pubKey
2. Alice derives key pair from secret
3. Alice sends events keypair's pubkey
4. Bob, knowing the privKey can decrypt events

• Only Alice and Bob know sender and recipient of each message
• For outside observers, these events are not attributable
• Alice might have to poke Bob. For example any kind-4 DM or opening the "DM with Bob" view could trigger clients to check kind-[to be announced] DMs, too.
• These DMs have plausible deniability as the recipient can forge messages to himself from the sender.
• Additional signatures in the content could attest actual authorship.

I find this protocol quite straight-forward and if I understood it correctly, could implement and try it in NostrPostrLib, too.

[fiatjaf]

I don't know the state of this, we're free to experiment with many things. So let me just throw some random thoughts:

NIP-04 was never supposed to be the perfect solution, just the simplest possible protocol until someone else made a good thing later (I was actually opposed to NIP-04 since the beginning).

My personal preferences are, in order:

1. Use NIP-04 to signal another transport mechanism outside of Nostr (like, I have no idea, direct Tor connections between the peers, or something else);
2. Use NIP-04 with relays that promise to not reveal messages to anyone except people who know a "room secret" or something like that.

Another unrelated thing I think is that we should try to address other use cases before DMs. It's very hard to gain users for DMs, the market is too crowded, and Nostr doesn't excel in things that are needed in DMs -- for example, phone notifications. I think group chats (which are public by design) could benefit more from Nostr (and benefit Nostr more).

[Giszmo]

@fiatjaf you are derailing this issue a bit.

I would hope to see the above to be pushed forward to replace nip-4 over some lets-use-telegram or lets-trust-relays-pinky-swear. The metadata leak of what I last described above is minimal and way better than nip-4 which sadly now is the default in all clients.

Not all group chats are public but for public group chats you could use kind a new "kind" but apply most of kind-1 to it. Clients would not show those in the feed but only show those groups that you subscribed to. A group is simply tagged like a hashtag. Most importantly for this would be to make abundantly clear that these chats are public.

[fiatjaf]

What I was saying is that NIP-04 should not be replaced at all. These ideas to make DMs better belong to different NIPs.

Also since there are multiple different approaches being talked about by different people and none of them seems to be obviously the best possible way to do anything (they all look hacky and ugly to me), my suggestion was that interested parties experiment with them.

[Giszmo]

So I think I came up with something simple enough to gain traction. I just updated the referenced nips PR to a completely different approach which just removes the leaky data from the messages. A summary is here.

The upside over nip-4:

• Not leaking recipient data in events at all if they are following us anyway or we can poke them out of band to check their inbox or they can just check all DMs (currently this is trivially doable).
• Good anonymity set against timestamp correlation analysis as events only reveal that pubkey sent a DM.
• Simple to implement if you already support nip-4

Downside:

• For privacy, the recipient has to query more events and check if they can be decrypted
• Once nostr traffic picks up and querying all kind-18 events becomes hard, recipient requires defence against relays spying on his events of interest.
• Replying to an event is not covered yet. I'm not sure it's needed. Let clients simply process an inline [](e/<event id>). Of course such event linking could lead to clients leaking interest in kind-18 events again.

The linked explanation:

All mentioned issues are addressed by a massive simplification of the protocol. The new protocol is:

• Pretend it's nip-04
• Put recipient only on first messages or not if they are following you already. False recipients are encouraged on subsequent messages.
• Query all your follows' kind-18 events regardless of the advertised recipient.
• Query all kind-18 events that advertised you as recipient.
• Query all events from pubkeys that previously sent you events using some privacy tools such as TOR.
• Or ... query all kind-18 events as long as it's low volume.

[tclementdev]

Alternatively, would there be any interest in allowing relays to require REQ messages for kind 4 events to include a pubkey and signature? So that relays only deliver events if the REQ signature is valid and pubkey matches the recipient for the kind 4 events. I know it doesn't help if you do not trust the relays, but could be a simple change towards improving the situation.

[fiatjaf]

@tclementdev that's what NIP-42 is doing, and is being implemented in multiple relays and clients already.

[tclementdev]

@fiatjaf ah thank you, I missed that. Now one would need to make sure to only use relays that restrict kind 4 access for DMs.