search

canarytail / standard

A document proposing the standardization of a model for warrant canaries to facilitate scalable adoption by enabling automated monitoring and validation.
https://canarytail.org
69 stars 3 forks source link

Standardize URI? #4

Open egberts opened 4 years ago

egberts commented 4 years ago

I read an article over at Hacker News (YCombinator.com) and then this current version of README.md; perhaps we also need a convergence upon a standardized mime:

And standardize URI like:

And URL like:

I’m leaning toward something like:

Or is it necessary to avoid a well known URI for this purpose?

xAptive commented 4 years ago

I had thought the same. I think using .well-known is probably the right way to do it. There is an RFC covering it, as well as a registration process. And canary.json appears to be available.

carrotcypher commented 4 years ago

That does make sense. Building a PoC at the moment, will incorporate that into it as a test and update.

egberts commented 4 years ago

Ummm, just one more thing to consider, if you have plans in the future to do multiple files, you should tack on a ‘canary’ subdirectory after that .well-known directory part.

But I highly suspect that won’t be the case.

carrotcypher commented 4 years ago

Ummm, just one more thing to consider, if you have plans in the future to do multiple files, you should tack on a ‘canary’ subdirectory after that .well-known directory part.

But I highly suspect that won’t be the case.

When you say multiple files, are you referring to having multiple canaries from the same entity? I envision them having just one canary.txt (or whatever extension), but later on either: (1) having a network of storage (IPFS? TahoeLAFS? OrbitDB?) to keep track of the canaries; and/or (2) some kind of canary history inside the canary itself to cryptographically prove the last (few?) states.

At the moment I'm keeping things as simple as I can but this PoC being worked on will be a good place to jump in with ideas and I welcome to help.

egberts commented 4 years ago

K.I.S.S. is good.

Just trying to ensure no pitfalls with forward thinking. Apparently, a web server by domain name should suffice ... for now.

carrotcypher commented 4 years ago

One problem I'm noticing from this is that if we keep the standard as it is now (with header/footer for easy cryptographic verification) then it's not valid JSON and basically just a text file that contains some JSON elements.

If we however move the cryptographic elements inside of the JSON array itself, I'm starting to notice that we would be complicating things exponentially at best, and at worst it becomes something that isn't regular JSON anymore anyway.

Some resources I've been reading:

https://stackoverflow.com/questions/4670494/how-to-cryptographically-hash-a-json-object https://latacora.micro.blog/2019/07/24/how-not-to.html https://github.com/jchris/canonical-json https://jose.readthedocs.io/en/latest/ https://ordina-jworks.github.io/security/2016/03/12/Digitally-signing-your-JSON-documents.html https://jwt.io/introduction/

At least for this iteration of the standard and PoC, I am opting to keep it as just a simple file akin to how GPG signatures look. If someone wants to jump in and help me understand how it could be done cleanly inside of JSON entirely, I am all ears and open to evolve the standard then.

At the moment there just doesn't seem a perfect choice to make.

egberts commented 4 years ago

Agreed. We can not insert comments into JSON (except as an object).

Perhaps the simplified INI configuration format (or Python script full of assignments, #headduck)

carrotcypher commented 4 years ago

@egberts what do you think about just collapsing all the data fields together into a single string and signing that?

example:

The base information is

{
    "DOMAIN": "www.example.com",
    "PUBKEY": "1GQH1cDFHLyq2KHHAqFTYMy9kWmMw9dsdR",
    "NEWPUBKEY": "1DEKZAMjiT1jUhWAh8RH6zwmQpuvtW6X3y",
    "PANICKEY": "1MKoEjCVyLcVrNZJEdxy4Utv4enNe5XNUn",
    "NEWPANICKEY": "",
    "VERSION": "0.1",
    "RELEASE": "2019-03-06T22:23:09.963",
    "EXPIRY": "2019-04-06T22:23:09.963",
    "FRESHNESS": "0000 0000 0000 0000 000e d66a e55a 308b 77e5 ff78 bc94 a69b 8c27 da81 9562 537b",
    "CODES": ["WAR", "GAG", "SUBP", "TRAP", "CEASE", "DURESS", "RAID", "SEIZE", "XCRED", "XOPERS", "SEPPU"]
}

and we add a field "SIGNED" which looks like

"SIGNED":"1HZwkjkeaoZfTSaJxDw6aKkxp45agDiEzNG5ggR44agGR90xngVZvpqGKfgcQUPWyQmY93BfPUnfKkFpV+u6NC3Q/Q8i6ID2dY2oGYW2gsgz6sfLJrUlVz4A="

and contains the signed data of the entire JSON body flattened to a single line

"DOMAIN": "www.example.com","PUBKEY": "1GQH1cDFHLyq2KHHAqFTYMy9kWmMw9dsdR","NEWPUBKEY": "1DEKZAMjiT1jUhWAh8RH6zwmQpuvtW6X3y","PANICKEY": "1MKoEjCVyLcVrNZJEdxy4Utv4enNe5XNUn","NEWPANICKEY": "","VERSION": "0.1","RELEASE": "2019-03-06T22:23:09.963","EXPIRY": "2019-04-06T22:23:09.963","FRESHNESS": "0000 0000 0000 0000 000e d66a e55a 308b 77e5 ff78 bc94 a69b 8c27 da81 9562 537b","CODES": ["WAR", "GAG", "SUBP", "TRAP", "CEASE", "DURESS", "RAID", "SEIZE", "XCRED", "XOPERS", "SEPPU"]

That way, to verify the canary you only need to flatten the data (with "SIGNED" omitted). As long as spaces are retained and the verification is always on the flattened data only, it should provide the functionality necessary.

carrotcypher commented 4 years ago

Perhaps for clarity as a standard it should even be:

{
    "CANARY": {
        "DOMAIN": "www.example.com",
        "PUBKEY": "1GQH1cDFHLyq2KHHAqFTYMy9kWmMw9dsdR",
        "NEWPUBKEY": "1DEKZAMjiT1jUhWAh8RH6zwmQpuvtW6X3y",
        "PANICKEY": "1MKoEjCVyLcVrNZJEdxy4Utv4enNe5XNUn",
        "NEWPANICKEY": "",
        "VERSION": "0.1",
        "RELEASE": "2019-03-06T22:23:09.963",
        "EXPIRY": "2019-04-06T22:23:09.963",
        "FRESHNESS": "0000 0000 0000 0000 000e d66a e55a 308b 77e5 ff78 bc94 a69b 8c27 da81 9562 537b",
        "CODES": ["WAR", "GAG", "SUBP", "TRAP", "CEASE", "DURESS", "RAID", "SEIZE", "XCRED", "XOPERS", "SEPPU"]
    },
    "SIGNED":"1HZwkjkeaoZfTSaJxDw6aKkxp45agDiEzNG5ggR44agGR90xngVZvpqGKfgcQUPWyQmY93BfPUnfKkFpV+u6NC3Q/Q8i6ID2dY2oGYW2gsgz6sfLJrUlVz4A="
}

That way, all that is needed is to detect the base array "CANARY" to know it is a canary, and to check if "SIGNED" is a valid signature of the flattened contents of "CANARY".

egberts commented 4 years ago

The only issue I could think of with self-enclosure of PGP keys inside a JSON is that it would require an API/toolkit to ensure proper and accurate text coverages for key signing to “SIGNED” value within the ever-shifting JSON fields to obtain its concise and recreateable key value.

Do we assign textual coverage (needed for key algorithmic computation by

If API is to be provided, which language(s) will it be available in and who provides?

Simply tacking on the PGP armor around the entire text file using a header line and its footer lines (containing its PGP key) by staying with the original text format (regardless of whether it’s JSON or not), several advantages are:

Overly simplified INI format should be interesting to investigate if there are benefits of section name (another issue)

Last note, the amouring method (of wrapping enclosed text file of any format type with a header/footer/key) should be firmly selected and established.

Such armouring has been long established by PGP/GnuPG.

Even latest Mozilla Thunderbird (e-mail client) has just recently built-in PGP armouring into its app after 12 years of waiting.

egberts commented 4 years ago

Dearmouring the canary file to reach the canary file is easy, consistent, and safe.

Fishing out the “SiGNED” from a JSON then figuring out which JSON format to use ... not so much.

carrotcypher commented 4 years ago

Just to clarify, it wouldn't be PGP/GPG but just ECDSA (ed25519 curve). As for the order, since the signature would be of the fields as they are laid out by the author, and no one will be reordering the JSON when loading it into the CLI, it shouldn't be an issue to guarantee the order of the fields is always the same when verifying.

As long as the CLI behaves the same for everyone the same way when squashing the JSON into a single line, I am not seeing that there'd be any discrepancies. Am I missing something?

egberts commented 4 years ago

Embedding a SIGNED field name and its value would require some overt declaration of where to start reading and stop reading the cipher block to obtain/verify such SIGNED value.

carrotcypher commented 4 years ago

The way I was thinking, the CLI would process the JSON by:

  1. grabbing the key to verify against from [CANARY][PUBKEY]
  2. taking the entire contents of [CANARY] and flattening it into a single line (let's called it FLAT_CANARY)
  3. confirming that the signature contained in [SIGNED] matches the integrity of FLAT_CANARY

I'll work on a PoC this month and post it to the client repo. If I'm missing something, I'll find out then! :)

egberts commented 4 years ago

Very Pythonese. Probably JavaScriptist too.

carrotcypher commented 3 years ago

The standard has been updated to solve the signing problem. Check it out! https://github.com/canarytail/standard/blob/master/README.md#format

egberts commented 1 year ago

Just to clarify, it wouldn't be PGP/GPG but just ECDSA (ed25519 curve).

Uh? By using an ECDSA, the keys would then become symmetric (unattributable and unchainable, yet, still verifiable; is this the design goal to somehow distribute the private key out-of-band un/securely that is needed for verification and validation process)?

GnuPGP/PGP is an asymmetric key management and has a verifiable, chainable, and attributable public key.

carrotcypher commented 1 year ago

Just to clarify, it wouldn't be PGP/GPG but just ECDSA (ed25519 curve).

Uh? By using an ECDSA, the keys would then become symmetric (unattributable and unchainable, yet, still verifiable; is this the design goal to somehow distribute the private key out-of-band un/securely that is needed for verification and validation process)?

GnuPGP/PGP is an asymmetric key management and has a verifiable, chainable, and attributable public key.

ECDSA is asymmetric, with a private and public key pair. The usecase here for canarytail is to allow for canary generators to generate a private key (for controlling and signing the canary), while sharing their public key for verification, in the simplest, least overhead way.

Adding GPG would of course add more features (like Web-of-trust) but also unnecessary overhead and compatibility issues.

snex commented 11 months ago

I have an example up on my personal homepage, https://home.xens.org. The code checks a default location of "canary/canary.json" but it will also read a custom location from a META tag.