Cleanup old JWT tables in 5.0.1+

[maxtropets]

TL;DR

We kept Legacy:: tables for JWT in https://github.com/microsoft/CCF/pull/6175, for reasons explained here. This can be safely removed after ensuring old tables are no longer used anywhere in the system.

---

In CCF we periodically fetch/store key certificates for the configured issuer.

Before 5.0.x we stored the needed info in this tables

• public_signing_keys (here)
• jwt.public_signing_key_issuer (here)

After 5.x.x, we only put the newly fetched one new table:

• public_signing_keys_metadata (here)

After upgrading 4.x.x - > 5.x.x the new code will store JWT certificates in the new tables, but will read from both new and old tables in order. Old tables are used if the keys haven’t been fetched yet by any of the new nodes.

This task is to support further upgrading from the first 5.x.x to the 5.x.y with a proper clean-up of the old tables. It's two-fold:

• Dev side
  • new code with deleted references to Legacy:: tables
  • new custom action JS which will drop old KVs. Example

    // actions.js
    ...
    [
    "remove_old_jwt_tables",
    new Action(
        function (args) {},
        function (args) {
            ccf.kv["public:ccf.gov.jwt.public_signing_keys_metadata"].clear();
        },
    ),
    ],

• Operator side
  • verify the new table has been populated
  • We may grep a brand new snapshot (or force-trigger one) and grep it with read_ledger.py
  • Alternatively, can just call set_jwt_issuer (or expose and call a private jwt_keys/refresh endpoint) to unconditionally reset the keys instead
  • propose new action to prune the old tables
  • To be created as a dev-part
  • verify old tables have been deleted
  • We may add triggerSnapshot to a proposal so we get a fresh snapshot to grep with read_ledger.py and check there's no entries in the old tables

[achamayou]

Checking before and after can be done with read_ledger.py against snapshots:

$ read_ledger.py -s workspace/app_space_js_e2e_logging_cft_0/0.snapshots/snapshot_987_992.committed -t public:ccf.gov.jwt.public_signing_keys_metadata
Reading snapshot from workspace/app_space_js_e2e_logging_cft_0/0.snapshots/snapshot_987_992.committed (committed)
  txid 0.987 (41 public tables) [144658 bytes]
    table "public:ccf.gov.jwt.public_signing_keys_metadata" (1 write):
      95254152-e9dd-465a-b4cf-7893ed2e1cf0:
        [
          {
            "cert": "MIICrDCCAZSgAwIBAgIUP1Ubyv8C+h/tam4a9o3lfjfiTUAwDQYJKoZIhvcNAQELBQAwEDEOMAwGA1UEAwwFZHVtbXkwHhcNMjQwNjI0MTM0NjI4WhcNMjQwNzA0MTM0NjI4WjAQMQ4wDAYDVQQDDAVkdW1teTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMChEwD6t6rHMkytndUAFCXb2P2KManI1EtDSCkeABANrqJdTYs1rBJifb1Z4f+Q6UTKWRYoRlzymM+2d/bmjWmsdk0gPWj+G33Uo8Kwx3343c0l1JrfFCY7eWcYv/eXVHLuu+V7+y6B1iYIX89+rgJ2Wrwfef8buIzXIR5AClb4Q2VJdBfflba4bOGk44ZLsTKIn5ZM5mxi/ApAU/yMipIwfPdHxfFsbhWcwIVVU1uwSbaQZ2rw6vm77EVn0gjF6cC9rgTXMlpWzgHJT1jrkvlNZJanwBSnqa5JzTmEdRkU6LNGBZZOIEbaujYjRTkaSVdxCLehWO/CN7ZTRYmnH7kCAwEAATANBgkqhkiG9w0BAQsFAAOCAQEAGA/g8OMP3WAYZST+rMOoIqfdmV335YZ3zzu12gk8MIb9dY+lSdcOWIB1VXG0TwwWqLQUqlGnoHVG5YMDWK+rQ7NB9zzXoaXRcJda1aiMsdSeT4YiB+/AO8BrRkGrN35AE4YAXZl/wa5ICshKWQTQY2deDrL4WkNAdRlFcFwUwDaEoqU5f/6FSpoKuzm0QCk7B/TWj//lc2u1M/Mxp3Q01OkKqS05/q2wR+bklm5EFugAidL8mvokyTgL9OLOC9IZCC+O4woWtH9BmD/n9AmaRFj+5lM2IpjRRxVpqqEDkF8zZeLSK78n+VyT/t9cS/k79yh4Fm/Bz0C2ABFbED/J+w==",
            "constraint": "https://example.issuer",
            "issuer": "https://example.issuer"
          }
        ]

A good idea may be to add a trigger snapshot action to the proposal for the removal, to get a fresh snapshot, post-removal.

[sidmore]

I am going to execute on this task....seems straight forward mostly deleting references to old JWT tables in code and adding a new Proposal for removing tables from KV