search

bcgov / DITP-DevOps

Digital Identity and Trust Program Team's DevOps Documentation Repository
Apache License 2.0
2 stars 5 forks source link

Upgrade IDIM Issuers to ACA-Py v0.10.2 and Askar only images. #109

Closed WadeBarnes closed 9 months ago

WadeBarnes commented 10 months ago

Related task:

WadeBarnes commented 10 months ago

Response from /revocation/active-registry/XpgeQa93eZvGSZBZef3PHn%3A3%3ACL%3A28075%3APersonDEV/rotate for dev:

{
  "rev_reg_ids": [
    "XpgeQa93eZvGSZBZef3PHn:4:XpgeQa93eZvGSZBZef3PHn:3:CL:28075:PersonDEV:CL_ACCUM:d3f96e1c-951a-4a9b-9d31-6a6d921ee0c8",
    "XpgeQa93eZvGSZBZef3PHn:4:XpgeQa93eZvGSZBZef3PHn:3:CL:28075:PersonDEV:CL_ACCUM:1b6373a4-3200-4cbd-89c2-0984283e50f9"
  ]
}
WadeBarnes commented 10 months ago

Response from /revocation/active-registry/XpgeQa93eZvGSZBZef3PHn%3A3%3ACL%3A28075%3APersonDEV/rotate for sit:

{
  "rev_reg_ids": [
    "7xjfawcnyTUcduWVysLww5:4:7xjfawcnyTUcduWVysLww5:3:CL:28075:PersonSIT:CL_ACCUM:50205010-00ae-4e47-b513-4ded5df05727",
    "7xjfawcnyTUcduWVysLww5:4:7xjfawcnyTUcduWVysLww5:3:CL:28075:PersonSIT:CL_ACCUM:6b8710bb-79c8-4846-b257-378853336abf"
  ]
}
WadeBarnes commented 10 months ago

Response from /revocation/active-registry/XpgeQa93eZvGSZBZef3PHn%3A3%3ACL%3A28075%3APersonDEV/rotate for qa:

{
  "rev_reg_ids": [
    "KCxVC8GkKywjhWJnUfCmkW:4:KCxVC8GkKywjhWJnUfCmkW:3:CL:20:PersonQA:CL_ACCUM:b8ada6b0-c3c6-4e26-bb52-3b1aa9bef1b8",
    "KCxVC8GkKywjhWJnUfCmkW:4:KCxVC8GkKywjhWJnUfCmkW:3:CL:20:PersonQA:CL_ACCUM:921ca010-04e7-4667-b613-5037e0459551"
  ]
}
usingtechnology commented 10 months ago

do we know the states of the registries (that are failing)?

if they are state=="init" nothing happens, and if they are not "active", then a new registry isn't created.

WadeBarnes commented 10 months ago

Where would I get that info?

usingtechnology commented 10 months ago

Try: GET /revocation/registry/{rev_reg_id}

WadeBarnes commented 10 months ago

dev - both "state": "decommissioned":

sit - both "state": "decommissioned":

qa - both "state": "decommissioned":

usingtechnology commented 10 months ago

Sigh, ok so they can get set to decommissioned and not kick off an init new registry. What we really need to know is (and I don't know how we could now...) is the state before the rotate call. It has to be "active" when you call rotate to kick off the init process.

WadeBarnes commented 10 months ago

They were active before the rotate call. All were being actively used to issue and revoke credentials.

swcurran commented 10 months ago

Looks to me like the RevReg rotation didn’t work. I don’t see any new RevRegDefs on CandyScan Dev, where the DIDs seem to be — not on Dev or SIT. It looks like the RevRegs were decommissioned, but the new RevRegDefs were not created.

swcurran commented 10 months ago

Link for SIT: https://candyscan.idlab.org/txs/CANDY_DEV/domain?page=1&pageSize=50&filterTxNames=[%22REVOC_REG_DEF%22]&sortFromRecent=true&search=7xjfawcnyTUcduWVysLww5

swcurran commented 10 months ago

They were active before the rotate call. All were being actively used to issue and revoke credentials.

That is the purpose of the rotate — to decommission the ones that were active, and to create new ones that become the active one. However, it looks like the creation step didn’t work in this execution.

usingtechnology commented 10 months ago

Well, someone put in a PATCH /revocation/registry/{rev_reg_id}/set-state?state=active API... it;s either that or create a new registry but don't think that's what you want?

I suppose if it doesn't get back to issuing then it will be POST /revocation/create-registry but that seems heavy handed.

usingtechnology commented 10 months ago

They were active before the rotate call. All were being actively used to issue and revoke credentials.

That is the purpose of the rotate — to decommission the ones that were active, and to create new ones that become the active one. However, it looks like the creation step didn’t work in this execution.

Yeah, the only thing I can see is that kicking off the "init" process is dependent on notifications on the event bus... which is the same no matter when it's created but any chance of a message being missed? 🤷

usingtechnology commented 10 months ago

They were active before the rotate call. All were being actively used to issue and revoke credentials.

That is the purpose of the rotate — to decommission the ones that were active, and to create new ones that become the active one. However, it looks like the creation step didn’t work in this execution.

Just to be clear, it will mark other registries as "decommissioned" but will only create/init a new one if the current state is active. So just because it is marked decommissioned after the call doesn't mean that it was active before the call.

WadeBarnes commented 10 months ago

I confirmed the state by looking at the state in the prod environment, which has not been rotated yet. All of the environments were identical, and the state in prod is active. So I'd expect that new RevRegs would have been created.

swcurran commented 10 months ago

Agree — they must have been active.

WadeBarnes commented 10 months ago

So the fix in this case is to create new registries manually using /revocation/create-registry; correct?

usingtechnology commented 10 months ago

I confirmed the state by looking at the state in the prod environment, which has not been rotated yet. All of the environments were identical, and the state in prod is active. SO I'd expect that new RevRegs would have been created.

OK, that sounds good and narrows it down... so more along the lines of what i put above and a hiccup on the notification...?

            init = IssuerRevRegRecord.STATE_ACTIVE == rec.state
            await self._set_registry_status(
                rec.revoc_reg_id, IssuerRevRegRecord.STATE_DECOMMISSIONED, init
            )
        if (state in IssuerRevRegRecord.TERMINAL_STATES) and init:
            return await self.init_issuer_registry(
                registry.cred_def_id,
                registry.max_cred_num,
                registry.revoc_def_type,
            )

terminal states are full and decommissioned, so if you pass in either of those AND that you want to init (prev state was active) then kick off the init process - which uses notifications and event bus.

WadeBarnes commented 10 months ago

Sorry, not sure what all that means.

usingtechnology commented 10 months ago

So the fix in this case is to create new registries manually using /revocation/create-registry; correct?

I think so. POST body is something like this:

{
  "credential_definition_id": cred_def_id, 
  "max_cred_num": 1000
}
usingtechnology commented 10 months ago

Sorry, not sure what all that means.

just showing the key parts of the logic on rotate... if they were active then init was true and it should have kicked off the init_issuer_registry... since they were active and we didn't get a new registry was just looking at points of failure. first thing I would think of is the notification didn't get pushed or received that does the work.

WadeBarnes commented 10 months ago

Ok, that covers the fix for our current situation. Do you want a ticket in ACA-Py for tracking the permanent fix? Would the fact that these agents are using an endorser service have anything to do with the new RevRegs not being created? i.e. messages not getting to the endorser?

swcurran commented 10 months ago

I hate to add more work, but can we back up DEV so we (might) have another try if this doesn’t work the first time? I think this is the right thing as well.

I think so. POST body is something like this:

{
  "credential_definition_id": cred_def_id, 
  "max_cred_num": 1000
}
swcurran commented 10 months ago

Ok, that covers the fix for our current situation. Do you want a ticket in ACA-Py for tracking the permanent fix? Would the fact that these agents are using an endorser service have anything to do with the new RevRegs not being created? i.e. messages not getting to the endorser?

Yes — we need an ACA-Py ticket. Nice to try it in the demo as the easiest way to verify the simple process works. We can even try with an Endorser there, I think.

WadeBarnes commented 10 months ago

I'll create a ticket and link to this ticket for the details.

usingtechnology commented 10 months ago

can we issue a few then rotate again? see if it repeatedly fails on dev?

so can you tell if the call didn't get to the endorser? or if the call didn't get back from the endorser? A lot of workflow creating the registry.

WadeBarnes commented 10 months ago

I hate to add more work, but can we back up DEV so we (might) have another try if this doesn’t work the first time? I think this is the right thing as well.

I think so. POST body is something like this:

{
  "credential_definition_id": cred_def_id, 
  "max_cred_num": 1000
}

A backup is not going to help much if things get written tot he ledger.

swcurran commented 10 months ago

A backup is not going to help much if things get written tot he ledger.

Agree — but our issue now is that things are not being written to the ledger.

WadeBarnes commented 10 months ago

The call to /revocation/create-registry returned 200 success, but nothing was written to the ledger.

{
  "result": {
    "state": "generated",
    "created_at": "2023-09-06T20:39:57.992060Z",
    "updated_at": "2023-09-06T20:40:01.104513Z",
    "record_id": "64293046-0895-403a-b695-6b3698e5f11b",
    "cred_def_id": "XpgeQa93eZvGSZBZef3PHn:3:CL:28075:PersonDEV",
    "issuer_did": "XpgeQa93eZvGSZBZef3PHn",
    "max_cred_num": 1000,
    "revoc_def_type": "CL_ACCUM",
    "revoc_reg_id": "XpgeQa93eZvGSZBZef3PHn:4:XpgeQa93eZvGSZBZef3PHn:3:CL:28075:PersonDEV:CL_ACCUM:64293046-0895-403a-b695-6b3698e5f11b",
    "revoc_reg_def": {
      "ver": "1.0",
      "id": "XpgeQa93eZvGSZBZef3PHn:4:XpgeQa93eZvGSZBZef3PHn:3:CL:28075:PersonDEV:CL_ACCUM:64293046-0895-403a-b695-6b3698e5f11b",
      "revocDefType": "CL_ACCUM",
      "tag": "64293046-0895-403a-b695-6b3698e5f11b",
      "credDefId": "XpgeQa93eZvGSZBZef3PHn:3:CL:28075:PersonDEV",
      "value": {
        "issuanceType": "ISSUANCE_BY_DEFAULT",
        "maxCredNum": 1000,
        "publicKeys": {
          "accumKey": {
            "z": "1 17D79CA6368D14B5A477BD5DB055DB3F723CA9CADF3429D414139308EE6B29C4 1 17EE55FCE54A448DBC92D2B2B0B9729C6005DB50F7154835DF77C088CF857430 1 06039690C3FB51CF04F104EBA451666C8CA743EAE0E632038545F7DDCBC7D7A2 1 177B8400364AAF2834B2BB8B0C4C057D80E8D7D9E0AF1013886FD0CCF8868729 1 1074873C05E4C528BD0CF6A4B8A81F23713FB3D01171D3FE370086459EAC175D 1 0F82C33B64D964C2482A22FD235631532414936965BCD03EEF1269B3A84C0CCF 1 06A2AC1C1B091066721878E5CCD7EFC2D5C5FC3795F80EF51C0D1E8485A47A49 1 22818EF0840B2D8E4E8D0B99F9D38EB1B328C0EC82CDA654CF5E475DFF9C6D90 1 15F857FADF48A3AEF47FB4F86BA4F84E78DB9D2A814DCABC7B3F132A389C63F6 1 0CA5EBD112CC7D2A87B23A0909A207E179C0749947404CD06DFE22F4F19CF9BA 1 103081758CA8CC9165A1DB256F5FD3FB1F6141EF82E20FB7C20DF5C7FF445DE8 1 09BEB8DDAF65EDC6616278A42E3D7CBD0F7C69E2ACBA2FFFC2EBE33E3F617FF8"
          }
        },
        "tailsHash": "FZScQrwGQWUfecfkMshLuoLFiMfJJPaXGLCeeo68m6RW",
        "tailsLocation": "/home/aries/.indy_client/tails/.hopper/FZScQrwGQWUfecfkMshLuoLFiMfJJPaXGLCeeo68m6RW"
      }
    },
    "revoc_reg_entry": {
      "ver": "1.0",
      "value": {
        "accum": "1 0391D9E0252B0239E20B10E14B410A03261C3C7E55B649D0785336EE8DA9CF88 1 1699333CCCD80488F67124DE4556CE56D0E874BA26E466CBB324ECB0F3A19140 1 20885B0A7CBF4970D1D9E9CDBDE1C3D61FA8FD50649D6BC738642DE31E7407B6 1 069CEA4B2FB138FABC87F3A92CC858BF8F4BDC18866E46069B0FAB86D61E5F88 2 095E45DDF417D05FB10933FFC63D474548B7FFFF7888802F07FFFFFF7D07A8A8 1 0000000000000000000000000000000000000000000000000000000000000000"
      }
    },
    "tag": "64293046-0895-403a-b695-6b3698e5f11b",
    "tails_hash": "FZScQrwGQWUfecfkMshLuoLFiMfJJPaXGLCeeo68m6RW",
    "tails_local_path": "/home/aries/.indy_client/tails/XpgeQa93eZvGSZBZef3PHn:4:XpgeQa93eZvGSZBZef3PHn:3:CL:28075:PersonDEV:CL_ACCUM:64293046-0895-403a-b695-6b3698e5f11b/FZScQrwGQWUfecfkMshLuoLFiMfJJPaXGLCeeo68m6RW",
    "pending_pub": []
  }
}
swcurran commented 10 months ago

Interesting — indicates perhaps that the /rotate was not the issue — the quietly failing non-ledger write is the issue.

WadeBarnes commented 10 months ago

Here are all of the registries the IDIM dev agent thinks it's created:

Of the 5 associated with XpgeQa93eZvGSZBZef3PHn:3:CL:28075:PersonDEV, only the two original ones were written to the ledger.

{
  "rev_reg_ids": [
    "XpgeQa93eZvGSZBZef3PHn:4:XpgeQa93eZvGSZBZef3PHn:3:CL:25547:BC Person Credential (Dev):CL_ACCUM:467a8b7a-4b2a-46a7-8d3a-ee8195b685af",
    "XpgeQa93eZvGSZBZef3PHn:4:XpgeQa93eZvGSZBZef3PHn:3:CL:25361:verified_person:CL_ACCUM:48864d46-0196-493f-9a6b-2d19a382f052",
    "XpgeQa93eZvGSZBZef3PHn:4:XpgeQa93eZvGSZBZef3PHn:3:CL:25361:ca.bc.gov.iddev.verified_person:CL_ACCUM:6dca0943-f523-48c7-b507-9cb351977b34",
    "XpgeQa93eZvGSZBZef3PHn:4:XpgeQa93eZvGSZBZef3PHn:3:CL:25361:verified_person:CL_ACCUM:d4006036-e86a-4163-a0ee-f290bad870ba",
    "XpgeQa93eZvGSZBZef3PHn:4:XpgeQa93eZvGSZBZef3PHn:3:CL:25472:BC Person Credential (Dev):CL_ACCUM:ba29bf92-3207-45ca-bc77-0ac85148e8a1",
    "XpgeQa93eZvGSZBZef3PHn:4:XpgeQa93eZvGSZBZef3PHn:3:CL:28060:Endorser Protocol Test:CL_ACCUM:553da5da-f50c-4701-89a0-e6f6beb230e2",
    "XpgeQa93eZvGSZBZef3PHn:4:XpgeQa93eZvGSZBZef3PHn:3:CL:26908:Person (Dev):CL_ACCUM:492c1ab1-455f-4e5c-a722-de0379bd70f5",
    "XpgeQa93eZvGSZBZef3PHn:4:XpgeQa93eZvGSZBZef3PHn:3:CL:25547:BC Person Credential (Dev):CL_ACCUM:efe40df7-fd33-464d-92bd-de6c3621709e",
    "XpgeQa93eZvGSZBZef3PHn:4:XpgeQa93eZvGSZBZef3PHn:3:CL:25472:BC Person Credential (Dev):CL_ACCUM:363558b8-859a-48ec-add7-4c21054f894d",
    "XpgeQa93eZvGSZBZef3PHn:4:XpgeQa93eZvGSZBZef3PHn:3:CL:28075:PersonDEV:CL_ACCUM:1b6373a4-3200-4cbd-89c2-0984283e50f9",
    "XpgeQa93eZvGSZBZef3PHn:4:XpgeQa93eZvGSZBZef3PHn:3:CL:28075:PersonDEV:CL_ACCUM:a57e7adb-a44c-4ccb-9af1-b50ce77673be",
    "XpgeQa93eZvGSZBZef3PHn:4:XpgeQa93eZvGSZBZef3PHn:3:CL:28075:Person (DEV):CL_ACCUM:5d4ef958-7073-4d19-9307-9aa3816c78fe",
    "XpgeQa93eZvGSZBZef3PHn:4:XpgeQa93eZvGSZBZef3PHn:3:CL:28075:PersonDEV:CL_ACCUM:d3f96e1c-951a-4a9b-9d31-6a6d921ee0c8",
    "XpgeQa93eZvGSZBZef3PHn:4:XpgeQa93eZvGSZBZef3PHn:3:CL:25361:ca.bc.gov.iddev.verified_person:CL_ACCUM:e0ec320e-c096-4075-b3a5-264b677b7e90",
    "XpgeQa93eZvGSZBZef3PHn:4:XpgeQa93eZvGSZBZef3PHn:3:CL:26908:Person (Dev):CL_ACCUM:6186caed-4f00-4581-bd72-879a1e83f016",
    "XpgeQa93eZvGSZBZef3PHn:4:XpgeQa93eZvGSZBZef3PHn:3:CL:25472:default:CL_ACCUM:59418b82-7364-4460-9fa0-0fb994b9377b",
    "XpgeQa93eZvGSZBZef3PHn:4:XpgeQa93eZvGSZBZef3PHn:3:CL:28075:PersonDEV:CL_ACCUM:64293046-0895-403a-b695-6b3698e5f11b",
    "XpgeQa93eZvGSZBZef3PHn:4:XpgeQa93eZvGSZBZef3PHn:3:CL:25361:Person (DEV):CL_ACCUM:89638367-d483-4fb6-a123-e73babc36c92",
    "XpgeQa93eZvGSZBZef3PHn:4:XpgeQa93eZvGSZBZef3PHn:3:CL:28075:PersonDEV:CL_ACCUM:4a165d42-1215-41cd-b4dc-806855c37f1a",
    "XpgeQa93eZvGSZBZef3PHn:4:XpgeQa93eZvGSZBZef3PHn:3:CL:25361:verified_person2:CL_ACCUM:71ff92a6-d7ad-4ae0-b824-a6659d0db4ee",
    "XpgeQa93eZvGSZBZef3PHn:4:XpgeQa93eZvGSZBZef3PHn:3:CL:25472:BC Person Credential (Dev):CL_ACCUM:73b22f70-0dbe-4d69-9f99-75a5c15038de",
    "XpgeQa93eZvGSZBZef3PHn:4:XpgeQa93eZvGSZBZef3PHn:3:CL:25361:verified_person4:CL_ACCUM:5135a8d7-8389-4b7a-8791-a314cd869f57",
    "XpgeQa93eZvGSZBZef3PHn:4:XpgeQa93eZvGSZBZef3PHn:3:CL:25361:verified_person2:CL_ACCUM:33e66a7e-a925-44d9-9db1-ec5376ffc653",
    "XpgeQa93eZvGSZBZef3PHn:4:XpgeQa93eZvGSZBZef3PHn:3:CL:28075:Person (DEV):CL_ACCUM:a2bde153-c624-4548-9b1d-94a9ec71d14f",
    "XpgeQa93eZvGSZBZef3PHn:4:XpgeQa93eZvGSZBZef3PHn:3:CL:28060:Endorser Protocol Test:CL_ACCUM:c2e1c810-2e0b-4f19-93c8-fe28e1cfeae2",
    "XpgeQa93eZvGSZBZef3PHn:4:XpgeQa93eZvGSZBZef3PHn:3:CL:25361:verified_person3:CL_ACCUM:16319413-7dfb-4899-9457-89a1d198f583",
    "XpgeQa93eZvGSZBZef3PHn:4:XpgeQa93eZvGSZBZef3PHn:3:CL:25361:Person (DEV):CL_ACCUM:33ccad0e-8100-4ef7-b499-d5b60ea52101",
    "XpgeQa93eZvGSZBZef3PHn:4:XpgeQa93eZvGSZBZef3PHn:3:CL:25361:verified_person4:CL_ACCUM:c69f5eaa-1405-4762-a5e2-118f42d94c49",
    "XpgeQa93eZvGSZBZef3PHn:4:XpgeQa93eZvGSZBZef3PHn:3:CL:25472:default:CL_ACCUM:0fd6902f-2ffd-4fc2-a971-f06b52b82400"
  ]
}

States of the XpgeQa93eZvGSZBZef3PHn:3:CL:28075:PersonDEV:

Decommissioned:

{
  "rev_reg_ids": [
    "XpgeQa93eZvGSZBZef3PHn:4:XpgeQa93eZvGSZBZef3PHn:3:CL:28075:PersonDEV:CL_ACCUM:1b6373a4-3200-4cbd-89c2-0984283e50f9",
    "XpgeQa93eZvGSZBZef3PHn:4:XpgeQa93eZvGSZBZef3PHn:3:CL:28075:PersonDEV:CL_ACCUM:d3f96e1c-951a-4a9b-9d31-6a6d921ee0c8"
  ]
}

Posted:

{
  "rev_reg_ids": [
    "XpgeQa93eZvGSZBZef3PHn:4:XpgeQa93eZvGSZBZef3PHn:3:CL:28075:PersonDEV:CL_ACCUM:a57e7adb-a44c-4ccb-9af1-b50ce77673be",
    "XpgeQa93eZvGSZBZef3PHn:4:XpgeQa93eZvGSZBZef3PHn:3:CL:28075:PersonDEV:CL_ACCUM:4a165d42-1215-41cd-b4dc-806855c37f1a"
  ]
}

Generated:

{
  "rev_reg_ids": [
    "XpgeQa93eZvGSZBZef3PHn:4:XpgeQa93eZvGSZBZef3PHn:3:CL:28075:PersonDEV:CL_ACCUM:64293046-0895-403a-b695-6b3698e5f11b"
  ]
}
WadeBarnes commented 10 months ago

@usingtechnology, Can I leave this with you to investigate a little bit? If you need anything from me just ask.

swcurran commented 10 months ago

@usingtechnology, Can I leave this with you to investigate a little bit? If you need anything from me just ask.

@WadeBarnes — I would think this has to do with the deployment, since this works elsewhere. Endorser? I think the logs are going to be needed at minimum.

swcurran commented 10 months ago

I just realized what you meant with the list. So all of the other RevRegs are there — just the last three — two that were attempted to be created via the “rotate” and the one you just created manually.

WadeBarnes commented 10 months ago

I just realized what you meant with the list. So all of the other RevRegs are there — just the last three — two that were attempted to be created via the “rotate” and the one you just created manually.

Yes. All of the records this agent has written to the ledger have been created through the endorser service.

usingtechnology commented 10 months ago

@WadeBarnes - if you can... the logs from the endorser and this agent would/may be helpful.

WadeBarnes commented 10 months ago

I'm not seeing any communication with the level logging is set to. I'll have to increase the logging and try creating the registries.

WadeBarnes commented 10 months ago

I've updated https://github.com/bcgov/DITP-DevOps/issues/109#issuecomment-1709104949 with the list of RevRegs associated to XpgeQa93eZvGSZBZef3PHn:3:CL:28075:PersonDEV and their states.

I'm assuming the posted ones we'd expect to have been written to the ledger. With the generated one, what are our expectations regarding that one?

usingtechnology commented 10 months ago

generated means that it hasn't been written to the ledger, so we shouldn't see them on the ledger but they are on local storage. they should show up in GET /revocation/registries/created (init will not as they are always filtered out of that call)

flow is init -> generated -> send to ledger ->posted.

WadeBarnes commented 10 months ago

Yeah, I have a few more steps to follow to get it posted; https://github.com/hyperledger/aries-cloudagent-python/blob/main/docs/GettingStartedAriesDev/CredentialRevocation.md#manually-creating-revocation-registries

WadeBarnes commented 10 months ago

I ran PATCH /revocation/registry/{rev_reg_id} to update the tails server URL, and then POST /revocation/registry/{rev_reg_id}/definition.

Received the error

500 Internal Server Error
Server got itself in trouble

and now XpgeQa93eZvGSZBZef3PHn:4:XpgeQa93eZvGSZBZef3PHn:3:CL:28075:PersonDEV:CL_ACCUM:64293046-0895-403a-b695-6b3698e5f11b is in the posted state and not written to the ledger.

Error Logs: aca-py-error.log

The errors about ERROR Handler error with exception: Revocation registry XpgeQa93eZvGSZBZef3PHn:4:XpgeQa93eZvGSZBZef3PHn:3:CL:28075:PersonDEV:CL_ACCUM:64293046-0895-403a-b695-6b3698e5f11b in state posted: cannot publish definition. were caused by me rerunning POST /revocation/registry/{rev_reg_id}/definition.

No errors on the endorser side.

We've found the pydid.doc.doc.IdentifiedResourceMismatch: ID did:sov:4wvr7DC2XUtsHbpUpYaKvM#didcomm already found in Index and Items do not match messages are associated to the endorser connection.

Endorser Connection:

{
  "results": [
    {
      "state": "active",
      "created_at": "2022-10-17T11:50:29.708523Z",
      "updated_at": "2023-08-11T17:22:06.478391Z",
      "connection_id": "703baf94-1b0b-4400-aae6-e54d1648f06a",
      "my_did": "Je1zon4UCT2xtQUMBWQSTy",
      "their_did": "4wvr7DC2XUtsHbpUpYaKvM",
      "their_role": "inviter",
      "connection_protocol": "didexchange/1.0",
      "rfc23_state": "completed",
      "request_id": "ddeb1487-2d78-460b-b9ef-e96a4e5f7a1f",
      "routing_state": "none",
      "accept": "manual",
      "invitation_mode": "once",
      "alias": "Endorser",
      "their_public_did": "85DZerr49BLCuG5wWyDviy"
    }
  ]
}
WadeBarnes commented 10 months ago

Not sure where to go from here. Let me know if you've got any ideas.

usingtechnology commented 10 months ago

Weird. THere is this in the error log: pydid.doc.doc.IdentifiedResourceMismatch: ID did:sov:4wvr7DC2XUtsHbpUpYaKvM#didcomm already found in Index and Items do not match

You've confirmed it's not the endorser, but may explain why nothing was getting out. For whatever reason this duplicate is in the pydid did docs and failing sending outbound messages.

  File "/home/aries/.local/lib/python3.9/site-packages/aries_cloudagent/revocation/routes.py", line 1199, in send_rev_reg_def
    await outbound_handler(transaction_request, connection_id=connection_id)
  File "/home/aries/.local/lib/python3.9/site-packages/aries_cloudagent/messaging/responder.py", line 101, in send
    return await self.send_outbound(
  File "/home/aries/.local/lib/python3.9/site-packages/aries_cloudagent/admin/server.py", line 147, in send_outbound
    return await self._send(profile, message)
  File "/home/aries/.local/lib/python3.9/site-packages/aries_cloudagent/core/conductor.py", line 637, in outbound_message_router
    status: OutboundSendStatus = await self._outbound_message_router(
  File "/home/aries/.local/lib/python3.9/site-packages/aries_cloudagent/core/conductor.py", line 665, in _outbound_message_router
    return await self.queue_outbound(profile, outbound, inbound)
  File "/home/aries/.local/lib/python3.9/site-packages/aries_cloudagent/core/conductor.py", line 697, in queue_outbound
    outbound.target_list = await self.dispatcher.run_task(
  File "/usr/local/lib/python3.9/asyncio/futures.py", line 284, in __await__
    yield self  # This tells Task to wait for completion.
  File "/usr/local/lib/python3.9/asyncio/tasks.py", line 328, in __wakeup
    future.result()
  File "/usr/local/lib/python3.9/asyncio/futures.py", line 201, in result
    raise self._exception
  File "/usr/local/lib/python3.9/asyncio/tasks.py", line 256, in __step
    result = coro.send(None)
  File "/home/aries/.local/lib/python3.9/site-packages/aries_cloudagent/connections/base_manager.py", line 609, in get_connection_targets
    targets = await self.fetch_connection_targets(connection)
  File "/home/aries/.local/lib/python3.9/site-packages/aries_cloudagent/connections/base_manager.py", line 570, in fetch_connection_targets
    return await self.resolve_connection_targets(
  File "/home/aries/.local/lib/python3.9/site-packages/aries_cloudagent/connections/base_manager.py", line 362, in resolve_connection_targets
    doc, didcomm_services = await self.resolve_didcomm_services(did)
  File "/home/aries/.local/lib/python3.9/site-packages/aries_cloudagent/connections/base_manager.py", line 269, in resolve_didcomm_services
usingtechnology commented 10 months ago

so this agent is maybe not going to be able to send any outbound messages (as least when there is did lookup involved?)

may have to tag old school experts on how to fix the resolve_didcomm_services?

WadeBarnes commented 10 months ago

Weird. THere is this in the error log: pydid.doc.doc.IdentifiedResourceMismatch: ID did:sov:4wvr7DC2XUtsHbpUpYaKvM#didcomm already found in Index and Items do not match

You've confirmed it's not the endorser, but may explain why nothing was getting out. For whatever reason this duplicate is in the pydid did docs and failing sending outbound messages.

It's not the endorser's public DID, but I did look further and it is the peer DID associated to the endorser connection. Sorry for the confusion. I added the info to https://github.com/bcgov/DITP-DevOps/issues/109#issuecomment-1709223208

swcurran commented 10 months ago

This might be a 0.10.1 issue if it is DID resolution. Wade — can you tell if the Endorser was ever contacted? Or the ledger? Seems not.

Jason — I assume the /rotate and the action that Wade DID should have gone through both the generated and posted steps? E.g. there is not two actions that have to be done by the controller? And if there are two, why there was no error surfaced in the operation. I guess they are asynch, but there still should have been a notification of an error, I would think.

usingtechnology commented 10 months ago

for the rotate and create, there may be errors in the logs but nothing would get surfaced because they are done via event/notification handlers. my guess (without the logs) is the same type of error happened when it finally sent something out the outbound queue to deliver to the endorser.

swcurran commented 10 months ago

@dbluhm — could you pretty please take a look at the logs a few comments above? This is an update to 0.10.1 and the connection to the endorser seems not to be working. Could this be a did resolution issue?

dbluhm commented 10 months ago

@WadeBarnes I've downloaded and scanned through the error log from the comment above. What is the result of GET /resolver/resolve/did:sov:4wvr7DC2XUtsHbpUpYaKvM?