Multitenancy and Endorsement

[Executioner1939]

Hi all,

I have some questions regarding the setup of my Multitenancy Node with a separate Endorser Node.

I feel as though I am going in circles and am stuck in a configuration nightmare so to speak and think I have just confused myself to the point that I need to ask for some clarifications on the setup.

I feel I am missing something as I can't get my Sub wallets (hereafter referred to simply as Tenants) to create the Schemas and associated Credential Definitions.

I have already read the following:

• 1657

• 983

• 1460

My takeaway from the above is to use a separate instance of ACA-py to run as an Endorser and do the Endorsement for transactions for my Tenants.

I am using the following version of ACA-py, denoted by this docker image, I am also using askar as the wallet type.

• ghcr.io/hyperledger/aries-cloudagent-python:py3.12-1.0.0

I have also read the documentation here:

• https://aca-py.org/main/features/Multitenancy/

I will detail below what I have done and where I am currently at.

I am using the von-network for testing, I have it running on a VM in GCP. I created a DID and Wallet for it following this guide:

• https://github.com/bcgov/von-network/blob/main/docs/Indy-CLI.md

This gives me a Public DID, Verkey, and Seed. I then registered, via the UI, the DID as an Endorser on the von-network.

I then have one instance of ACA-py running as my "Endorser" with the following startup parameters (db parameters excluded):

--admin 0.0.0.0 11000 \
--admin-insecure-mode false \
--webhook-url <url> \
--auto-respond-credential-proposal \
--auto-store-credential \
--auto-respond-presentation-request \
--auto-respond-presentation-proposal \
--auto-respond-credential-request \
--auto-respond-credential-offer \
--auto-accept-requests \
--auto-accept-invites \
--auto-respond-messages \
--debug-connections \
--debug-credentials \
--endpoint <url> \
--genesis-url <von-nertwork-url> \
--accept-taa on_file 1.0 \
--log-level debug \
--emit-new-didcomm-mime-type \
--emit-new-didcomm-prefix \
--exch-use-unencrypted-tags \
--tails-server-base-url <url> \
--auto-provision \
--inbound-transport http 0.0.0.0 8000 \
--outbound-transport http \
--image-url <image> \
--wallet-type askar \
--wallet-storage-type postgres_storage

Then I have my Multitenant Agent running with these parameters (db parameters excluded):

--admin 0.0.0.0 11000 \
--admin-insecure-mode false \
--webhook-url <url> \
--auto-respond-credential-proposal \
--auto-store-credential \
--auto-respond-presentation-request \
--auto-respond-presentation-proposal \
--auto-respond-credential-request \
--auto-respond-credential-offer \
--auto-accept-requests \
--auto-accept-invites \
--auto-respond-messages \
--debug-connections \
--debug-credentials \
--endpoint <url> \
--genesis-url <von-nertwork-url> \
--accept-taa on_file 1.0 \
--log-level debug \
--emit-new-didcomm-mime-type \
--emit-new-didcomm-prefix \
--exch-use-unencrypted-tags \
--tails-server-base-url <url> \
--auto-provision \
--inbound-transport http 0.0.0.0 8000 \
--outbound-transport http \
--image-url <image> \
--wallet-type askar \
--wallet-storage-type postgres_storage
--multitenant
--multitenant-admin true
--jwt-secret <secret>

If someone can help me strip down the arguments to the bare minimum to get this working as well that would help.

Now, I have a bootstrap script, this is used to:

• Create a new Tenant
• Create a DID for the Tenant
• Write the Tenant DID to the Ledger (I call the Endorser to do this)
• Create a static connection between the Tenant and the Endorser
• Set the Tenant DID as Public

This can be seen below (Please note, I have summarized my code into a small single function to convey the steps I am following)

async def create_subwallet_with_setup(
    multitenant_client: AcaPyClient,
    endorser_client: AcaPyClient,
    wallet_name: str,
    endorser_info: Dict[str, Any]
) -> Tuple[str, str, str]:
    try:
        # Step 1: Create a new subwallet
        response = await multitenant_client.multitenancy.create_wallet(
            body=CreateWalletRequest(
                wallet_name=wallet_name,
                wallet_key=f"{wallet_name}_key",
                wallet_type="askar",
                label=wallet_name,
                wallet_webhook_urls=["<endorser-url>"]
            )
        )
        token = response.token

        # Step 2: Create a client for the new subwallet
        async with managed_sub_client(
            base_url=multitenant_client.configuration.host,
            api_key=multitenant_client.api_key,
            token=token
        ) as subwallet_client:
            # Step 3: Create a new DID for the subwallet
            create_did_response = await subwallet_client.wallet.create_did(DIDCreate())
            did_info = create_did_response.result

            # Step 4: Register the DID on the ledger using the endorser
            await endorser_client.ledger.register_nym(
                did=did_info.did,
                verkey=did_info.verkey,
                alias=None
            )

            # Step 5: Establish a static connection with the endorser
            static_conn_request = ConnectionStaticRequest(
                alias=f"{wallet_name}-to-endorser",
                their_did=endorser_info['did'],
                their_endpoint=endorser_info['endpoint'],
                their_verkey=endorser_info['verkey']
            )
            static_conn_result = await subwallet_client.connection.create_static_connection(
                body=static_conn_request
            )
            connection_id = static_conn_result.record.connection_id

            # Step 6: Set the Endorser Role
            await endorser_client.endorse_transaction.set_endorser_role(
                conn_id=connection_id,
                transaction_my_job="TRANSACTION_AUTHOR"
            )

            # Step 7: Set the Endorser Info via metadata
            endorser_info_metadata = {
                "endorser_did": endorser_info['did'],
                "verkey": endorser_info['verkey'],
                "endpoint": endorser_info['endpoint'],
                "name": "Endorser"
            }
            metadata_request = ConnectionMetadataSetRequest(metadata={"endorser_info": endorser_info_metadata})
            await subwallet_client.connection.set_metadata(connection_id, body=metadata_request)

            # Step 8: Set the DID as public for this subwallet
            await subwallet_client.wallet.set_public_did(did_info.did)

        return token, did_info.did, connection_id
    except Exception as e:
        logger.error(f"Error creating subwallet '{wallet_name}': {str(e)}")
        raise

Questions:

1. I am unsure about points 5, 6, and 6. I am not sure if I am doing this correctly.
2. I am unsure if I am supposed to be creating this connection via the static connection endpoint.

When I run the code, it successfully creates the Tenant, DID, Writes it to the Ledger, creates the connection, sets the metadata and sets the DID as public as be seen from my logs:

2024-09-22 14:23:50,505 - INFO - Created subwallet 'Test' with token: <token>
2024-09-22 14:23:51,271 - INFO - Created new DID in subwallet: <did>
2024-09-22 14:23:52,435 - INFO - Registered DID <did> on the ledger using endorser client
2024-09-22 14:23:53,016 - INFO - Established static connection. Connection ID: 0611db15-9488-47df-8af1-64228c6b99c6
2024-09-22 14:23:53,237 - INFO - Connection 0611db15-9488-47df-8af1-64228c6b99c6 is active.

But when I try to perform actions against my Subwallet I am running into issues, I shall just show the one method I use to create a Schema, I have a feeling this approach is completely wrong:

async def create_and_endorse_schema(
    subwallet_client: AcaPyClient,
    endorser_client: AcaPyClient,
    schema_name: str,
    schema_version: str,
    attributes: List[str],
    connection_id: str
) -> SchemaSendResult:
    try:
        # Step 1: Publish the schema, requesting transaction creation for the endorser
        schema_request = SchemaSendRequest(
            schema_name=schema_name,
            schema_version=schema_version,
            attributes=attributes
        )
        result = await subwallet_client.schema.publish_schema(
            conn_id=connection_id,
            create_transaction_for_endorser=True,
            body=schema_request
        )

        if result.txn is not None:
            transaction_id = result.txn.transaction_id

            # Step 3: Endorse the transaction using the endorser client
            endorsed_txn = await endorser_client.endorse_transaction.endorse_transaction(
                tran_id=transaction_id
            )

            # Step 4: Write the endorsed transaction to the ledger
            written_txn = await endorser_client.endorse_transaction.write_transaction(
                tran_id=endorsed_txn.transaction_id
            )

            # Step 5: Fetch and return the schema details
            schema = await subwallet_client.schema.get_schema(schema_id=written_txn.schema_id)
            return schema
        else:
            raise Exception("Unexpected result: Transaction data not returned.")
    except Exception as e:
        logger.error(f"Error creating schema '{schema_name}': {str(e)}")
        raise

This always resulted in a Transaction Not Found error.

NB: Please also note, the Multitenant Agent and Endorser Agent are using separate databases, I am not sure if this should be the case.

I then started looking at the Endorser CLI arguments and amended my Endorser Agent to include:

--endorser-protocol-role endorser
--endorser-endorse-with-did <endorser-public-did>
--auto-endorse-transactions

and my Multitenant Agent to include:

--endorser-protocol-role author
--endorser-public-did <endorser-public-did>
--auto-request-endorsement
--auto-write-transactions
--auto-create-revocation-transactions
--auto-promote-author-did

Now when I run my script, it fails with the following error:

HTTP response body: 400: No endorser connection found

I would like some guidance if anyone has some time as I feel I took a wrong turn and wound up in an abandoned mine full of confusion and despair.

I will need to document this entire setup at work and would be very willing to contribute it back to the ACA-py documentation once done.

Please let me know if anything is unclear or if I have simply misunderstood something basic.

[swcurran]

@esune @WadeBarnes — could one of help out (or find someone on the BC team) to help out with this question. It sounds like the answer is very close, and we’ve definitely had this going for some time.

[esune]

@Executioner1939 thanks for the very through details, this helps understand where you are at.

The startup scripts you are using (with the added parameters described at the bottom of the issue) seem correct to me, and it looks like you have your endorser agent working correctly.

I suspect your issue is related to two things:

• you are creating a static connection (which if I am honest, I am not sure what use-case targets) rather than a "normal" connection.
• your DID is endorsed/written to the ledger because you are doing it directly via the endorser, rather than as a transaction to be endorsed.

What I would try is the following:

• Use /didexchange/create-request or /out-of-band/create-invitation to create a connection invitation and accept it before you do anything else (after provisioning the endorser and author agents, of course).
• Make sure the connection becomes active (you have the auto-accept flags set in your startup, so it should happen automatically).
• Set the connection metadata on the author end as you're already doing. You may also need to set transaction_their_job="TRANSACTION_ENDORSER" on the author agent side.
• Create new requests for the endorser for public DID, and schema/creddef afterwards.

With the auto flags turned on, you should see the transaction going to the endorser, getting processed and sent back to the author. Depending on what you choose when you set-up the request, the endorser may write the transaction for you OR might send a signed transaction back for your author to write/publish.

I think this might be it. Give it a try and, if it doesn't work, we'll dig deeper.

[Executioner1939]

Hi @esune and @swcurran thank you for the reply! I'll tackle this tomorrow and provide feedback!