New tenant added in an existing network is not getting identified from other nodes in the network

[Purbaja]

I have setup a network enabling multitenancy with Quorum 21.7.1 and Tessera 21.7.2 integrated with hashicorp vault(docker image - quorumengineering/tessera:hashicorp-21.7.2)

Initially each node had one tenant. Some transactions were performed. Then in the existing network I tried adding one new tenant in Node1.

The steps I followed -

1. Created new key pair in the hashicorp vault
2. Updated the tessera config jason like :

"keys": { "hashicorpKeyVaultConfig": { "url": "http://x.x.x.x:8200" }, "keyData": [ { "hashicorpVaultSecretEngineName": "xxx", "hashicorpVaultSecretName": "xxx-tessera-key", "hashicorpVaultPrivateKeyId": "privateKey", "hashicorpVaultPublicKeyId": "publicKey" }, { "hashicorpVaultSecretEngineName": "xxx", "hashicorpVaultSecretName": "yyy-tessera-key", "hashicorpVaultPrivateKeyId": "privateKey", "hashicorpVaultPublicKeyId": "publicKey" } ] }, "alwaysSendTo": [], "features": { "enableMultiplePrivateStates": "true" }, "residentGroups":[ { "name":"private", "description":"default privacy group", "members":["XXX"] }, { "name":"PS1", "description":"Privacy Group 1", "members":["YYY"] } ]

3. restarted tessera and quorum in Node1.
4. Initiated private transaction from Node2 keeping the new tenant of Node 1(i.e. PS1 resident group) as party of the txn. But got error in contract mining - Error: Non-200 status code: &{Status:404 Not Found StatusCode:404 Proto:HTTP/1.1 ProtoMajor:1 ProtoMinor:1 Header:map[Content-Length:[73] Content-Type:[text/plain] Date:[Fri, 12 Nov 2021 16:22:58 GMT] Server:[Jetty(9.4.25.v20191220)]] Body:0xc01dab0e00 ContentLength:73 TransferEncoding:[] Close:false Uncompressed:false Trailer:map[] Request:0xc01d7c1600 TLS:} undefined undefined

Error in Tessera log -

2021-11-12 12:51:34.443 [qtp751413576-33] INFO c.q.tessera.q2t.TransactionResource - Enter Request : POST : /sendraw 2021-11-12 12:51:34.474 [qtp751413576-33] INFO c.q.t.d.EncryptedTransactionDAOImpl - Stored transaction sswThPBnMMW+e9PSRVF2g6YpjRiRxrLFgBhLVYAS0KjpyItrsJ9aeTzDGiz9naUWRnzgCm6kRJaWf1bHc+7tqg== 2021-11-12 12:51:34.475 [qtp751413576-33] WARN c.q.t.a.e.KeyNotFoundExceptionMapper - Recipient not found for key: JVBix+8O4Ch7xi7j7fa372iC7Ssp3Op7LnT37Ft5Rms= 2021-11-12 12:51:34.475 [qtp751413576-33] INFO c.q.tessera.q2t.TransactionResource - Exit Request : POST : /sendraw 2021-11-12 12:51:34.475 [qtp751413576-33] INFO c.q.tessera.q2t.TransactionResource - Response for sendraw : 404 Not Found

[subhasisbanik]

Hi @achraf17 , could you please help here?

[antonydenyer]

I wanted to confirm a couple of things:

1. you updated all of the instances of tessera with the same config files?
2. You restarted all tessera nodes after changing the files.

The error is stating that Node2 can't talk to Node1 or more specifically it doesn't know what to do with JVBix+8O4Ch7xi7j7fa372iC7Ssp3Op7LnT37Ft5Rms=

[Purbaja]

@antonydenyer I updated the tessera config for Node1 only(added one new tenant) and hence restarted the tessera and quorum in Node1 only.

[antonydenyer]

That would be it then, you need to update the other enclaves. If this doesn't work please feel free to re-open.

[Purbaja]

@antonydenyer For every time a new tenant gets added in Node1, should we have to update other Nodes' tessera config? would the new tenant information not propagate to other nodes?

[Purbaja]

@antonydenyer This is working if I restart the Tessera service in Node2 also along with Node1. But my question is if new tenant gets added in Node1, then why should I have to restart Tessera of Node2 ?? Shouldn't it be propagated automatically as peer discovery is enabled in the network.

[antonydenyer]

It's a little grey! I suspect what is happening is that the keys aren't refreshed. This is being looked at on https://github.com/ConsenSys/tessera/issues/1386 - if there's an issue it'll be clarified there.

[macfarla]

I haven't found any anomalies with peer discovery and propagation of keys between Tessera nodes - with the scenario you described ie adding a new key to Node1 and restarting Node1 only, I get the same list from all Tessera nodes for partyinfo/keys endpoint. Note however that it does take some time for the peer discovery and key propagation to occur (in my environment this is in the order of seconds, however depending on network configuration and latency, this could take longer).

Couple of things to try:

• can you call the partyinfo/keys third-party endpoint on Tessera Nodes 1 and 2 and compare the results after the new key is added: eg curl -X GET http:///partyinfo/keys
• if the results differ, wait a minute and check again?

[Purbaja]

Hi @macfarla , I checked with the above command. Response from Node1 : curl -X GET http://x.x.x.x:9080/partyinfo/keys {"keys":[{"key":"G27/Rz02qexxYTI7h54W9Ebzoz3LgFppBhLSWbzgxCA="},{"key":"swAqIm6pDee0Y2mTEH7wmFPFPoK7QQL3WdZzXBVeASQ="}]} ***** this keys are of default tenant and Tenant1 of Node1

Response from Node2: curl -X GET http://y.y.y.y:9080/partyinfo/keys {"keys":[{"key":"O/ruD6wozXhv8FNg5WTlDDUY4LA9p/StZLzTFeGajW8="}]} *** this keys are of default tenant of Node2

[Purbaja]

@macfarla any comment on the above output I shared?

[macfarla]

@Purbaja I'm re-reading your comments above.

• Are you saying that Node1 doesn't get any keys from node2 and vice versa? ie it's not just the newly added key that is missing?
• After node1 restarts, do nodes 1 and 2 peer with each other?

[macfarla]

You can check what are the node's peers by this API call curl -X GET http://peer.to.peer.url:port/partyinfo https://consensys.github.io/tessera/#operation/getPartyInfo

[macfarla]

@Purbaja something else to check - does the node you're restarting have a list of peers configured - ie node2 won't automatically find node1, it needs to be node1 that searches for node2 when it starts up

[Purbaja]

@MadelineMurray , Node1 and Node2 were already peer and transaction was happening between them. Then i added one new Tenant in Node1 and restarted only Node1. But Node2 was not able to recognize the new tenant of Node1. After i restarted Node 2 it was able to recognize the new tenant of Node1.

And with the partyInfo command I get the below output always- Response from Node1 : curl -X GET http://x.x.x.x:9080/partyinfo/keys {"keys":[{"key":"G27/Rz02qexxYTI7h54W9Ebzoz3LgFppBhLSWbzgxCA="},{"key":"swAqIm6pDee0Y2mTEH7wmFPFPoK7QQL3WdZzXBVeASQ="}]} ***** this keys are of default tenant and Tenant1 of Node1

Response from Node2: curl -X GET http://y.y.y.y:9080/partyinfo/keys {"keys":[{"key":"O/ruD6wozXhv8FNg5WTlDDUY4LA9p/StZLzTFeGajW8="}]} *** this keys are of default tenant of Node2

[macfarla]

@Purbaja can you share the peer section of the config for nodes 1 and 2?

[antonydenyer]

Hi @Purbaja

I've been trying to reproduce the problem but to no avail.

Using the https://github.com/ConsenSys/quorum-examples/blob/master/docker-compose-4nodes-mt.yml I remove tm7.key from keyData and residentGroups.

Start the network using docker-compose -f docker-compose-4nodes-mt.yml up

Attempt to perform a private transaction from node2 with a privateFrom tm2 and privateFor tm7.

    private static final String TESSERA2_PUBLIC_KEY = "QfeDAys9MPDs2XHExtc84jKGHxZg/aj52DTh0vtA3Xc=";

    private static final String TESSERA7_PUBLIC_KEY = "ROAZBWtSacxXQrOe3FGAqJDyJjFePR5ce4TSIzmJ0Bc=";

    public static void main(String[] args) throws Exception {
        Quorum quorum = Quorum.build(new HttpService("http://localhost:22001"));
        EnclaveService enclaveService = new EnclaveService("http://localhost", 9082, new OkHttpClient());
        Enclave enclave = new Tessera(enclaveService, quorum);

        Credentials credentials = WalletUtils.loadCredentials("", "examples/7nodes/keys/key1");

        QuorumTransactionManager qrtxm = new QuorumTransactionManager(quorum,
                credentials,
                TESSERA2_PUBLIC_KEY,
                Arrays.asList(TESSERA7_PUBLIC_KEY),
                enclave,
                30,
                1000);

        SimpleStorage ssContract = SimpleStorage.deploy(quorum,
                qrtxm,
                BigInteger.valueOf(0),
                BigInteger.valueOf(4300000),
                BigInteger.valueOf(42)).send();

        System.out.println("Contract address:" + ssContract.getContractAddress());
        System.out.println(ssContract.getTransactionReceipt());
    }
}

Get the expected response from node2.

404 status: Recipient not found for key: ROAZBWtSacxXQrOe3FGAqJDyJjFePR5ce4TSIzmJ0Bc=

Add tm7 keys to tessera config on txmanager1 only (both keyData and resident group)

restart txmanager1

Send the private transaction again using node2 with a successful result.

The likely candidates for the problem are a misconfiguration in the keys or as @macfarla was suggesting a misconfiguration in the list of peers. Unless we can reliably reproduce the problem on our side there's not much we can do.

[Purbaja]

Hi @MadelineMurray, @antonydenyer please find below my peer config in Tessera-config.json for Node-1: Please note we are using docker image of 'quorumengineering/tessera:hashicorp-22.1.0' for Tessera.

{ "useWhiteList": false, "jdbc": { "username": "tessera", "password": "XXX", "url": "jdbc:mysql://10.x.x.x:3306/tessera", "autoCreateTables": true }, "serverConfigs":[ { "app":"ThirdParty", "enabled": true, "serverAddress": "http://10.x.x.x:9080", "bindingAddress": "http://0.0.0.0:9080", "communicationType" : "REST" }, { "app":"Q2T", "enabled": true, "serverAddress": "unix:$${DDIR}/tm.ipc", "communicationType" : "REST" }, { "app":"P2P", "enabled": true, "serverAddress": "http://10.x.x.x:9001", "bindingAddress": "http://0.0.0.0:9001", "sslConfig": { "tls": "OFF" }, "communicationType" : "REST" } ], "peer": [ { "url": "http://10.x.x.x:9001" }, { "url": "http://10.x.x.x:9001" } ], "keys": { "hashicorpKeyVaultConfig": { "url": "http://10.x.x.x:8200" }, "keyData": [ { "hashicorpVaultSecretEngineName": "secret", "hashicorpVaultSecretName": "$${HOST_NAME}-tessera-key", "hashicorpVaultPrivateKeyId": "privateKey", "hashicorpVaultPublicKeyId": "publicKey" }, { "hashicorpVaultSecretEngineName": "secret", "hashicorpVaultSecretName": "t2-tessera-key", "hashicorpVaultPrivateKeyId": "privateKey", "hashicorpVaultPublicKeyId": "publicKey" } ] }, "alwaysSendTo": [], "features": { "enableMultiplePrivateStates": "true" }, "residentGroups":[ { "name":"private", "description":"default privacy group", "members":["G27/Rz02qexxYTI7h54W9Ebzoz3LgFppBhLSWbzgxCA="] }, { "name":"PS2", "description":"Tenant2 privacy group", "members":["swAqIm6pDee0Y2mTEH7wmFPFPoK7QQL3WdZzXBVeASQ="] } ]

[subhasisbanik]

@antonydenyer and @macfarla any update on this?

[MadelineMurray]

hi @subhasisbanik - I'm following up on this re trying to reproduce. Thanks for sharing the config.

[subhasisbanik]

Hi @MadelineMurray , did you succeed?

[macfarla]

Hi @subhasisbanik, sorry for the delayed response. Can you also check the logs with regard to peering - that is, verify that Nodes 1 and 2 peer with each other after Node1 restarts? If this is working fine then I think we can rule out peering behaviour as the issue. In which case it may indeed be a nuance with the use of HashiCorp Vault.

[namtruong]

Hi,

From the output of your api call

Response from Node1 : curl -X GET http://x.x.x.x:9080/partyinfo/keys {"keys":[{"key":"G27/Rz02qexxYTI7h54W9Ebzoz3LgFppBhLSWbzgxCA="},{"key":"swAqIm6pDee0Y2mTEH7wmFPFPoK7QQL3WdZzXBVeASQ="}]} ***** this keys are of default tenant and Tenant1 of Node1

Response from Node2: curl -X GET http://y.y.y.y:9080/partyinfo/keys {"keys":[{"key":"O/ruD6wozXhv8FNg5WTlDDUY4LA9p/StZLzTFeGajW8="}]} *** this keys are of default tenant of Node2

It looks like node1 does not have node2's key in its partyinfo, and also node2 has none of node1's tenants in its partyinfo - meaning they don't peer at all hence I have doubts about transactions happening between them. Could you please double check this?

"peer": [ { "url": "http://10.x.x.x:9001" }, { "url": "http://10.x.x.x:9001" } ]

---> Are these 2 urls node1 and node2 p2p urls ?

If possible please share both node1 and node2 full configs - just so we can confirm that nothing is misconfigured.

Also please share the starting logs of both node1 and node2 - so we can look for any connection issues related to the partyinfo calls between 2 nodes

[namtruong]

Additionally,

there are a couple of issues I noticed with the tessera logs that you sent

2021-11-12 12:51:34.443 [qtp751413576-33] INFO c.q.tessera.q2t.TransactionResource - Enter Request : POST : /sendraw 2021-11-12 12:51:34.474 [qtp751413576-33] INFO c.q.t.d.EncryptedTransactionDAOImpl - Stored transaction sswThPBnMMW+e9PSRVF2g6YpjRiRxrLFgBhLVYAS0KjpyItrsJ9aeTzDGiz9naUWRnzgCm6kRJaWf1bHc+7tqg==

1. POST : /sendraw is a legacy endpoint which support compatibility with constellation. It seems that your GoQuorum instance has constructed a legacy private transaction manager (PTM). It should instantiate a tessera ptm and use /send instead. Please double check this on quorum side or perhaps the /version api call to tessera was somehow not successful.
2. These log messages from tessera also look old and not on the recent versions (or 21.7.2). Please verify this also

[Purbaja]

Response from Node1 : curl -X GET http://x.x.x.x:9080/partyinfo/keys {"keys":[{"key":"G27/Rz02qexxYTI7h54W9Ebzoz3LgFppBhLSWbzgxCA="},{"key":"swAqIm6pDee0Y2mTEH7wmFPFPoK7QQL3WdZzXBVeASQ="}]} ***** this keys are of default tenant and Tenant1 of Node1

Response from Node2: curl -X GET http://y.y.y.y:9080/partyinfo/keys {"keys":[{"key":"O/ruD6wozXhv8FNg5WTlDDUY4LA9p/StZLzTFeGajW8="}]} *** this keys are of default tenant of Node2

Hi @namtruong i can see this output from partyinfo command before and after Node-1 restart. And Node-1 and Node-2 is already a peer. I am able to do private transaction between Tenant-1 of Node1 and default tenant of Node-2.

[Purbaja]

I am sharing the full configs shortly.

[namtruong]

@Purbaja

And Node-1 and Node-2 is already a peer.

--> Your output from partyinfo command indicated otherwise.

How did you send private transactions ? Could you also send details ?

Thanks

[Purbaja]

@namtruong we attached the geth console for Tenant-1 of Node1 like below: geth attach https://10.7.4.15:22000?PSI=PS1 --rpcclitls.insecureskipverify --rpcclitoken "bearer $accessToken"

Then did a simple getter setter private txn with the tessera key of Node2 in privateFor like below:

var sampleContract = web3.eth.contract([{"constant":true,"inputs":[],"name":"getNum","outputs":[{"name":"","type":"uint256"}],"payable":false,"stateMutability":"view","type":"function"},{"constant":false,"inputs":[{"name":"newNum","type":"uint256"}],"name":"setNum","outputs":[],"payable":false,"stateMutability":"nonpayable","type":"function"},{"anonymous":false,"inputs":[{"indexed":false,"name":"blockNum","type":"uint256"}],"name":"fetchBlockNumber","type":"event"}]);

var sample = sampleContract.new( { from: web3.eth.accounts[0], data: '0x60806040526000805534801561001457600080fd5b50610116806100246000396000f3006080604052600436106049576000357c0100000000000000000000000000000000000000000000000000000000900463ffffffff16806367e0badb14604e578063cd16ecbf146076575b600080fd5b348015605957600080fd5b50606060a0565b6040518082815260200191505060405180910390f35b348015608157600080fd5b50609e6004803603810190808035906020019092919050505060a9565b005b60008054905090565b806000819055507fb35117ff5fdfca3f0d253cb88af1d529df640149d2f40738124a2acc8679097b436040518082815260200191505060405180910390a1505600a165627a7a7230582007290a05c57a8fd776218da3fda793ef8cf04f6f851e45f5cfebbb4cee4132830029', gas: '4700000',privateFor : ["O/ruD6wozXhv8FNg5WTlDDUY4LA9p/StZLzTFeGajW8="]

}, function (e, contract){

console.log(e, contract);

if (typeof contract.address !== 'undefined') {

     console.log('Contract mined! address: ' + contract.address + ' transactionHash: ' + contract.transactionHash);

}

});

[Purbaja]

Node1-tessera-config.txt Node2-tessera-config.txt

Tessera config file for both the Nodes is attached here:

[namtruong]

@Purbaja

Thanks for the full configs.

My observations from the configs you sent below

• Node1's peer list config only has itself in it (both URLs are identical and its own urls). This doesn't make much sense so my guess is it might have been a typo in the config. ---> As a result, node1 will never attempt to reach out to node2 (1)
• However, since node2's peer list has both urls (node1's and node2's), the discovery might still work out ok - hence you have been able to transact between 2 nodes.
• Then when you add a tenant onto node1 and restart node1, due to the peer list housekeeping mechanism in Tessera, node2 will remove node1's url in its peer list when a connection failure occurs. ---> As a result, node2 will stop reaching out to node1 (2)
• (1),(2) -> they will not peer again

There are 2 ways you can fix this

1. Correct the configuration on node1 so that it also includes node2's url (x.x.x.5) in the peer list.

or

2. In node2's peer list, remove its own url in the peer list and only have node1's url (x.x.x.4).

You can choose option 2 if for some reasons you deliberately did not want node1 to actively reach out. The reason behind the fix is that - Tessera will only housekeep the peer's url with connection problem if it has already successfully connected to at least 1 peer. By only having 1 url (node1's url) in its peer list, when node1 is down during the restart, node2 will keep on trying again rather than removing the url.

Hope that makes sense

[Purbaja]

@namtruong Thanks for your detail response.

In Node1 we haven't kept Node2's url intentionally as per our business usecase, when we setup Node1 we don't have information about Node2. Later Node2 comes and joins the network.

So, i will try the second fix you provided and let you know if that works.

Thanks.

[namtruong]

@Purbaja could you please confirm if the issue is resolved thanks

[baptiste-b-pegasys]

open it again if this is still happening