A small list of questions regarding MultiTenancy and docker

[vongohren]

Failing things first:

The 1)2)3).... list in the demo, that does not work in multitennancy? It stops the whole process when I try to use them.

General multitennacny questions

Webhooks used in the demo, on the first wallet created is: http://192.168.65.3:8023/webhooks, but the default data object for the next sub wallet is localhost:8022? Will both work or what is the dealio?

When I request token to a sub wallet, I can do that without providing a key, and it is working. How come? What does the value of providing a key give me?

Docker question

Im just a bit confused about getting this hosted as things are baked into scripts and hidden away in the demo. So I cannot find any examples on what the docker command of running this would be? Lets say I want to initiate this into heroku or cloud run, do I need to dig into the bash files and find what is the docker run command to use? Or are there any pointers I have missed to get the insight needed to do that?

Just want to verify, that I can define my own postgres connection with a bit of configurations?

[ianco]

Hi @vongohren

I just tested the alice/faber demo with multitenancy (main branch) and it seems to be working:

In one bash shell ./run_demo faber --multitenant and in another ./run_demo alice --multitenant and then following the demo instructions I'm able to send a credential, request a proof and send a basic message (options 1, 2, 3). If there are specific errors you are getting please let us know your local configuration and any specific commands you are running. (e.g. are you running a local ledger?)

Re the webhooks - both scenarios should work - the configuration supports a separate webhook per wallet, or a single webhook for all wallets. I don't know offhand where the documentation is for the webhooks, but if you run ./scripts/run_docker start --help it will display all the startup parameters (and optional environment variables).

Re your question about request token to a sub wallet, not completely sure what you're asking. You create a new sub-wallet (and get a token) using the agency api. This uses an api key if aca-py is configured to use api keys (for the alice/faber demo it is not). To make api requests for that wallet you need to provide the token. AFAIK this is working as expected.

The demo displays the actual aca-py command that is executed, including all parameters, so that should give you enough info to run aca-py in a defferent deployment confguration? (As I mentioned the --help parameter can be used to get a full list of options.) For example the Faber command I ran above tells me that it runs aca-py as:

Faber      | ['/home/indy/.pyenv/versions/3.6.9/bin/python', '-m', 'aries_cloudagent', 'start', '--endpoint', 'http://192.168.65.3:8020', '--label', 'Faber.Agent', '--auto-ping-connection', '--auto-respond-messages', '--inbound-transport', 'http', '0.0.0.0', '8020', '--outbound-transport', 'http', '--admin', '0.0.0.0', '8021', '--admin-insecure-mode', '--wallet-type', 'indy', '--wallet-name', 'faber.agent746444', '--wallet-key', 'Faber.Agent746444', '--preserve-exchange-records', '--auto-provision', '--multitenant', '--multitenant-admin', '--jwt-secret', 'very_secret_secret', '--genesis-transactions', '{... genesis info not included ...}', '--seed', 'd_000000000000000000000000746444', '--webhook-url', 'http://192.168.65.3:8022/webhooks', '--trace-target', 'log', '--trace-tag', 'acapy.events', '--trace-label', 'Faber.Agent.trace', '--auto-accept-invites', '--auto-accept-requests']

The aca-py documentation is scattered around a bit but if there is something specific you need I can give you some pointers.

[ianco]

Regarding postgres, there are aca-py parameters for specifying the required postgres configurations.

There is an example configuration here: https://github.com/hyperledger/aries-cloudagent-python/blob/main/demo/local-indy-args.yaml

[vongohren]

@ianco great follow ups and thanks for replying so fast! I will try to elaborate!

Reponse 1

I will answer these quotes below

request a proof and send a basic message (options 1, 2, 3). If there are specific errors you are getting please let us know your local configuration and any specific commands you are running. (e.g. are you running a local ledger?)

Re your question about request token to a sub wallet, not completely sure what you're asking.

In one bash shell ./run_demo faber --multitenant and in another ./run_demo alice --multitenant and then following the demo instructions I'm able to send a credential

First of all, it is my understanding that when i run multi tenancy, I would not need to run a second agent of alice, because I can do alice and faber on the same agent, hence the multi tenency? Is this not correct? With this assumption I will proceed. Run one faber agent with this command. Im also specifying ledger her. LEDGER_URL=http://greenlight.bcovrin.vonx.io ./run_demo faber --multitenant --events --no-auto

Then setup up a connection with the normal flow. But to get a "normal" flow with multitenancy you need to have two browsers on the same swagger api url and then get a token from this endpoint. I have seen I get a token from the response of creating a new wallet, but I also see this endpoint, which by default is showing to send a wallet_key, but I dont have to. Which confuses me.

But after I have two browsers running with each of its token, and a connection going. When the connection is made I get the possibility to execute 1),2),3) commands in the ONE terminal running as the multi tennant agent. And if I click 3), send message, it fails with this error

EVENT: Controller POST /connections/c9fb78ed-d43f-4d2d-914b-918e81f498d9/send-message request to Agent with data:
{
    "content": "lala"
}
Faber      |
Faber      | =================
Faber      | 2021-02-19 14:40:54,402 aries_cloudagent.admin.server ERROR Handler error with exception: connection record not found: c9fb78ed-d43f-4d2d-914b-918e81f498d9. WalletItemNotFound.
Shutting down faber agent ...

Reponse 2

The demo displays the actual aca-py command that is executed, including all parameters, so that should give you enough info to run aca-py in a defferent deployment confguration? (As I mentioned the --help parameter can be used to get a full list of options.) For example the Faber command I ran above tells me that it runs aca-py as:

Ok, so just to scratch my own brain, the base docker file to run is this one https://github.com/hyperledger/aries-cloudagent-python/blob/main/docker/Dockerfile.run And there I use those parametere you mention inside the entrypoint to get things started from a hosted CI envrionment right? So the basics here is get aca_py running with the right flags? Is it possible to set the flags based on the config file? I thought I had seen it somewhere?

Reponse 3

There is an example configuration here: https://github.com/hyperledger/aries-cloudagent-python/blob/main/demo/local-indy-args.yaml

[ianco]

Hi @vongohren

Response 1:

Yes it's possible to execute all the use cases using one aca-py multi-tenant agent, however not using the alice/faber demo apps.

(Or rather, it's theoretically possible to do this using only Faber, however you have to be very particular about what you do through the faber console api vs what you do through the browser api. Alice and Faber keep internal state about what the "current wallet" is, so if you create a wallet (or 2 wallets) through the browser api, the console api doesn't know anything about these new wallets and is still running in the context of the wallet that it created, the only one it knows about. The best way to use the demo apps is to either run everything through the alice/faber console api's, or run everything through the browser api.)

For your example, you created two wallets through the browser api. Each wallet has a token associated with it, so when you call subsequent api's you pass this token (as described here: https://github.com/hyperledger/aries-cloudagent-python/blob/main/Multitenancy.md#authentication) to identify the wallet "context". It sounds like you got this working (two browsers each with the appropriate token) so using the browser api you should be able to establish a connection between the two wallets and send a basic message. (The functions won't work in the faber console api because faber doesn't know anything about these two wallets that you created nor their tokens.)

Response 2 (and 3):

Yes the trick is to get aca-py running with the correct parameters. The alice/faber demo's print out the specific parameters they use (so you can see an example of running aca-py with specific configurations) and there is the --help option that I mentioned.

Regarding docker, we also publish docker images that contain all the necessary dependencies, which you can find here: https://hub.docker.com/r/bcgovimages/aries-cloudagent. The dockerfiles in the aca-py repository build based on the local codebase, but the published images match the aca-py releases.

[TimoGlastra]

Regarding the wallet key that the OpenAPI shows, this is only required for unmanaged mode. In managed mode the base wallet stores the key to unlock the wallet. In unmanaged mode it doesn't and you need to provide it. Currently only managed mode is supported.

This is a point of confusion for a lot of people so I'll make a PR to remove unmanaged wallet related endpoints for now

[ianco]

Regarding the wallet key that the OpenAPI shows, this is only required for unmanaged mode. In managed mode the base wallet stores the key to unlock the wallet. In unmanaged mode it doesn't and you need to provide it. Currently only managed mode is supported.

This is a point of confusion for a lot of people so I'll make a PR to remove unmanaged wallet related endpoints for now

@TimoGlastra rather than removing these endpoints can we just add documentation to clarify they apply to unmanaged wallets only? The first iteration of persistent queues (required for unmanaged wallets) is available so we should be able to complete the un-managed wallet scenario soon. (Leaving the un-managed endpoints in place will help keep this top-of-mind for us.)

[TimoGlastra]

I din't mean to remove the endpoints, just the attributes related to unmanaged mode. I'm also fine with documenting, just that we don't confuse people with something that's not working yet.

But I think it is already covered by the docs: https://github.com/hyperledger/aries-cloudagent-python/blob/main/Multitenancy.md#getting-a-token. I also recently added an extra check that will throw if the wallet key is provided but not needed (either in memory wallet or managed mode). @vongohren are you on the newest update of the repo, and did you get an error stating Wallet xxx doesn't require the wallet key to be provided?

What we could do is add the link to the multi-tenancy docs to the SwaggerUI. I think that'll definitely help (see image)

[vongohren]

@TimoGlastra I did not see a message like that, and the repo is about 1 week old main branch.

[vongohren]

@TimoGlastra @ianco I have to admit, im super confused into how to get this running standalone, and not dependent on the demo scripts. It is all a very black box that is hidden away in scripts that I have to interperet.

And going through the documentation, all running points back to demo, and if you dont do demo, you are really left alone with just aca-py, or docker run, and trying to guess all the required variables. It is not really easy to understand 📦

If any of you have 30 minutes to just point out something Im missing, I more than happy to also write up a document that will be called something like CloudAgent for dummies. And see if I can pinpoint a couple of missing pieces for me, and maybe future users

[ianco]

@vongohren it's a complicated app to be sure!

The documentation around the demo is to provide context and understanding of the different features and options in aca-py. If you go through the various demo's and tutorials you should get an understanding of the options that are available in aca-py. It shows how to used the admin api (https://github.com/hyperledger/aries-cloudagent-python/blob/main/demo/AriesOpenAPIDemo.md), how to interact with a mobile client (https://github.com/hyperledger/aries-cloudagent-python/blob/main/demo/AliceGetsAPhone.md) and as well the demo code shows how to write a controller in python.

The help (./scripts/run_docker start --help lists all the parameters and provides some description.

There are two demo configuration files (https://github.com/hyperledger/aries-cloudagent-python/blob/main/demo/demo-args.yaml and https://github.com/hyperledger/aries-cloudagent-python/blob/main/demo/local-indy-args.yaml) that illustrate some combination of parameters to run aca-py, and in addition when you run the demo it displays which parameters are applied for the options you selected.

There are some demo controllers available (https://github.com/hyperledger/aries-acapy-controllers) and 2 edX courses (https://www.edx.org/course/identity-in-hyperledger-aries-indy-and-ursa and https://www.edx.org/course/becoming-a-hyperledger-aries-developer)

There is documentation in the repo around multitenancy, mediation and other aca-py features.

@swcurran may be able to provide further documentation sources.

If you've got some specific ideas around an aca-py 101 (what does a new aca-py developer/user need to know?) we'd love to get your input!

[ianco]

... there's also a channel on HL rocketchat: https://chat.hyperledger.org/channel/aries-cloudagent-python

[vongohren]

Thanks I will see where these things will lead tomorrow. Its afternoon on my end here. I will join the chat and when Im able to achieve what I want to achieve Im super positive into contributing to 101. Because I want to share my view of this and what untangled the ball of yarn called aca-py :P

Il be back on this tomorrow

[swcurran]

To give a little more context to the state of the demos...

The demo started as a simple "issuer" (Faber) and "holder" (Alice) demo, and when run with no parameters, that is all that is executed. Over time, as capabilities were added to the ACA-Py framework, additional startup parameters and code were added to the demo so that those interested could try out the different features and the devs would have ways to make sure that we can test the implementation. This includes:

• revocation
• a performance demo - issue N credentials
• multi-tenancy
• proofs with self-attested credentials
• flexible web-hook handling

and so on.

What I think we are now missing is an overview of the different ways to run the demos in the various modes and what each means. Most importantly, the basic demo for a newcomer can see exactly what is going on. We'll get that on the list to add.

To understand this better, I suggest a few things:

• The ACA-Py architectural overview --
• These labs from the second edX course Ian mentioned go over the lab with links to the code being used:
  • Agents connecting: https://github.com/cloudcompass/ToIPLabs/blob/master/docs/LFS173x/agentsConnecting.md
  • Alice gets a credential: https://github.com/cloudcompass/ToIPLabs/blob/master/docs/LFS173x/AliceGetsCredential.md
• Once that is understood, this walkthrough provides insights into the controller API that is used: https://github.com/hyperledger/aries-cloudagent-python/blob/main/demo/AriesOpenAPIDemo.md

Hopefully that helps. If that doesn't or you have done these steps already, we'd be happy to have a call to help.

Thanks for your patience.

[ianco]

What I think we are now missing is an overview of the different ways to run the demos in the various modes and what each means. Most importantly, the basic demo for a newcomer can see exactly what is going on. We'll get that on the list to add.

There's quite a lot of documentation describing the demo. The "additional options" are described here:

https://github.com/hyperledger/aries-cloudagent-python/tree/main/demo#additional-options-in-the-alicefaber-demo

@swcurran may be worth reviewing all the docs with Alex to suggest a TOC or overview of aca-py docs?

[swcurran]

My apologizes, @ianco -- I need to stay up. Awesome stuff...

Yes, that's what @alexgmetcalf might be able to help with.

[vongohren]

@swcurran @ianco I really appriciate the conversation here! Thanks for clarifying. I will try to give my context of where im at based on your feedbacks.

I do understand the architecture, I believe. And the features available is also understandable, and something I would like to setup for myself. I would firstly like to achieve a setup where I have a multitenant agent running, with postgres, with a simple node.js controller. This controller would just work as a receiver of webhooks, no executor, that would happen from the admin swagger. From my understanding this is quite achievable with current state of the system?

I would need to run the agent, via docker with all the configurations needed. I have been starting super clean, meaning removing everything else except for the agent, from this example: https://github.com/bcgov/indy-email-verification/blob/master/docker/docker-compose.yml

Just getting it running and setting up a minimalistic node express router, that console.logs all webhooks coming in. Im then navigating the swagger API to execute commands. Later to build out a more extensible controller that executes on the agent API. I believe if I setup this I can proove that it is a feasible thing for our architecture.

Have I understood the architecture correctly?

Problems

Postgres

The errors for postgres connection is not very optimal. Been through these issues to no avail: https://github.com/hyperledger/aries-cloudagent-python/issues/568 https://github.com/hyperledger/aries-cloudagent-python/issues/720

Meaning I have verified how to connect to my local postgres by curling inside the docker container, but the system will not connect.

Config

This is were I still have some issues. I think that the documentation is not clear enough on the internal docker configurations meaning that all the different urls used here and there, which can tend to become a bit mixed up with internal docker urls, and ways to do external urls.

So I think that for running a minimalistic version, setting things together myself, there is alot of missing info that I would love to contribute with. I have started writing some things for now.

The end

I would love to have a 30 min chat with anyone to confirm my understanding so I can confirm from my understanding that I can write up a document, there will be some value in it. My endgoal is to get this hosted on google cloud run. So my tutorial would take you from scratch, to an agent running, agent running with postgres, and multitenant. With a simple webhook controller to receive the data it needs to receive. And then get it into google cloud run.

[ianco]

I think you understand the architecture correctly.

Regarding the postgres setup, the typical issue (other than making sure you have the correct url/IP for connecting within the docker network) is with the configuration, which is in json-within-json format. (Nothing we can do about that, it's how we need to provide the config to indy sdk - there are 2 layers of config, one for indy and one for postgres.) There are examples of providing this within a docker compose file as well as within a yml config (that can be passed to aca-py) so the best advice I can give is make sure to compare what you're passing in against these examples.

Regarding config yes there is some complexity about what url's are used where (internal vs external).

If you want to hop on a call you can find my availability here: https://calendly.com/ianco ... early mornings or afternoons work best for me (I'm on pacific time).

[vongohren]

Thanks for quick reply, and sorry for my bad writing in the last post, it went a bit fast. I have done some editing now. But I will spend all day tomorrow on this, so I am ready for our 30 minutes which I just booked. I hope it is early enough for you? After that I will look into contributing a write up/tutorial that takes somone from 0 to hero to get this running, with the nessecary references to things you guys have already made. I do this also for my own understanding as I learn through writing down, and learn to express my thoughts in a clear manner.

[vongohren]

@ianco I will close down this issue now and will look to contribute back a write up as suggested. Cannot promise a deadline on it, but as I said, writing up this will also help myself get the full understanding needed to run this properly.

So thanks for the call, and have a great day onwards!