Introduce Permissions to more granularly manage access

[steffh]

Is your feature request related to a problem? Please describe.

As an Airy user I want to be able to configure permissions (e.g. conversations:read, conversations:write), so that I can more granularly manage the access of different users to endpoints within my Airy instance, utilizing pre-defined roles or build custom roles via the authentication system I use

Describe the solution you'd like

1. Introduce the following Permission Scopes:

• admin: Administer an Airy instance
• admin.teams:read: Read teams configuration (included in admin)
• admin.teams:write: Create and edit teams (included in admin)
• channels:read: Read channels of an instance
• channels:write: Create and modify channels
• conversations:read: Read conversations
• conversations:write: Edit conversation metadata
• media:read: Read Files
• media:write: Upload Files
• messages:read: Read messages
• messages:write: Send messages
• sources:read: Read sources of an instance
• sources:write: Create and edit sources
• tags:read: Read tags
• tags:write: Create and edit tags
• templates:read: Read templates
• templates:write: Create and edit templates
• webhooks:read: Read Information on webhooks
• webhooks:write: Create and edit webhooks

2. Introduce the following pre-defined Roles:

• Member: channels:read, conversations:read, conversations:write, media:write, media:read:, messages:read, messages:write, sources:read, tags:read, tags:write, templates:read, templates:write
• Admin: all of Member permissions + admin, channels:write, conversations:write, sources:write, webhooks:read, webhooks:write

3. Additional requirements:

• This should be ideally managed on the level of the authentication system being used (e.g. Auth0, Github, etc.) to the extent the authentication system supplies such functionality, so that the relevant Airy instance doesn't have to know about the roles, but can focus on if the permission scope being required to access a specific endpoint.

• This also creates the option to define further roles e.g. for Marketer, etc. by assigning individual permissions to such roles.

• The JWTs issued by the Source API (https://github.com/airyhq/airy/issues/2274) for creating the relevant source could be assigned specific permissions this way as well.

Describe alternatives you've considered

Additional context

[chrismatix]

@steffh Looks good sofar. Some questions:

admin.teams:read: Read teams configuration (included in admin)
admin.teams:write: Create and edit teams (included in admin)

The current cloud teams feature will have to be rebuilt anyway, so I'd propose that we leave these scopes out for now.

This should be ideally managed on the level of the authentication system being used (e.g. Auth0, Github, etc.) to the extent the authentication system supplies such functionality, so that the relevant Airy instance doesn't have to know about the roles, but can focus on if the permission scope being required to access a specific endpoint.

This seems to be the most important part of the implementation, but from what I can gather the protocols don't implement a standard way of achieving this. It seems like the only way forward would be to store users and expose an API for assigning roles.

[steffh]

@chrismatix Thanks for the feedback.

• admin.teams:read/writewas only an example for admin permissions, of course happy to keep it out

• I am afraid you are right; it works easily on Auth0 to manage permissions but not for most other OIDC/OAuth providers, leaving you know choice other than storing it on instance level for the relevant users; let's discuss in the context of Contacts once more if storing users was sth we could not avoid anyways e.g. for teams, assignments, permissions, etc.

[chrismatix]

@steffh Luckily we anticipated storing users, but only indirectly. So once a user accesses the instance for the first time we know about her. We could then assign a default role that can be changed later by administrators.

[steffh]

@chrismatix - so would it be easily possible to fetch a list of users that accessed the instance and make that available to the clients? how would you store users, e.g. to add a full name and role to them?

As you start to work on contacts, would you store them separately from contacts and why - or just like contacts having a special role attached to them?

[chrismatix]

@steffh They are completely different because we can only infer users from the access log. So fetching a list of users that accessed the instance would be exactly that use case.

Contacts on the other hand are completely unrelated to authentication. Their schemata might intersect, but the entities are unrelated.

[M-Shorouk]

Adding a comment for Control Center permissions:

• client.config : read: read/view component status view.
• component : read :View all available components in Airy Installed/Uninstalled.
• component : write : Ability to install/uninstall, configure, manage configure, and enable/disable a component.

[chrismatix]

@M-Shorouk I guess that client.config:read and component.read will be part of the member role and component.write the admin role.

Implementation proposal

As an Airy user I want to be able to configure permissions (e.g. conversations:read, conversations:write), so that I can more granularly manage the access of different users to endpoints within my Airy instance, utilizing pre-defined roles or build custom roles via the authentication system I use

To me this implies the following tasks:

1. Allow protecting routes with permissions
2. Provide default roles (member, admin) via a management API
3. Use roles from external providers

I will now go over them one by one and sketch out the behavior:

1. Allow protecting routes with permissions

Users could enable this config by adding this to their airy.yaml

security:
    enableRoles: true

Once this is configured all users will be logged out and by default assigned the member role. The only way to then add the first admin user is by using the system token API.

2. Provide default roles

Currently, we already have a /users.list endpoint that shows all users that have sofar interacted with the instance. We store roles as additional string tags that are added to the JWT when authenticating against the admin service.

POST /users.addRole

{
    "id": "user-id",
    "role": "admin"
}

Returns Accepted 202

POST /users.removeRole

{
    "id": "user-id",
    "role": "admin"
}

Returns Accepted 202

3. Support authentication provider roles

We can allow users to provide a custom mapping from the roles encoded in their id token to our roles in the config:

security:
    oidc:
        roles: "member=ROLE_MEMBER,admin=CUSTOM_ROLE"
        roleAttribute: "role" # Where on the id token the roles are stored 

However this feature puts roles stored in the airy instance in 2 at odds with the ones provided by the third party. Given that editing roles is a very quick task I am wondering if implementing this mapping is even worth it.

Let me know what you think. cc @juan-sebastian @AitorAlgorta

[AitorAlgorta]

I like this proposal @chrismatix

The only thing I am missing is the response with the given permission for the role. I guess it will be inside the client.config and it can change depending on the customer.

Apart from that, everything else looks good.

What do you think? @juan-sebastian

[juan-sebastian]

I think we should have a granular permissions like @steffh proposed above this follows the oauth2 guide lines. Then having roles that group this permissions.

But for example the /client.config endpoint should look for client.config:read or client.config:write and not for an specific role.

[juan-sebastian]

They are call scopes on the Oauth2 RFC. Here is an example in Spring

[chrismatix]

The only thing I am missing is the response with the given permission for the role. I guess it will be inside the client.config and it can change depending on the customer.

Good idea, we should add this to the client.config response.

[chrismatix]

@juan-sebastian I am not sure I understand how that's different from what I wrote?

They are call scopes on the Oauth2 RFC. Here is an example in Spring

We can go ahead and call permissions scopes, but they are not the same thing because we don't implement an oauth2 resource server. Rather upon successful authentication we exchange the authentication token for our own JWT.

[juan-sebastian]

I think we are arriving to the point of needing a gate the purpose of the gate is to handle authentication and permissions scopes. Then that way the other components won't need an ingress rule and all calls goes throw the gate. Then the gate just set in the request header the permission scopes. And each component will only need to check its own scopes

[chrismatix]

Then that way the other components won't need an ingress rule and all calls goes throw the gate

That's certainly something we could do in the future, but it is not required to do this issue or am I missing something?

[juan-sebastian]

I think it won't be too much extra work to go directly with this route. Specially that Spring has all the tools already there. The only thing we have to change is the ingress rules.

[chrismatix]

If I understand the proposal correctly in order to build this api gateway we'd have to do at least the following:

• Write the gateway app
• Deployment descriptors for said app
• Migrate all ingresss routes to the gateway app
• Decide and implement a method of dependency injection (aka how do new components register their routes with the gateway
• Migrate all other features currently handled by the nginx ingress (includes testing payload sizes, default redirects, etc.)
• More docs
• More e2e testing since this touches everything

This is just at the top of my head, but it seems to be a lot of extra work compared to what is asked in the ticket. What do you think?

[juan-sebastian]

You are correct. My concern is that if we go for the regular easer path we will end up creating more technical depth. And when we will have to evolve the permissions flow. It will be much harder if this flow is all over the place. Contrary of having everything in one place in this case the gate.

The questions is how pressing is this feature? If it is not pressing it will be better in my opinion to go for the gateway.

This ticket was created 1 year ago. Not sure if we should rush things now.

[chrismatix]

I am failing to see how this creates any technical debt. If we build the gateway permissions still need to be checked by the apps so it will still be everywhere. If we ever want to move to the gateway we can still do this at zero marginal cost.

Let's talk about this in sync tomorrow. It would be great if you could flesh out your gateway proposal a bit so we can compare the costs better (and expedite the process).

[juan-sebastian]

Yes let's talk tomorrow. I can explain in details what I have in mind. But basically in my proposal no component needs to handle any authentication. They only need to check if the permissions to do a specific action are set on the request header.

It is the job of the gateway to validate the authenticity of the token. And set the correct permissions associated with this token

[chrismatix]

Yes, I get that part. That's just how authentication gateways work. But in order to compare costs, we need an implementation proposal like the one I posted above. Importantly it must take the change from the "is" to the "should" state into account.

[juan-sebastian]

Ok I will write one after the sync tomorrow. After we brainstorm it a bit.

[juan-sebastian]

After the talk we had this morning this is the alpha version of the gateway proposal

Implementation proposal

The idea is to have a gate keeper which its sole role is to authenticate and set the correct permissions scopes before passing the call to the relevant component. The image below shows the basic concept

To implement this we will need the following:

• Write the gateway component (more of that below)
• The other components needs to notify their presence to the gate using /components.register
• Delete all other ingress rules. Everything goes now through the gateway that will have an ingress rule for everything.

I did not mentioned the docs because this is transparent from the client perspective.

The gate

There are solutions that we could adapt for our need like hydra I think for our needs and with the help of spring.

this could be a good start for reverse proxy and security

We could write fairly quickly the gateway . Here below you can see the gateway flow

In a few words the hole role of the gateway is to match the token (more specific the user id associated with that token) to the permission scopes that we defined for this user on our database.

Please feel free to ask any questions and/or improve on the design.

[chrismatix]

Thank you @juan-sebastian for this detailed proposal. I have some questions:

1. When and how do we register components in the Airy lifecycle?
2. Do we keep the existing gateway (the nginx ingress controller)?

I think it's good that we are introducing a database for this. It will greatly simplify the current authentication code at the small cost of introducing another database.

[juan-sebastian]

So I think it will be.

1. during the startup process (probably after kafka is ready) the component will register itself to the gate. 1.2 The gate can be discovered by using kuberneters api. or by simple broadcasting an UDP packet. on which the gate will reply to the sender 1.3 The gateway can call periodically the /heath endpoint to keep track of the alive components
2. I think at some point we won't need it anymore. But at this stage I am not sure yet.

Maybe you could write a short text regarding the propuse(s) of this ingress controller besides the routing. Ljupco, and you knows way better than me 🙏

[juan-sebastian]

This is way beyond our needs but it is a project we can inspire from too. envoy proxy

[juan-sebastian]

Gateway and components discovery proposal.

After the components start and Kafka streams (from some) has finished its initialisation. We can use Kubernetes API to locate the gateway or we could be a bit more agnostic with the following method

Ones we know the location of the gateway we can simple call /components.register As follows

Keep in mind that /components.register is an internal endpoint so it only can be called within the network

Then the gateway will simply store the caller IP and associate it with the received endpoints and permission scopes.

This approach will also support in a very simple step having multiples pods registering for the same endpoints. opening the possibility for load balancing. But this is something for later if needing.

[ljupcovangelski]

@juan-sebastian couple of questions on the implementation proposal: 1) Is the proposal to write our own own API gateway or to use hydra? 1) Will every component/pod have a sidecar container that talks to the gateway or will this be something implemented in the main contaner? The reason why I am asking is because the Istio gateway ingress controller works that way - it automatically deploys an envoy container, which handles some paths defined by a specific Kubernetes resource - VirtualService.