Implement Personal Access Tokens (PATs)

[dborovcanin]

API keys will be used for communication between services. This can be done even as a separate service, but we'll first need some investigation and specs before implementing it. @arvindh123 Please write all the findings in the comment section of this issue.

[dborovcanin]

@arvindh123 Please link here comments from the initial implementation you've done.

[arvindh123]

API Keys which here mentioned are Personal Access Token

PAT have scopes, PAT might or might not have full access to Magistrala. A Magistrala user can create PAT with limited scopes like Example: PAT with scopes of read only things, users, domains. This PAT could not create any entities.

PAT are helps to run automation scripts without user interventions. We could not use password for automations scripts, because If authentication system have Multi-Factor authentication, then it needs human intervention, which becomes interruptions for automation scripts.

If PAT is compromised, we can simply revoke it and create new . PAT might not have full access. So with compromised PAT (PAT with limited scope) no one could not do much things like beyond the scope mentioned in PAT, Example if a compromised PAT is created with read only access, then no one could not do operations like create or delete.

[arvindh123]

PAT Data structure

{
    "platform": {
        "users": {
            "create": {},
            "read": {},
            "list": {},
            "update": {},
            "delete": {}
        }
    },
    "domains": {
        "domain_1": {
            "entities": {
                "groups": {
                    "create": {}, // this for all groups in domain
                },
                "channels": {
                    // for particular channel in domain
                    "delete": {
                        "channel1": {},
                        "channel2":{}
                    }
                },
                "things": {
                    "update": {} // this for all things in domain
                }
            }
        }
    }
}

[arvindh123]

I'm trying to explain here challenges with example use case

Lets take things share function, It requires Token for Authn and Authz.

The PAT will be passed Bearer Token via header. The service function input variable token will either access token or PAT

Typically, for access token , grpc function Identify returns user id , domain id , subject id. These ids are required for further process.

In auth Identify grpc call we can verify the PAT originality. But in response we can return only user ID. Because PAT are not specific to domain, it is specific to user. A PAT might contains scopes for multiple domains.

So service function could not determine on which domain request need to process. The Authorization grpc call needs domain id to do authorization.

In service function, input variable id provide the thing ID. Using this thing ID we can find the PAT have access that thing ID and domain ID. But down side, since we don't know the domain ID , So we need to iterating over all domains in PAT to find. This is not efficient process.

// Token - Access token / PAT encoded ID
func (svc service) Share(ctx context.Context, token, id, relation string, userids ...string) error {
    user, err := svc.identify(ctx, token) // grpc identify call
    // users = { user id , pat id } -> response for PAT,  missing domain ID
    if err != nil {
        return err
    }

    if _, err := svc.authorize(ctx, user.GetDomainId(), auth.UserType, auth.TokenKind, user.GetId(), auth.DeletePermission, auth.ThingType, id); err != nil {
        return errors.Wrap(svcerr.ErrAuthorization, err)
    }

    policies := magistrala.AddPoliciesReq{}
    for _, userid := range userids {
        policies.AddPoliciesReq = append(policies.AddPoliciesReq, &magistrala.AddPolicyReq{
            SubjectType: auth.UserType,
            Subject:     auth.EncodeDomainUserID(user.GetDomainId(), userid),
            Relation:    relation,
            ObjectType:  auth.ThingType,
            Object:      id,
        })
    }
    res, err := svc.auth.AddPolicies(ctx, &policies)
    if err != nil {
        return errors.Wrap(errAddPolicies, err)
    }
    if !res.Added {
        return err
    }
    return nil
}

Options:

• Change endpoint from "/things/{thingid}/share" to "/domains/{domainID}/things/{thingid}/share" Additional Advantage: We can remove domain login, and single login is enough for all domains. Downside: It not give feel like multitenant application experience from API point. With this approach we can remove domain login.

• Moving Domain ID to the end of data structure , like below example: DownSide: Unwanted duplication of Data, which easily makes PAT size heavier.

  "channels": {
      // for particular channel in domain
      "delete": {
              "channel1": "domain1",
              "channel2":"domain2"
        }
  },

• Having separate PAT for domain each domain and separate Platform level operations. PAT will be created for specific domain, So PAT metadata will contain domain ID . In this method, In request , domain ID is not needed, On PAT identify we can return the domain ID. That can be used further by authorization. Downside: In approach Platform administrator / maintainers need multiple PATs for automation tasks

ToDo:

• [ ] Check with other platforms about PAT.
• [ ] Check with Key value store like etcd, badger

[arvindh123]

@dborovcanin If we add domainID to all the endpoints like /domains/{domainID}/things/{thingid}/share Then we need to modify the addons endpoints , since they also works on domain level. And also we need to extend PAT for Addons services. I hope with modifying auth function (make it work with PAT) we can make it work for all services including addons

[arvindh123]

Embedded KV Store

• https://github.com/boltdb/bolt
• https://github.com/etcd-io/bbolt
• https://github.com/dgraph-io/badger

[arvindh123]

• https://github.com/qichengzx/m2 m2 is a simple http key/value cache system based on hashicorp/raft.
• https://github.com/vaccovecrana/vephar vephar is a minimal K/V store with an API and a dashboard using: hashicorp-raft for cluster coordination. badger for Raft logs and data storage. raft-badger with slight modifications for current badger compatibility.
• https://github.com/mosuka/cete Cete is a distributed key value store server written in Go built on top of BadgerDB. It provides functions through gRPC (HTTP/2 + Protocol Buffers) or traditional RESTful API (HTTP/1.1 + JSON). Cete implements Raft consensus algorithm by hashicorp/raft. It achieve consensus across all the instances of the nodes, ensuring that every change made to the system is made to a quorum of nodes, or none at all. Cete makes it easy bringing up a cluster of BadgerDB (a cete of badgers) .

[arvindh123]

• https://github.com/qichengzx/m2 m2 is a simple http key/value cache system based on hashicorp/raft.
• https://github.com/vaccovecrana/vephar vephar is a minimal K/V store with an API and a dashboard using: hashicorp-raft for cluster coordination. badger for Raft logs and data storage. raft-badger with slight modifications for current badger compatibility.
• https://github.com/mosuka/cete Cete is a distributed key value store server written in Go built on top of BadgerDB. It provides functions through gRPC (HTTP/2 + Protocol Buffers) or traditional RESTful API (HTTP/1.1 + JSON). Cete implements Raft consensus algorithm by hashicorp/raft. It achieve consensus across all the instances of the nodes, ensuring that every change made to the system is made to a quorum of nodes, or none at all. Cete makes it easy bringing up a cluster of BadgerDB (a cete of badgers) .

I'm leaning towards Badger, It looks promising for me. Badger’s design is based on a paper titled WiscKey: Separating Keys from Values in SSD-conscious Storage.

And we can implement distribute Badger with Raft as mentioned in previous comment

[dborovcanin]

We will need this for Dashboards sharing on our UI.

[dborovcanin]

This is blocked by the new Auth model, but it is also blocking dashboards sharing on the UI side, so we must address it in this sprint.

[dborovcanin]

This ticket is becoming urgent due to plans to use PAT in Dashboard sharing and other projects that utilize MG auth services.