ocis-accounts service

[butonic]

We need to use the ocis-accounts service as the user backend. It contains normal users and guest accounts that we want to make accessible using LDAP so that kopano can access them.

Tasks

• [x] Create an architecture overview (maybe diagram) before starting to implement
  • https://github.com/owncloud/ocis/pull/229 - request flow sequence diagram
  • https://github.com/owncloud/ocis/pull/243 - login flow sequence diagram
  • [x] Review of the Concept

ocis-glauth tasks @butonic

• [ ] test ocis-glauth against ocis-graph @butonic
  • [ ] allow basic auth in graph api to implement bind, similar to the hack in the graphapi oc10 app?
  • done in https://github.com/owncloud/ocis-graph/pull/30
• [ ] Skeleton Users need to display the displayName (currently surname)

ocis-accounts Tasks

• [x] allow account lookup using properties using email, displayname, username & password (in addition to uuid and sub) https://github.com/owncloud/ocis-accounts/pull/28
• [ ] current lookup implemented in https://github.com/owncloud/ocis-accounts/pull/30 using symlinks
• [x] track efficient file layout for lookup https://github.com/owncloud/ocis-accounts/issues/2
• [ ] Demo Users (einstein, marie, feynmann) should be provided by accouts service

ocis-proxy steps (order is important) @refs @IljaN

• [ ] 1. Middleware for exchanging sub@iss and map them to UUIDs
  • use in proxy: [Spec]: Expose Account UUID proxy#35 @refs @IljaN
  • PR Retrieve Account UUID From User Claims proxy#36 @refs @IljaN
  • do not autoprovision users. They are explicitly provsioned later or during migration.
  • [x] clarify if middlewares can depend on each other or how they can wrap each other
  • order is important, we could create a middleware that ensures the order of multiple middlewares, like oidc auth, then account lookup: https://medium.com/@chrisgregory_83433/chaining-middleware-in-go-918cfbc5644d
  • nice diagrams on middleware order: https://docs.microsoft.com/en-us/aspnet/core/fundamentals/middleware/?view=aspnetcore-3.1#middleware-order and a table explcaining the order of middlewares: https://docs.microsoft.com/en-us/aspnet/core/fundamentals/middleware/?view=aspnetcore-3.1#built-in-middleware
  • middlewares do in fact rely on each other
  • [x] store uuid in context @refs @IljaN
• [ ] 2. check if oc10 endpoint is configured @refs @IljaN
  • check if user exists in oc10 (because user does not exist in the accounts service)
  • if user exists forward to oc10
  • else create account in ocis and forward to ocis
• [ ] 3. Mint internal token using reva token: @refs @IljaN
  • like in ocis-migrations
  • currently using a shared secret, can be changed later

ocis-pkg

• middleware for authenticating the requests using a jwt token, similar to the reva x-access-token

ocis-settings Tasks @kulmann

• settings requests is a seperate requestst that follows tho normal request flow (uses ocis-proxy to authenticate and authorize the user)
• [x] unmint token from x-access-token header using shared secret, see code reference, see issue
• [x] use me in URLs as placeholder for logged in uuid

Other

• [ ] CRUD API for Users in accounts service
  • [ ] extend query capabilities: https://github.com/owncloud/ocis-accounts/issues/2
• [ ] Create Frontend for accounts service
  • List Users (MVP)
  • Deactivate / Delete (optional)

Estimation

• Effort: H
• Complexity: H

[micbar]

Feedback from reviews:

• maybe split up client / system browser -> like oauth
• more description, link to external resources

@michaelstingl

[butonic]

A problem I ran into, which requires additional design.

We need to deal out unique uidnumbers but the graph api has no property for them. So I currently, cannot, without bendingf the rules use the graph api as the backend for glauth. We are using glauth as an ldap server to provide posixaccounts to eos. a posixaccount MUST have the cn, uid, uidNumber, gidNumber, and homeDirectory attributes.

Furthermore, if we are effectively provisioning the accounts, we may need to use a specific uid range to prevent collisions with existing users in other ldap domains. See Red Hat Enterprise Linux > 6 > Identity Management Guide > 9.9. Managing Unique UID and GID Number Assignments for some background. Also we should skip some uidNumbers when looking at UID, GID, SID and RID ... who are you. which deals with mixing ids from ldap and windows. It seems to be recommended to start over 65535 to prevent collisions with local uids.

In the OS the uidNumber becomes the numeric uid, uid (userid) becomes the username or login id and cn is the common name or display name.

The questions are:

• where do we ultimately store those numbers?
• who generates them?
• do we use different uid ranges for guests and users?
  • if so, do we use different glauth services for that, so that the user ldap can be exchanged with an existing one?

[butonic]

There are two scenarios where we need an ldap server:

1. the storage should integrate the accounts to allow bypassing ocis/reva, eg using ssh. In that case the uidNumbers must be unique across internal and guest accounts.
2. the firewall should check if a guest exists before allowing the request to pass into the dmz

We could request an ldap server or credentials with write permissions for these scenarios if the guest provisioning algorithm from owncloud should be used. If another process for guest accounts exists we can assume the uidnumber to be conflict free, the storage can bo configured to use the existing ldap server.

But what if uidnumbers collide in case of a merger?

The other option is to always manage accounts ourself, which is what we do with the accounts service and provide an ldap interface for the customer so he can use it to integrate the storage and firewall.

But can and do we want to implement a scalable ldap server?

Putting users into the filesystem to let eg eos georeplicate the users for the storage makes sense.

Conlusion

• ocis-glauth is source of truth
  • needs to be able to create accounts using ldap because ocis accounts needs to generate and store the uuid
  • holds unique set of uidnumbers
  • can be exchanged with an existing ldap server -> what if a merger happens and uidnumbers need to be merged ... same range collision problems?
  • uuids can be merged, grants and shares are persisted using the uuid.
  • but collisions with the uidnumber are more likely
    • accounts are stored on disk using the uuid
    • in case of a uidnumber collision
    • 1. files of one of the user need to be chowned
    • 2. all shares need to update the ACLs
  • can we merge two ocis instances using the proxy?
• ocis-accounts is generating uuids for sub@iss for new accounts
  • writes them into ldap

if we use acis-accounts as the source of truth, the merge problem remains the same. We could apply same uidnumber range strategy as in Red Hat Enterprise Linux > 6 > Identity Management Guide > 9.9. Managing Unique UID and GID Number Assignments

[butonic]

ocis-accounts only exists to generate a uuid for users. we look up the sub@iss in the accounts service and generate a uuid if necessary.

technically, this can be replaced with a lookup of the user in an ldap server by email or username, whatever claim is sent by the oidc provider. might be a userPrincipalName or a guid. MS has the same problem for their Azure AD / graph: https://docs.microsoft.com/en-us/azure/active-directory/hybrid/plan-connect-design-concepts

The sourceAnchor is used to link existing objects in Azure AD with objects on-premises. IT is also called immutableId and the two names are used interchangeable. It used to default to the objectGUID, but it can change, which is why it now defaults to ms-DS-ConsistencyGuid. ms-DS-ConsistencyGuid defaults to the objectGUID, but it can also take non GUID values aka as byte arrads for legacy compatability.

Which brings us back to the mapping sub@iss to a stable attribute problem.

We need a stable attribute for federated sharing. The id must never change, but it must be mergeable / conflict free when merging two ocis instances. The only type of id that allows that are UUIDs.

Note that this kind of uniqueness is also needed for groups. But there are occasions where millions of groups are in place. And during a migration they might have to be merged. Which leads to the problem of replacing uids and gids in shares and permissions.

This is similar to cross storage moves. The storage id will change ... which will change the fileid. For ocis we plan to drop support for cross storage move and make it a copy and delete operation to notify the user that shares to that file will be removed until we implement a history of old file ids. When the storage providers cooperate we can redirect to the new target and the requesting client can update his reference.

Maybe we can implment something that can deal with changing uids and gids using a history. ms already has a sid history in ad.

[butonic]

currently

1. during auth the sub@iss is used to look up the uuid in ocis-accounts
2. the corresponding ldap user is looked up using the email or username
3. the uidnumber and gidnimber are used to persist acls and file ownership

other approach:

1. during auth the auuh token is used to look up the email at the idp (or it is in the id token)
2. the corresponding ldap user is looked up using the email
3. the uidnumber and gidnimber are used to persist acls and file ownership

changes to the uidnumber and gidnumber must be dealt with by the directory admin. should ocis support changing usernames (chmod) and acls (getfacl -R ... & setfacl --restore=... maybe?)?

The sub@iss won't change unless the idp product changes. oidc knows nothing about groups so we need to talk to an ldap anyway. but ldap has no persistent dentifiers unless we add our own. we could add a schema and copy the current uuid (objectguid or entryuuid or whatever) into that custom attribute. which might still collide when we have to merge two ldap directories.

[butonic]

If we store the sub@iss in cs3 grants we need a way to update them when the sub@iss changes due to a change of the idp product. Or we need to be able to rebuild the shares in the share manager from the storage.

For groups it will be even harder to maintain a uuid ...

we either need an ldap attribute with a uuid value we can use, or we need write access to the ldap server so we can generate the uuid on the fly.

we can define a schema for that attribute.

https://ldapwiki.com/wiki/Best%20Practices%20For%20Unique%20Identifiers

[butonic]

another thing we need for sharing is a way to map the claims in the idtoken / from the userinfo endpoint to an account. when sharing files with others the account might not have been provisioned yet. if we can only share with existing accounts we also need a way to map an email to that account.

to share with other users the email allows starting an oidc discovery to find the issuer for the user. the webfinger protocol allows looking up users: https://openid.net/specs/openid-connect-discovery-1_0.html#AcctURISyntax or https://openid.net/specs/openid-connect-discovery-1_0.html#EmailSyntax

another interesting link: fastfed spec https://openid.net/specs/fastfed-core-1_0-02.html#ScimSchemaGrammar very recent: https://bitbucket.org/openid/fastfed/src/master/

[butonic]

In the end switching the idp is the same as migrating a user.

[butonic]

The first user story is changing the idp due to a merger, or because an external user becomes an internal user or vice versa. In those cases the sub@iss combo will change. But we need a stable identifier for federated shares. ocis-accounts will generate and store the accounts, which consist of the uuid and a set of identities. An identity consists of an issuer, an issuer assigned id and a type. similar to the ms graph objectIdentity resource. This allows adding multiple identities to an account.

With a generated uuid we can identify the users. Instead of generating the id ourself we should use the scim_id or scim_externalid claim if it is present in the oidc id token claims. That spec is still a draft, but scim and oidc are also tied together in the recent OIDC FastFed draft:https://openid.net/specs/fastfed-scim-1_0-02.html

To persist shares as acls we need the current username under which the user is known in the os. For unix this can be done by configuring pamto use ldap to resolve accounts. We could use other modules like pam_mysql or even implement new ones, but the fact that we a numeric uid and a username pair does not change. It is however optional because it is only necessary when trying to persist shares as acls to the filesystem.

When using a fuse overlay the owner user uuid and group uuid could be stored as extended attributes. shares can use an acl like syntax but with the uuid and store that in a dedicated file, because extended attributes are limited in space.

I already mentioned group uuids. Which AFAICT is only needed for federated group shares ...

[butonic]

hm stumbled over https://systemd.io/USER_RECORD/ again.

The oidc claims have a preferred username. we could check if a user with the name exists. either by checking /home/<username>.homedir/.identity or a different path, eg /var/lib/ocis/<username>.homedir/.identity.

that would be lookup by username.

We could cache the identities in glauth to enable searching other properties using ldap. then searching for recipients would have to use ldap to find accounts. or do an oidc discovery by email. a homed backend for glauth. well ... homed is hardcoded to /home ... but we could at least use the USER_RECORD json format ...

[butonic]

glauth with write support and homed backend as a minimal persistence layer for accounts

[butonic]

Scenarios:

no LDAP integration

-> ocis uses ocis-glauth to persist accounts using LDAP -> uuids are generated by accounts service and stored in glauth using LDAP -> posixaccount.numericUid (for os integration) is generated by accounts service and stored in glauth using LDAP -> guests are provisioned by ocis on the fly as users using LDAP

write access to external LDAP

-> same as no LDAP integration -> add LDAP schema for our uuid or configure a suited attribute

• keep in mind the uuid will be exposed to the outside not only during federated sharing but also other API, eg to tie together settings bundles and maybe even other extensions

read only access to LDAP

-> ocis cannot manage accounts -> uuid must be provided by LDAP (with schema) -> posixaccount.numericuid must be provided by LDAP -> guests must be provisioned by external process

• we may provide a hook when creating guest accounts to call a http endpoint or execute a script so the customer can implement his own workflow.

Remarks

We always need to look up users using username, email or our LDAP uuid that is sent by the IdP as an additional claim. sub claim is useless to us because we cannot look up users with it. One exception: no ldap integration, means we manage all the accounts and could use sub as the uuid ... but then we leak the sub, which might be considered ok if it is a Pairwise Pseudonymous Identifier (PPID)

[butonic]

systemd-homed seems to evolve in the direction we need: https://github.com/systemd/systemd/blob/a9ab5cdb505d7d368c44fc02cc0183e75db1f657/docs/USERDB_AND_DESKTOPS.md#future-additions

[butonic]

Handover:

Where are we? Starting with protocol changes in ocis-accounts:

• we have a new (minimal) protobuf for the accounts service. The accounts limit themselves to

  message Account {
  // the non reassignable and stable account_id of an account
  string account_id = 1;
  
  // The identities that are mapped to this account
  repeated Identities identities = 2; // keep track of every identity of a given user
  
  string username = 3;
  string display_name = 4;
  string mail = 5;
  // TODO avatar needed? because the standard claims contain it or our own avatar service will use the accountid or email to look up the avatar
  
  repeated Group groups = 6;
  }

  The rpcs now follow the API recommendations from Google on

• Methods
• pagination
• naming_conventions

The PR for this is https://github.com/owncloud/ocis-accounts/pull/30

Update depending extensions

Some services need to be updated

• ocis-migration Update to new accounts api owncloud/ocis-glauth#7
  • @IljaN Maybe you just want to update an existing account?
• ocis-proxy Update to new accounts api #39
• ocis-graph: Switch to ocis-accounts as the user backend #30

Update ocis-glauth to use the glauth:homed branch

• glauth PR for homed implementation in https://github.com/glauth/glauth/pull/139

  • [ ] add, modify and delete still need implementation

• ocis-glauth needs https://github.com/owncloud/ocis-glauth/pull/16

  • to create the reva user that is hardcoded in the config datastore of ocis-glauth create a ./users/reva.homedir/.identity file with this content:

    {
    "userName":"reva",
    "uid":10001,
    "gid":15000,
    "emailAddress":"storage@example.org",
    "realName":"Reva Inter Operability Platform",
    "OwnCloudUUID":"9b02e06a-4ad0-43bd-afe9-2e4fccdc2a0a",
    "privileged": {
    "hashedPassword": ["$6$kvkJWETzJsEYgVM8$Mo4Pcei7K/oPvIMqVpcWuMYoItfaSrAklriG/RO4ARhI52.LCu8hFIKU1E4j1DDW5H8LRxH0DYT31L36qB3au1"]
    }
    }

• json data is currently not signed

• the hashed password is created using bcrypt. I just created a user locally, set his password and copied the string from /etc/shadow. glauth uses https://github.com/tredoe/osutil/user/crypt to parse more hashes than provided by the stdlib.

• all users are indexed using bleve (it will create a homed.bleve directory, use the bleve cli to examine the index)

  • the document id is the new ownclouduuid attribute
  • [ ] register a new OID for the owncloud uuid attrubute in the owncloud namespace (the email is currently an alias to my work address)

Testing:

After adding all the changed repos to your ocis go.mod

replace github.com/owncloud/ocis-accounts => ../ocis-accounts
replace github.com/owncloud/ocis-proxy => ../ocis-proxy
replace github.com/owncloud/ocis-migration => ../ocis-migration
replace github.com/owncloud/ocis-glauth => ../ocis-glauth
replace github.com/glauth/glauth => ../glauth

the only thing that currently works is listing users using the api gateway:

$ curl 'http://localhost:8082/rpc' \                  
  -H 'Content-type: application/json' \
  -H 'Accept: */*' \
  -H 'Origin: http://localhost:8082' \
  -d '{"service":"com.owncloud.api.accounts","endpoint":"AccountsService.ListAccounts","request":{}}'

{"accounts":[{"accountId":"9b02e06a-4ad0-43bd-afe9-2e4fccdc2a0a","username":"reva","mail":"storage@example.org"},{"accountId":"e0b79cfd-fa8c-436b-aca5-0a45fbd9e38c","username":"einstein","mail":"einstein@example.org"}]}

Next steps

• [ ] in ocis-accounts implement a query parser using https://github.com/araddon/qlbridge to search for a user by username or email
  • we should not invenst our own query language, options I see are scim or graphapi / odata
  • for minimal ldap filter mimicking we need eq, and, or, and not
  • these operators are part of graphapi AND scim
  • they need to be mapped to ldap filters
  • I'd go with graphapi for future considerations
• [ ] implement middleware in proxy to add accounts on the fly
  • this needs to build the query to look up a user by username or email, which is thens sent to ocis-accounts using grpc and the new ListAccounts rpc
  • [ ] this requires a writeable ldap server / or implementing write support in glauth
• [ ] flesh out logic for updating the identity when a users username or email changed
  • no longer needed? we store the uuid in the ldap server (or the admin has to provide a uuid attribute) an can always look it up.
  • quick hack: for openldap use entdryuuid for ad use objectguid BUT we aware that at some point that Id needs to be copied to the new ownclouduuid attribute
  • [ ] the ldap syntax is currently uuid, but we may want to allow strins for migration. needs design

[exalate-issue-sync[bot]]

Remote key is https://jira.owncloud.com/browse/OC-1972

[butonic]

next steps

ocis-accounts

• update api with graph api properties (onpremis*) to sync
• persist userts on disk using json marshaling of tho go struct
• implement query using bleve (use code from experimental glauth implementation)