search

imulab / go-scim

Building blocks for servers implementing Simple Cloud Identity Management v2
MIT License
145 stars 56 forks source link

Why do we enqueue group membership requests? #59

Closed plamenGo closed 4 years ago

plamenGo commented 4 years ago

Trying to understand the architecture. Can you help me grok why we are using queueing to process certain requests asynchronously?

imulab commented 4 years ago

Hi @plamenGo

The server component of this repo has an opinionated architecture. It is created to serve as an example of how core module, mongo module and other modules in the future can be put together to create a working server. I put some design thoughts into it, but it won't represent everyone's use case.

Now I will try to explain your question.

When modifying the Group resource, the server responds immediately after the Group resource itself has been successfully modified. The Group's modified membership (user joined the group, or left the group) is queued up to be refreshed asynchronously. The task will update the group property of the User resource so it truly reflects the membership of the Group resource. This is done because we do not control the size of the membership sync. And it has the potential to grow to a large number, which, if refreshed synchronously, has the potential to result in very slow response or even timeouts.

The server component is currently using the mongo module as the storage provider. When updating a resource, the resource actually holds an optimistic lock (which is meta.version). If someone is modifying the resource concurrently and finishes before us, our operation will fail because the stored version has changed. This is normally good practice and ensures data consistency. But if we try to update multiple Resources at the same time (i.e. modify the Group and then modify the User membership in one request), we exposes ourselves to increase chance of concurrent modification. And having the API client retry requests is quite frustrating.

That being said, if one's use case is confident that these are non-issues, it is very much possible to update the Group resource synchronously, as the core module was designed to be not so opinionated. The services that updates Resources are interfaces. You can create wrapper implementations around it to call other services to carry out additional operations.

Another possible approach is to have the User resource not store the group properties, but calculate them ad hoc when requested. However, because SCIM is generally considered a protocol to manage user data and hence would favor read speed over write speed. I didn't entertain that thought too much. But once again, if that's someone would like to do, it is possible through tweaking schemas and creating a few custom implementations.

Hope that answers your question.

plamenGo commented 4 years ago

Thank you for the detailed and thoughtful response!

plamenGo commented 4 years ago

@imulab what do you think about enqueueing all requests, not just group sync ones? Thanks in advance for your advice!

imulab commented 4 years ago

@plamenGo enqueueing all requests would violate the SCIM protocol.

The SCIM HTTP protocol expects immediate response, which does not sit well with message based processing logics.

It was possible to enqueue group sync requests because they were internal requests that coordinate between two different kinds of resources. Group sync fulfills eventual consistency instead of immediate consistency between User.groups and Group.members, but you will still get immediate response for the resource operated.

Enqueueing all requests is an interesting thought, and I am sure there are use cases out there that would favor this design, but it definitely wouldn't be able to call itself SCIM implementation if implemented that way.