pomerium / datasource

external data source contrib
Apache License 2.0
1 stars 1 forks source link

datasource/okta: inefficient group sync #342

Closed calebdoxsey closed 1 month ago

calebdoxsey commented 1 month ago

What happened?

Group synchronization is possible using two API methods:

  1. GET /api/v1/groups: to retrieve all the groups, which supports a filter on lastUpdated and lastMembershipUpdated
  2. GET/api/v1/groups/${groupId}/users: to retrieve all the group members for a group

Using these two methods directory synchronization should first involve querying all the groups and then querying all the group members for each of the groups.

On each subsequent synchronization we should utilize the list group filters to only query the groups which have changed.

Although we have code written that attempts to do this, in testing it does not appear to function properly. I see requests for all groups on every synchronization call. This is leading to rate limiting problems.

calebdoxsey commented 1 month ago

This should be fixed.