search

awslabs / ssosync

Populate AWS SSO directly with your G Suite users and groups using either a CLI or AWS Lambda
Apache License 2.0
512 stars 175 forks source link

SSO Lambda Deletes then Recreates Users #194

Closed snorberhuis closed 2 months ago

snorberhuis commented 3 months ago

Describe the bug AWS SSO Sync running in a cron cycle detects no users or groups and removes all the users and groups in AWS. The next cycle, it will detect all users and groups and create them again. Permission Set assignments are lost resulting in breaking access for engineers.

This was also reported here:https://github.com/awslabs/ssosync/issues/159#issuecomment-1927312096

For me it seems likely a HTTP call is slow or unresponsive somewhere on the Google and this failure is not correctly caught resulting in assuming users and groups are deleted.

Logs before deletion:

<html><head></head><body>
INIT_START Runtime Version: provided:al2.v35    Runtime Version ARN: arn:aws:lambda:eu-west-1::runtime:dbecfbc4d1fffffa8cecb35fe69cfc2464c506b2575789fc74b2e9611417ccd5
--
time="2024-05-04T06:25:08Z" level=info msg="Executing as Lambda"
START RequestId: 40d39cec-c64f-4f14-af34-fa6588e685dd Version: $LATEST
{     "level": "info",     "msg": "Syncing AWS users and groups from Google Workspace SAML Application",     "time": "2024-05-04T06:25:08Z" }
{     "level": "info",     "msg": "syncing",     "sync_method": "groups",     "time": "2024-05-04T06:25:08Z" }
{     "level": "info",     "msg": "get google groups",     "query": "name:AWS*",     "time": "2024-05-04T06:25:08Z" }
{     "level": "debug",     "msg": "preparing list of google users and then google groups and their members",     "time": "2024-05-04T06:25:09Z" }
{     "group": "AWSExampleEcs",     "level": "debug",     "msg": "get group members from google",     "time": "2024-05-04T06:25:09Z" }
{     "group": "AWSExampleEcs",     "level": "debug",     "msg": "get users",     "time": "2024-05-04T06:25:09Z" }
{     "group": "AWSExampleEcs",     "id": "steffan@***.nl",     "level": "debug",     "msg": "get user",     "time": "2024-05-04T06:25:09Z" }
{     "group": "AWSExampleFrontend",     "level": "debug",     "msg": "get group members from google",     "time": "2024-05-04T06:25:10Z" }
{     "group": "AWSExampleFrontend",     "level": "debug",     "msg": "get users",     "time": "2024-05-04T06:25:10Z" }
{     "group": "AWSExampleFrontend",     "id": "steffan@***.nl",     "level": "debug",     "msg": "get user",     "time": "2024-05-04T06:25:10Z" }
{     "group": "AWSExampleLambda",     "level": "debug",     "msg": "get group members from google",     "time": "2024-05-04T06:25:10Z" }
{     "group": "AWSExampleLambda",     "level": "debug",     "msg": "get users",     "time": "2024-05-04T06:25:10Z" }
{     "group": "AWSExampleLambda",     "id": "steffan@***.nl",     "level": "debug",     "msg": "get user",     "time": "2024-05-04T06:25:10Z" }
{     "group": "AWSPlatform",     "level": "debug",     "msg": "get group members from google",     "time": "2024-05-04T06:25:11Z" }
{     "group": "AWSPlatform",     "level": "debug",     "msg": "get users",     "time": "2024-05-04T06:25:11Z" }
{     "group": "AWSPlatform",     "id": "steffan@***.nl",     "level": "debug",     "msg": "get user",     "time": "2024-05-04T06:25:11Z" }
{     "group": "AWSSecurity",     "level": "debug",     "msg": "get group members from google",     "time": "2024-05-04T06:25:11Z" }
{     "group": "AWSSecurity",     "level": "debug",     "msg": "get users",     "time": "2024-05-04T06:25:12Z" }
{     "group": "AWSSecurity",     "id": "steffan@***.nl",     "level": "debug",     "msg": "get user",     "time": "2024-05-04T06:25:12Z" }
{     "level": "info",     "msg": "get existing aws groups",     "time": "2024-05-04T06:25:12Z" }
{     "level": "info",     "msg": "get existing aws users",     "time": "2024-05-04T06:25:12Z" }
{     "level": "info",     "msg": "get active status for aws users",     "time": "2024-05-04T06:25:12Z" }
{     "level": "info",     "msg": "preparing map of user id's to user",     "time": "2024-05-04T06:25:12Z" }
{     "level": "debug",     "msg": "preparing list of aws groups and their members",     "time": "2024-05-04T06:25:12Z" }
{     "level": "info",     "msg": "syncing changes",     "time": "2024-05-04T06:25:12Z" }
{     "level": "debug",     "msg": "deleting aws users deleted in google",     "time": "2024-05-04T06:25:12Z" }
{     "level": "debug",     "msg": "updating aws users updated in google",     "time": "2024-05-04T06:25:12Z" }
{     "level": "debug",     "msg": "creating aws users added in google",     "time": "2024-05-04T06:25:12Z" }
{     "level": "debug",     "msg": "creating aws groups added in google",     "time": "2024-05-04T06:25:12Z" }
{     "level": "debug",     "msg": "validating groups members, equals in aws and google",     "time": "2024-05-04T06:25:12Z" }
{     "group": "AWSExampleEcs",     "level": "debug",     "msg": "finding user",     "time": "2024-05-04T06:25:12Z",     "user": "steffan@***.nl" }
{     "group": "AWSExampleEcs",     "level": "debug",     "msg": "checking user is in group already",     "time": "2024-05-04T06:25:12Z",     "user": "steffan@***.nl" }
{     "group": "AWSExampleFrontend",     "level": "debug",     "msg": "finding user",     "time": "2024-05-04T06:25:12Z",     "user": "steffan@***.nl" }
{     "group": "AWSExampleFrontend",     "level": "debug",     "msg": "checking user is in group already",     "time": "2024-05-04T06:25:13Z",     "user": "steffan@***.nl" }
{     "group": "AWSExampleLambda",     "level": "debug",     "msg": "finding user",     "time": "2024-05-04T06:25:13Z",     "user": "steffan@***.nl" }
{     "group": "AWSExampleLambda",     "level": "debug",     "msg": "checking user is in group already",     "time": "2024-05-04T06:25:13Z",     "user": "steffan@***.nl" }
{     "group": "AWSPlatform",     "level": "debug",     "msg": "finding user",     "time": "2024-05-04T06:25:13Z",     "user": "steffan@***.nl" }
{     "group": "AWSPlatform",     "level": "debug",     "msg": "checking user is in group already",     "time": "2024-05-04T06:25:13Z",     "user": "steffan@***.nl" }
{     "group": "AWSSecurity",     "level": "debug",     "msg": "finding user",     "time": "2024-05-04T06:25:13Z",     "user": "steffan@***.nl" }
{     "group": "AWSSecurity",     "level": "debug",     "msg": "checking user is in group already",     "time": "2024-05-04T06:25:13Z",     "user": "steffan@***.nl" }
{     "level": "debug",     "msg": "delete aws groups deleted in google",     "time": "2024-05-04T06:25:13Z" }
{     "level": "info",     "msg": "sync completed",     "time": "2024-05-04T06:25:13Z" }
END RequestId: 40d39cec-c64f-4f14-af34-fa6588e685dd
REPORT RequestId: 40d39cec-c64f-4f14-af34-fa6588e685dd  Duration: 5214.62 ms    Billed Duration: 5323 ms    Memory Size: 128 MB Max Memory Used: 37 MB  Init Duration: 108.29 ms

</body></html>

Logs during deletion cycle as confirmed by CloudTrail

<html><head></head><body>
INIT_START Runtime Version: provided:al2.v35    Runtime Version ARN: arn:aws:lambda:eu-west-1::runtime:dbecfbc4d1fffffa8cecb35fe69cfc2464c506b2575789fc74b2e9611417ccd5
--
time="2024-05-04T06:40:08Z" level=info msg="Executing as Lambda"
START RequestId: 84b49601-30a8-42a8-ad48-581f3d27b336 Version: $LATEST
{     "level": "info",     "msg": "Syncing AWS users and groups from Google Workspace SAML Application",     "time": "2024-05-04T06:40:09Z" }
{     "level": "info",     "msg": "syncing",     "sync_method": "groups",     "time": "2024-05-04T06:40:09Z" }
{     "level": "info",     "msg": "get google groups",     "query": "name:AWS*",     "time": "2024-05-04T06:40:09Z" }
{     "level": "debug",     "msg": "preparing list of google users and then google groups and their members",     "time": "2024-05-04T06:40:19Z" }
{     "level": "info",     "msg": "get existing aws groups",     "time": "2024-05-04T06:40:19Z" }
{     "level": "info",     "msg": "get existing aws users",     "time": "2024-05-04T06:40:19Z" }
{     "level": "info",     "msg": "get active status for aws users",     "time": "2024-05-04T06:40:19Z" }
{     "level": "info",     "msg": "preparing map of user id's to user",     "time": "2024-05-04T06:40:19Z" }
{     "level": "debug",     "msg": "preparing list of aws groups and their members",     "time": "2024-05-04T06:40:19Z" }
{     "level": "info",     "msg": "syncing changes",     "time": "2024-05-04T06:40:20Z" }
{     "level": "debug",     "msg": "deleting aws users deleted in google",     "time": "2024-05-04T06:40:20Z" }
{     "level": "debug",     "msg": "finding user",     "time": "2024-05-04T06:40:20Z",     "user": "steffan@***.nl" }
{     "level": "warning",     "msg": "deleting user",     "time": "2024-05-04T06:40:20Z",     "user": "steffan@***.nl" }
{     "level": "debug",     "msg": "updating aws users updated in google",     "time": "2024-05-04T06:40:20Z" }
{     "level": "debug",     "msg": "creating aws users added in google",     "time": "2024-05-04T06:40:20Z" }
{     "level": "debug",     "msg": "creating aws groups added in google",     "time": "2024-05-04T06:40:20Z" }
{     "level": "debug",     "msg": "validating groups members, equals in aws and google",     "time": "2024-05-04T06:40:20Z" }
{     "level": "debug",     "msg": "delete aws groups deleted in google",     "time": "2024-05-04T06:40:20Z" }
{     "group": "AWSPlatform",     "level": "debug",     "msg": "finding group",     "time": "2024-05-04T06:40:20Z" }
{     "group": "AWSPlatform",     "level": "warning",     "msg": "deleting group",     "time": "2024-05-04T06:40:20Z" }
{     "group": "AWSSecurity",     "level": "debug",     "msg": "finding group",     "time": "2024-05-04T06:40:20Z" }
{     "group": "AWSSecurity",     "level": "warning",     "msg": "deleting group",     "time": "2024-05-04T06:40:20Z" }
{     "group": "AWSExampleFrontend",     "level": "debug",     "msg": "finding group",     "time": "2024-05-04T06:40:20Z" }
{     "group": "AWSExampleFrontend",     "level": "warning",     "msg": "deleting group",     "time": "2024-05-04T06:40:20Z" }
{     "group": "AWSExampleEcs",     "level": "debug",     "msg": "finding group",     "time": "2024-05-04T06:40:20Z" }
{     "group": "AWSExampleEcs",     "level": "warning",     "msg": "deleting group",     "time": "2024-05-04T06:40:20Z" }
{     "group": "AWSExampleLambda",     "level": "debug",     "msg": "finding group",     "time": "2024-05-04T06:40:20Z" }
{     "group": "AWSExampleLambda",     "level": "warning",     "msg": "deleting group",     "time": "2024-05-04T06:40:20Z" }
{     "level": "info",     "msg": "sync completed",     "time": "2024-05-04T06:40:20Z" }
END RequestId: 84b49601-30a8-42a8-ad48-581f3d27b336
REPORT RequestId: 84b49601-30a8-42a8-ad48-581f3d27b336  Duration: 12641.85 ms   Billed Duration: 12753 ms   Memory Size: 128 MB Max Memory Used: 36 MB  Init Duration: 110.62 ms

</body></html>

Next cycle where users are created

<html><head></head><body>
INIT_START Runtime Version: provided:al2.v35    Runtime Version ARN: arn:aws:lambda:eu-west-1::runtime:dbecfbc4d1fffffa8cecb35fe69cfc2464c506b2575789fc74b2e9611417ccd5
--
time="2024-05-04T06:55:07Z" level=info msg="Executing as Lambda"
START RequestId: fcae8b7c-49b7-4342-9046-d3b16bf9e656 Version: $LATEST
{     "level": "info",     "msg": "Syncing AWS users and groups from Google Workspace SAML Application",     "time": "2024-05-04T06:55:08Z" }
{     "level": "info",     "msg": "syncing",     "sync_method": "groups",     "time": "2024-05-04T06:55:08Z" }
{     "level": "info",     "msg": "get google groups",     "query": "name:AWS*",     "time": "2024-05-04T06:55:08Z" }
{     "level": "debug",     "msg": "preparing list of google users and then google groups and their members",     "time": "2024-05-04T06:55:09Z" }
{     "group": "AWSExampleEcs",     "level": "debug",     "msg": "get group members from google",     "time": "2024-05-04T06:55:09Z" }
{     "group": "AWSExampleEcs",     "level": "debug",     "msg": "get users",     "time": "2024-05-04T06:55:09Z" }
{     "group": "AWSExampleEcs",     "id": "steffan@***.nl",     "level": "debug",     "msg": "get user",     "time": "2024-05-04T06:55:09Z" }
{     "group": "AWSExampleFrontend",     "level": "debug",     "msg": "get group members from google",     "time": "2024-05-04T06:55:10Z" }
{     "group": "AWSExampleFrontend",     "level": "debug",     "msg": "get users",     "time": "2024-05-04T06:55:10Z" }
{     "group": "AWSExampleFrontend",     "id": "steffan@***.nl",     "level": "debug",     "msg": "get user",     "time": "2024-05-04T06:55:10Z" }
{     "group": "AWSExampleLambda",     "level": "debug",     "msg": "get group members from google",     "time": "2024-05-04T06:55:10Z" }
{     "group": "AWSExampleLambda",     "level": "debug",     "msg": "get users",     "time": "2024-05-04T06:55:10Z" }
{     "group": "AWSExampleLambda",     "id": "steffan@***.nl",     "level": "debug",     "msg": "get user",     "time": "2024-05-04T06:55:10Z" }
{     "group": "AWSPlatform",     "level": "debug",     "msg": "get group members from google",     "time": "2024-05-04T06:55:11Z" }
{     "group": "AWSPlatform",     "level": "debug",     "msg": "get users",     "time": "2024-05-04T06:55:11Z" }
{     "group": "AWSPlatform",     "id": "steffan@***.nl",     "level": "debug",     "msg": "get user",     "time": "2024-05-04T06:55:11Z" }
{     "group": "AWSSecurity",     "level": "debug",     "msg": "get group members from google",     "time": "2024-05-04T06:55:11Z" }
{     "group": "AWSSecurity",     "level": "debug",     "msg": "get users",     "time": "2024-05-04T06:55:12Z" }
{     "group": "AWSSecurity",     "id": "steffan@***.nl",     "level": "debug",     "msg": "get user",     "time": "2024-05-04T06:55:12Z" }
{     "level": "info",     "msg": "get existing aws groups",     "time": "2024-05-04T06:55:12Z" }
{     "level": "info",     "msg": "get existing aws users",     "time": "2024-05-04T06:55:12Z" }
{     "level": "info",     "msg": "get active status for aws users",     "time": "2024-05-04T06:55:12Z" }
{     "level": "info",     "msg": "preparing map of user id's to user",     "time": "2024-05-04T06:55:12Z" }
{     "level": "debug",     "msg": "preparing list of aws groups and their members",     "time": "2024-05-04T06:55:12Z" }
{     "level": "info",     "msg": "syncing changes",     "time": "2024-05-04T06:55:12Z" }
{     "level": "debug",     "msg": "deleting aws users deleted in google",     "time": "2024-05-04T06:55:12Z" }
{     "level": "debug",     "msg": "updating aws users updated in google",     "time": "2024-05-04T06:55:12Z" }
{     "level": "debug",     "msg": "creating aws users added in google",     "time": "2024-05-04T06:55:12Z" }
{     "level": "info",     "msg": "creating user",     "time": "2024-05-04T06:55:12Z",     "user": "steffan@***.nl" }
{     "level": "debug",     "msg": "creating aws groups added in google",     "time": "2024-05-04T06:55:12Z" }
{     "group": "AWSExampleEcs",     "level": "info",     "msg": "creating group",     "time": "2024-05-04T06:55:12Z" }
{     "group": "AWSExampleEcs",     "level": "debug",     "msg": "finding user",     "time": "2024-05-04T06:55:12Z" }
{     "group": "AWSExampleEcs",     "level": "info",     "msg": "adding user to group",     "time": "2024-05-04T06:55:12Z",     "user": "steffan@***.nl" }
{     "group": "AWSExampleFrontend",     "level": "info",     "msg": "creating group",     "time": "2024-05-04T06:55:12Z" }
{     "group": "AWSExampleFrontend",     "level": "debug",     "msg": "finding user",     "time": "2024-05-04T06:55:12Z" }
{     "group": "AWSExampleFrontend",     "level": "info",     "msg": "adding user to group",     "time": "2024-05-04T06:55:12Z",     "user": "steffan@***.nl" }
{     "group": "AWSExampleLambda",     "level": "info",     "msg": "creating group",     "time": "2024-05-04T06:55:12Z" }
{     "group": "AWSExampleLambda",     "level": "debug",     "msg": "finding user",     "time": "2024-05-04T06:55:12Z" }
{     "group": "AWSExampleLambda",     "level": "info",     "msg": "adding user to group",     "time": "2024-05-04T06:55:13Z",     "user": "steffan@***.nl" }
{     "group": "AWSPlatform",     "level": "info",     "msg": "creating group",     "time": "2024-05-04T06:55:13Z" }
{     "group": "AWSPlatform",     "level": "debug",     "msg": "finding user",     "time": "2024-05-04T06:55:13Z" }
{     "group": "AWSPlatform",     "level": "info",     "msg": "adding user to group",     "time": "2024-05-04T06:55:13Z",     "user": "steffan@***.nl" }
{     "group": "AWSSecurity",     "level": "info",     "msg": "creating group",     "time": "2024-05-04T06:55:13Z" }
{     "group": "AWSSecurity",     "level": "debug",     "msg": "finding user",     "time": "2024-05-04T06:55:13Z" }
{     "group": "AWSSecurity",     "level": "info",     "msg": "adding user to group",     "time": "2024-05-04T06:55:13Z",     "user": "steffan@***.nl" }
{     "level": "debug",     "msg": "validating groups members, equals in aws and google",     "time": "2024-05-04T06:55:13Z" }
{     "level": "debug",     "msg": "delete aws groups deleted in google",     "time": "2024-05-04T06:55:13Z" }
{     "level": "info",     "msg": "sync completed",     "time": "2024-05-04T06:55:13Z" }
END RequestId: fcae8b7c-49b7-4342-9046-d3b16bf9e656
REPORT RequestId: fcae8b7c-49b7-4342-9046-d3b16bf9e656  Duration: 5555.26 ms    Billed Duration: 5667 ms    Memory Size: 128 MB Max Memory Used: 35 MB  Init Duration: 111.63 ms

</body></html>

To Reproduce Steps to reproduce the behavior: Unknown

Expected behavior Users and groups are created once.

Additional context The sync has been running continually since December

ChrisPates commented 3 months ago

Thanks for raising I'll investigate. If the problem is in the upstream library, I'll figure out some mitigation. whilst the dependancy is fixed. switching from delete to disable and retrying on empty datasets would be worth considering.

ChrisPates commented 3 months ago

Out of interest how many large users/groups in you syncing?

I'm wondering whether the slow response might due to assembling the response with many entries. It might help me build a test harness.

snorberhuis commented 3 months ago

I am syncing 1 user and 5 groups.

snorberhuis commented 2 months ago

Thank you for fixing this issue @ChrisPates !

ChrisPates commented 2 months ago

No problem, I have a couple of other bug fixes to complete and then I'll push a new release to the serverless application repository.