Groups and Users deleted that are synced using CLI

[snorberhuis]

Describe the bug

1. I have run the CLI to test the necessary configuration options. This now consistently imports the necessary AWS Users and Groups to AWS. This works fine.

➜ ./ssosync  -c credentials.json --google-admin steffan@norberhuis.nl --identity-store-id d-936741ec26 --region eu-west-1 
INFO[0000] Syncing AWS users and groups from Google Workspace SAML Application 
INFO[0000] syncing                                       sync_method=groups
INFO[0000] get google groups                             query=
INFO[0004] get existing aws groups                      
INFO[0004] get existing aws users                       
INFO[0004] get active status for aws users              
INFO[0004] preparing map of user id's to user           
INFO[0004] syncing changes                              
INFO[0004] creating user                                 user=**
INFO[0004] creating group                                group=**
INFO[0004] adding user to group                          group=** user=**
INFO[0004] creating group                                group=**
INFO[0004] adding user to group                          group=** user=**
INFO[0004] creating group                                group=**
INFO[0005] adding user to group                          group=** user=**
INFO[0005] creating group                                group=**
INFO[0005] adding user to group                          group=** user=**
INFO[0005] creating group                                group=**
INFO[0005] adding user to group                          group=** user=**
INFO[0005] sync completed                               

2. I then deployed the SSOSync app from the AWS Serverless Application Repository with the same configuration. This lambda removes all Groups and Users synced using the CLI and does not import any group or users.

INIT_START Runtime Version: provided:al2.v28    Runtime Version ARN: arn:aws:lambda:eu-west-1::runtime:c2f9d23694707c1161c0d3fbc0673f4ee14150265f4f3081161948aa71fad0a0
time="2023-12-28T16:25:07Z" level=info msg="Executing as Lambda"
START RequestId: e9ffd9c0-3743-4700-b0ac-b3f8a740f180 Version: $LATEST
{
    "level": "info",
    "msg": "Syncing AWS users and groups from Google Workspace SAML Application",
    "time": "2023-12-28T16:25:08Z"
}

{
    "level": "info",
    "msg": "syncing",
    "sync_method": "groups",
    "time": "2023-12-28T16:25:08Z"
}

{
    "level": "info",
    "msg": "get google groups",
    "query": "name:AWS*",
    "time": "2023-12-28T16:25:08Z"
}

{
    "level": "debug",
    "msg": "preparing list of google users and then google groups and their members",
    "time": "2023-12-28T16:25:09Z"
}

{
    "level": "info",
    "msg": "get existing aws groups",
    "time": "2023-12-28T16:25:09Z"
}

{
    "level": "info",
    "msg": "get existing aws users",
    "time": "2023-12-28T16:25:09Z"
}

{
    "level": "info",
    "msg": "get active status for aws users",
    "time": "2023-12-28T16:25:09Z"
}

{
    "level": "info",
    "msg": "preparing map of user id's to user",
    "time": "2023-12-28T16:25:09Z"
}

{
    "level": "debug",
    "msg": "preparing list of aws groups and their members",
    "time": "2023-12-28T16:25:09Z"
}

{
    "level": "info",
    "msg": "syncing changes",
    "time": "2023-12-28T16:25:09Z"
}

{
    "level": "debug",
    "msg": "deleting aws users deleted in google",
    "time": "2023-12-28T16:25:09Z"
}

{
    "level": "debug",
    "msg": "finding user",
    "time": "2023-12-28T16:25:09Z",
    "user": "**"
}

{
    "level": "warning",
    "msg": "deleting user",
    "time": "2023-12-28T16:25:09Z",
    "user": "**"
}

{
    "level": "debug",
    "msg": "updating aws users updated in google",
    "time": "2023-12-28T16:25:10Z"
}

{
    "level": "debug",
    "msg": "creating aws users added in google",
    "time": "2023-12-28T16:25:10Z"
}

{
    "level": "debug",
    "msg": "creating aws groups added in google",
    "time": "2023-12-28T16:25:10Z"
}

{
    "level": "debug",
    "msg": "validating groups members, equals in aws and google",
    "time": "2023-12-28T16:25:10Z"
}

{
    "level": "debug",
    "msg": "delete aws groups deleted in google",
    "time": "2023-12-28T16:25:10Z"
}

{
    "group": "**",
    "level": "debug",
    "msg": "finding group",
    "time": "2023-12-28T16:25:10Z"
}

{
    "group": "**",
    "level": "warning",
    "msg": "deleting group",
    "time": "2023-12-28T16:25:10Z"
}

{
    "group": "**",
    "level": "debug",
    "msg": "finding group",
    "time": "2023-12-28T16:25:10Z"
}

{
    "group": "**",
    "level": "warning",
    "msg": "deleting group",
    "time": "2023-12-28T16:25:10Z"
}

{
    "group": "**",
    "level": "debug",
    "msg": "finding group",
    "time": "2023-12-28T16:25:10Z"
}

{
    "group": "**",
    "level": "warning",
    "msg": "deleting group",
    "time": "2023-12-28T16:25:10Z"
}

{
    "group": "**",
    "level": "debug",
    "msg": "finding group",
    "time": "2023-12-28T16:25:10Z"
}

{
    "group": "**",
    "level": "warning",
    "msg": "deleting group",
    "time": "2023-12-28T16:25:10Z"
}

{
    "group": "**",
    "level": "debug",
    "msg": "finding group",
    "time": "2023-12-28T16:25:10Z"
}

{
    "group": "**",
    "level": "warning",
    "msg": "deleting group",
    "time": "2023-12-28T16:25:10Z"
}

{
    "level": "info",
    "msg": "sync completed",
    "time": "2023-12-28T16:25:10Z"
}

END RequestId: e9ffd9c0-3743-4700-b0ac-b3f8a740f180
REPORT RequestId: e9ffd9c0-3743-4700-b0ac-b3f8a740f180  Duration: 2514.13 ms    Billed Duration: 2624 ms    Memory Size: 128 MB Max Memory Used: 39 MB  Init Duration: 109.13 

To Reproduce Steps to reproduce the behavior:

1. Sync the users and groups using the cli ssosync
2. Ran with args ./ssosync -c credentials.json --google-admin @.nl --identity-store-id d-* --region eu-west-1 -d
3. Deploy Serverless App with the same configuration

Expected behavior The users and groups are not deleted by the Lambda. They are synced into Identity Center.

Additional context I have verified that the configuration is the same. I also see the lambda not erroring and acting as intended.

[snorberhuis]

I can reproduce the error locally using --group-match 'name:AWS*' which is configured in the AWS Stack. This filters out all groups not beginning with AWS.

I did not configure a group match so I would expect the stack and the CLI to behave the same way.

[alexhermida]

Hey @snorberhuis, we suddenly came across the same issue you're sharing in a production environment.

We do have a group match in our deployment, and the lambda removed the users and groups, and during the next run a few minutes later, it recreated them. Do you you have any additional insights on this or did it happen to you again? 

Thank you.

[snorberhuis]

No, it did not happen to me again after I left the following parameter blank:

GoogleGroupMatch: <>

I kept the bug report open because I think the CLI should have the same default as the serverless repository. After my initial attempt for myself, I deployed this sso-sync to 2 more clients, and they all stumbled on this point.

[alexhermida]

Thanks for the quick reply 😄 ok, we're deploying the lambda ourselves instead of using the serverless application repository, so might be the case is another issue in the code. In our case we do need to keep the group match to sync only AWS groups.

Thanks again!

[ChrisPates]

That bit of context around the group match is probably key. I will need to review the code and run tests to reproduce but I think it’s likely to be related default values.

[skydoctor]

Wondering if folks have any further insight on this. We likely hit the same issue that deleted all the groups and users and then recreated them in the next tick.

[ChrisPates]

So I will need to dig further but the two configurations are not the same the first would have wildcard'd to all groups in the directory but only users that are members of those groups. The second would have synced all groups with prefix AWS*.

However, that doesn't explain the delete and recreate.

I'm just finishing feature release v2.2 which adds greater match flexibility and the next feature release I'm about to start working is looking moving all CRUD back to SCIMv2 to improve least privilege and more accurate matching.

I'll try to reproduce your experience above and build a test case from it.

[ChrisPates]

As of v2.2.x Defaults are the same in the SAR template and the cli tool.

[snorberhuis]

@alexhermida and @ChrisPates I have now also seen the same bug Alex reported. I have created a separate issue: https://github.com/awslabs/ssosync/issues/194