Closed hope-portal-services closed 2 years ago
hope-portal-services
commented
2 years ago Left this out, but after the signup succeeds, this is the result:
The user is deleted from the pool before even having a chance to enter the verification code sent to the email.
nadetastic
commented
2 years ago Hi @hope-portal-services 👋
Thanks for opening this issue, I'd like to request some additional information before diving into it.
Since you mentioned that the user is being deleted after sign up, could you take a look and share the CloudTrail event logs for a DeleteUser Cognito API action related to one of the users facing this issue? This may help in determining what/why the user is being deleted. Also can we verify that the user is actually being created by locating a corresponding AdminCreateUser API action?
Please note to redact any sensitive info 🙂
hope-portal-services
commented
2 years ago Hi @nadetastic,
So these users are being created using Amplify, ie: from the Javascript Library using Auth.SignUp(), would those events show up in CloudTrail? If so, I'm not seeing any of them. I see Events from AdminCreateUser which is used elsewhere in the application when a user is created manually, but not through the registration process.
Heres a log from the SignUp Event:
{
"eventVersion": "1.08",
"userIdentity": {
"type": "Unknown",
"principalId": "Anonymous"
},
"eventTime": "2022-09-28T16:29:20Z",
"eventSource": "cognito-idp.amazonaws.com",
"eventName": "SignUp",
"awsRegion": "us-east-1",
"sourceIPAddress": "REDACTED",
"userAgent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36",
"requestParameters": {
"clientId": "REDACTED",
"username": "HIDDEN_DUE_TO_SECURITY_REASONS",
"password": "HIDDEN_DUE_TO_SECURITY_REASONS",
"userAttributes": "HIDDEN_DUE_TO_SECURITY_REASONS"
},
"responseElements": {
"userConfirmed": false,
"codeDeliveryDetails": {
"destination": "HIDDEN_DUE_TO_SECURITY_REASONS",
"deliveryMedium": "EMAIL",
"attributeName": "email"
},
"userSub": "REDACTED"
},
"additionalEventData": {
"sub": "REDACTED"
},
"requestID": "c4825548-5c67-49f2-8aac-c4b70c84469a",
"eventID": "3353ed0b-19c7-4063-aacc-9c5c094e8325",
"readOnly": false,
"eventType": "AwsApiCall",
"managementEvent": true,
"recipientAccountId": "REDACTED",
"eventCategory": "Management",
"tlsDetails": {
"tlsVersion": "TLSv1.2",
"cipherSuite": "ECDHE-RSA-AES128-GCM-SHA256",
"clientProvidedHostHeader": "cognito-idp.us-east-1.amazonaws.com"
}
}For DeleteUser there are 0 Events in CloudTrail, I think because that function is for account users, but I AM seeing a lot of AdminDeleteUser Events.
ie:
{
"eventVersion": "1.08",
"userIdentity": {
"type": "IAMUser",
"principalId": "REDACTED",
"arn": "REDACTED",
"accountId": "REDACTED",
"accessKeyId": "REDACTED",
"userName": "REDACTED"
},
"eventTime": "2022-09-28T16:29:21Z",
"eventSource": "cognito-idp.amazonaws.com",
"eventName": "AdminDeleteUser",
"awsRegion": "us-east-1",
"sourceIPAddress": "REDACTED",
"userAgent": "aws-sdk-nodejs/2.1223.0 linux/v12.22.11 exec-env/AWS_Lambda_nodejs12.x promise",
"requestParameters": {
"userPoolId": "REDACTED",
"username": "HIDDEN_DUE_TO_SECURITY_REASONS"
},
"responseElements": null,
"additionalEventData": {
"sub": "REDACTED"
},
"requestID": "1c5d2bc6-51b3-4287-84f4-b364a8338dc6",
"eventID": "81fcb6ae-e664-4fa8-bee1-5f778dc8df29",
"readOnly": false,
"eventType": "AwsApiCall",
"managementEvent": true,
"recipientAccountId": "REDACTED",
"eventCategory": "Management",
"tlsDetails": {
"tlsVersion": "TLSv1.2",
"cipherSuite": "ECDHE-RSA-AES128-GCM-SHA256",
"clientProvidedHostHeader": "cognito-idp.us-east-1.amazonaws.com"
}
}In the video attached, toward the end you can see that the user does get created, it shows up in the user pool and then disappears on the next reload of the user pool, ie: within about 5 seconds.
This is the obvious result of trying to confirm the user with their verification code once the user has been deleted:
{
"eventVersion": "1.08",
"userIdentity": {
"type": "Unknown",
"principalId": "Anonymous"
},
"eventTime": "2022-09-28T16:29:31Z",
"eventSource": "cognito-idp.amazonaws.com",
"eventName": "ConfirmSignUp",
"awsRegion": "us-east-1",
"sourceIPAddress": "REDACTED",
"userAgent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36",
"errorCode": "UserNotFoundException",
"errorMessage": "Username/client id combination not found.",
"requestParameters": {
"clientId": "REDACTED",
"username": "HIDDEN_DUE_TO_SECURITY_REASONS",
"confirmationCode": "HIDDEN_DUE_TO_SECURITY_REASONS",
"forceAliasCreation": true
},
"responseElements": null,
"requestID": "572c01e3-4f86-4293-a71d-6badb42ee51e",
"eventID": "e89c6008-50f1-4bc4-bd07-b8e7d05239fe",
"readOnly": false,
"eventType": "AwsApiCall",
"managementEvent": true,
"recipientAccountId": "REDACTED",
"eventCategory": "Management",
"tlsDetails": {
"tlsVersion": "TLSv1.2",
"cipherSuite": "ECDHE-RSA-AES128-GCM-SHA256",
"clientProvidedHostHeader": "cognito-idp.us-east-1.amazonaws.com"
}
}@nadetastic I want to note here that while I cannot reliably duplicate this issue since its so sporadic, I'd like to note that most often the issue is seen when navigating to the page FROM a Gmail email, ie: Clicking a link that says for example: Register which navigates to my registration page.
I know Gmail manipulates its body links to be for example, https://google.com/url?q={YOUR-ORIGINAL-URL}?{GOOGLES-CUSTOM-TRACKING-PARAMETERS}
I'm wondering if any of these parameters could be conflicting with Amplify and causing some anomaly.
I did a side by side comparison of the URLs for reference: Left is Google's manipulated URL, Right is the original.
hope-portal-services
commented
2 years ago @nadetastic - Updating here:
Have been heavily testing this using Proton Mail in place of Gmail to try to rule out something within my own code, cannot duplicate this bug so far, 30+ attempts, seems specific to Gmail redirected URLs.
Edit: Switched back to Gmail, bug present on attempt 1.
hope-portal-services
commented
2 years ago @nadetastic Still cannot pinpoint this to my code, this poses a pretty huge issue considering our invite system runs on email which contains URL parameters. As long as those redirects are coming from Gmail, this issue seems to be present.
Not sure where to turn next aside from revamping the entire system to work differently, even then, this is a really inconvenient issue right now and testing every permutation of this bug is massively time consuming.
Any chance of a followup from engineering?
nadetastic
commented
2 years ago @hope-portal-services thank you for the update.
Regarding the CloudTrail logs, since we are seeing AdminDeleteUser API calls, can you confirm from your end which entity is making this call? This will be under userIdentity. This can help us trace where the trigger to remove the user is originating from
hope-portal-services
commented
2 years ago Hi @nadetastic - The identity making the call is indeed one of my application users, meant to run admin-level functions like this, but this identity is used throughout the application for AWS related functions, so it would seem normal that the request would come from the identity configured in the application. Question is, what is causing this particular function to run, it's allowed to run because it has permission to.
nadetastic
commented
2 years ago @hope-portal-services This looks like a pretty unique issue that I am not yet able to reproduce. Would you be able to provide me with a sample repo of that I can use to test with?
One thing Im curious about, is if we can confirm that your method for deleting users is indeed getting invoked. There are two ways to do this:
nadetastic
commented
2 years ago An additional note, by default, adding Admin actions doesn't setup a route for deleteUser, I am assuming that you manually added that to yourself.
hope-portal-services
commented
2 years ago Hi @nadetastic -
Correct, I use Amplify, but not the automated setup, a manual setup in this case.
I've made every change I can think of to halt this additional call to the API and I can NEVER see this route being called twice from the frontend, even went as far as to make sure dev tools stays open the entire time to see if any initial calls are sent from the frontend. Nothing.
When this anomaly occurs, if you watch the network, it only ever gets fired off one time, so this really doesn't feel like an issue coming from the frontend.
Here's a log of the bug when it happens: The functions runs so incredibly close together, it almost seems like a retry, but this function is not asynchronous, so I don't believe Lambda max retries even apply here.
For some additional insight: This function checks for a registration in progress, for example: A user begins registration and gets past the point where a cognito user is created, but has not finished registration far enough to create a user record in my database.
This function checks for that scenario, if the function finds a cognito user with the matching email, but does not find a user record, we delete the cognito user and return a Great, this email is available rather than This email is in use, etc...
This had been working as expected for years. I truly don't know what has changed here, but it feels like it's either:
At this point, I really am at a loss.
I suppose a way to duplicate the actions that I'm taking would be to run two adminGetUser functions back to back, in which case one of them is returning a user and one is not.
The obvious scenario that is happening here is:
Video of the actual process: https://user-images.githubusercontent.com/57410369/194350873-350a3718-3de0-474f-8748-be86273ecba5.mov
-- Email check Lambda
Cognito user lookup
Delete cognito user
hope-portal-services
commented
2 years ago @nadetastic - Happy to say this has been solved. Thank you for your help either way, it was helpful to hash this out here.
For reference, Google's redirects use a headless Chromium to render your URL's to check for blacklisted or malicious sites. In doing this, it means that anything that happens to run on your URL, will run. In this case, the site passed Google's checks quickly, redirected our URL and likely sits in a queue before being checked which is why it takes about 20-30 seconds to clear, in which case, we have already reached the end of registration, created our Cognito user and boom, the site renders on Google's servers, runs the email check for validity and the Cognito user is deleted.
I was able to block Google's IP's from hitting our email check Lambda and problem solved.
Feel free to close out this issue, thanks.
nadetastic
commented
2 years ago @hope-portal-services glad to hear that this was resolved. Thanks for the context on the root cause as it help me in understanding the problem and I can share with the team.
Will close this out.
Before opening, please confirm:
JavaScript Framework
React
Amplify APIs
Authentication
Amplify Categories
auth
Environment information
Describe the bug
Sporadic issue: Succeeds 75% of the time, failure 25% of the time
Expected behavior
Reproduction steps
The issue is sporadic, I have not been able to reliably duplicate.
Code Snippet
Log output
aws-exports.js
No response
Manual configuration
Additional configuration
No response
Mobile Device
No response
Mobile Operating System
No response
Mobile Browser
No response
Mobile Browser Version
No response
Additional information and screenshots
https://user-images.githubusercontent.com/57410369/192565921-2f64689c-fe1a-4a1f-bcb4-0a2f39ebb827.mp4
Video of sporadic issue, succeeds once, fails once. You can see the user created in the pool, and you can see the user immediately deleted from the pool with no additional interaction.
The register button fires one function, the one referenced above in the
Code Snippetfield.This integration has been functioning with no issues for over 3 years.