search

aws / aws-xray-daemon

The AWS X-Ray daemon listens for traffic on UDP port 2000, gathers raw segment data, and relays it to the AWS X-Ray API.
Apache License 2.0
190 stars 69 forks source link

aws sso login Error in creating session object : profile "default" is configured to use SSO but is missing required configuration: sso_region, sso_start_url #200

Open ptnutrien opened 1 year ago

ptnutrien commented 1 year ago

2023-06-01T20:14:16Z [Info] Initializing AWS X-Ray daemon 3.3.7 2023-06-01T20:14:16Z [Debug] Listening on UDP xray-daemon:2000 2023-06-01T20:14:16Z [Info] Using buffer memory limit of 78 MB 2023-06-01T20:14:16Z [Info] 1248 segment buffers allocated 2023-06-01T20:14:16Z [Debug] Using proxy address: 2023-06-01T20:14:17Z [Error] Error in creating session object : profile "default" is configured to use SSO but is missing required configuration: sso_region, sso_start_url

services: xray-daemon: command:

FROM amazonlinux RUN yum install -y unzip RUN curl -o daemon.zip https://s3.us-east-2.amazonaws.com/aws-xray-assets.us-east-2/xray-daemon/aws-xray-daemon-linux-3.x.zip RUN unzip daemon.zip && cp xray /usr/bin/xray ENTRYPOINT ["/usr/bin/xray", "-t", "0.0.0.0:2000", "-b", "0.0.0.0:2000"] EXPOSE 2000/udp EXPOSE 2000/tcp

ptnutrien commented 1 year ago

I was able to fix the first error

services: xray-daemon: command: ["-o", '--log-level=debug'] image: amazon/aws-xray-daemon volumes:

but now it is giving other errors, its like it doesn't use the sso config

Debug] processor: segment batch size: 2. capacity: 50 xray-daemon-1 | 2023-06-02T16:17:16Z [Error] Unable to sign request: NoCredentialProviders: no valid providers in chain. Deprecated. xray-daemon-1 | For verbose messaging see aws.Config.CredentialsChainVerboseErrors

ptnutrien commented 1 year ago

I am creating an xray daemon like this

xray-daemon: command: ["-o", '--log-level=dev'] image: amazon/aws-xray-daemon volumes:

I added region because I was getting an error even though region is specified in my config, I am using aws configure sso to generate config file [default] sso_session = newbie sso_account_id = ACCOUNT_ID sso_role_name = the_role region = us-east-2 output = json [sso-session newbie] sso_start_url = obfuscated sso_region = us-east-1 sso_registration_scopes = sso:account:access

However, I keep getting this error [Error] Unable to sign request: NoCredentialProviders: no valid providers in chain. Deprecated. xray-daemon-1 | For verbose messaging see aws.Config.CredentialsChainVerboseErrors

after compiliing locally and running I get this ./xray -v AWS X-Ray daemon version: xray-mac-arm64 % ./xray -o -n us-east-2 2023-06-02T13:40:20-07:00 [Info] Initializing AWS X-Ray daemon 2023-06-02T13:40:20-07:00 [Info] Using buffer memory limit of 163 MB 2023-06-02T13:40:20-07:00 [Info] 2608 segment buffers allocated 2023-06-02T13:40:20-07:00 [Error] Error in creating session object : profile "default" is configured to use SSO but is missing required configuration: sso_region, sso_start_url

But as you can see above these values are set

this pr should have fixed this? https://github.com/aws/aws-xray-daemon/issues/147

I have verified the aws sdk go from this link https://github.com/aws/aws-sdk-go/issues/4054#issuecomment-901520334

Version 1.44.275 Trying V1.... COULD NOT DO V1 profile "default" is configured to use SSO but is missing required configuration: sso_region, sso_start_url

Trying V2.... Account: ACCOUNT_ID UserID:USER ARN: arn:aws:sts::

wangzlei commented 1 year ago

Checking doc AWS SSO Support in the AWS SDK for Go v1

    session.Options{
        SharedConfigState: session.SharedConfigEnable, // Must be set to enable
        Profile:           "dev-profile",
    }

X-Ray Daemon code does contain the similar code SharedConfigEnable https://github.com/aws/aws-xray-daemon/blame/56bcdadc0e5808f4428ed6e3e54a88a2ceca2f82/pkg/conn/conn.go#L294

ptnutrien commented 1 year ago

How can I enable the session.SharedConfigEnable ? I have - AWS_SDK_LOAD_CONFIG=1 I don't understand how to set the session.Options for this library to be able to use the SharedConfigState

I have reviewed the doc you shared as well as https://docs.aws.amazon.com/xray/latest/devguide/xray-daemon-configuration.html https://docs.aws.amazon.com/sdk-for-go/api/aws/session/

quetzalcoatl commented 1 year ago

I'm struggling with setting up a local instance of xra-daemon on my Windows developer machine. A bit of a different issue, but similar symptoms, so I thought I'll pass you what I found.

The package I'm using is: https://s3.us-east-2.amazonaws.com/aws-xray-assets.us-east-2/xray-daemon/aws-xray-daemon-windows-process-3.x.zip

My set up is, basically, from scratch (remove all profiles from ~/.aws/config, remove ~/.aws/sso/cache completely): (yes, I said I'm on Windows and now you see I'm using / and ~ -- that's because I'm using git-bash, not cmd.exe).

aws configure sso --no-browser --profile my-profile
#   it asks for data, I provide (data below is anonymized)
#session:    sess01
#sso url:     https://xxxxxxx.awsapps.com/start#/
#account:  123456789
#role:         MyDesiredRole
#region:     eu-west-1
#output:     yaml

#then, some time later
aws sso login --no-browser --profile my-profile
# it gives me a browser link, I open it, and accept

After that, I'm able to use aws cli etc. However, when I run xray-process-3.x, it provides the same error as reported here:

> export AWS_PROFILE=my-profile
> ./xray_windows.exe -l dev -o -n eu-west-1
2023-07-12T12:12:41+02:00 [Info] Initializing AWS X-Ray daemon 3.3.7
2023-07-12T12:12:41+02:00 [Debug] Listening on UDP 127.0.0.1:2000
2023-07-12T12:12:41+02:00 [Info] Using buffer memory limit of 324 MB
2023-07-12T12:12:41+02:00 [Info] 5184 segment buffers allocated
2023-07-12T12:12:41+02:00 [Debug] Using proxy address:
2023-07-12T12:12:41+02:00 [Debug] Fetch region eu-west-1 from commandline/config file
2023-07-12T12:12:41+02:00 [Error] Error in creating session object : profile "my-profile" is configured to use SSO but is missing required configuration: sso_region, sso_start_url

I've got an idea, and I edited the ~/.aws/config and literally copied sso_region, sso_start_url sections from sso-session sess01 into profile my-profile:

[default]
region = eu-north-1
[profile my-profile]
sso_session = sess01
sso_account_id = 1234567890
sso_role_name = MyDesiredRole
region = eu-west-1
output = yaml
sso_start_url = https://xxxxxxx.awsapps.com/start#/
sso_region = eu-west-1
[sso-session sess01]
sso_start_url = https://xxxxxxx.awsapps.com/start#/
sso_region = eu-west-1
sso_registration_scopes = sso:account:access

and this change allowed the xray-windows.exe to start!

> export AWS_PROFILE=my-profile
> ./xray_windows.exe -l dev -o -n eu-west-1
xray_windows.exe -o -n eu-west-1
2023-07-12T12:19:04+02:00 [Info] Initializing AWS X-Ray daemon 3.3.7
2023-07-12T12:19:04+02:00 [Info] Using buffer memory limit of 324 MB
2023-07-12T12:19:04+02:00 [Info] 5184 segment buffers allocated
2023-07-12T12:19:04+02:00 [Info] Using region: eu-west-1
2023-07-12T12:19:04+02:00 [Info] HTTP Proxy server using X-Ray Endpoint : https://xray.eu-west-1.amazonaws.com
2023-07-12T12:19:04+02:00 [Info] Starting proxy http server on 127.0.0.1:2000

After I got this, I accidentaly found this similar report elsewhere, and the person there did almost exactly the same: https://github.com/gruntwork-io/terragrunt/issues/2604#issuecomment-1588485327 This post may be important. It mentions that the tools are picky about the SSO URL, and that they had to keep an eye on #/ at the end of it.

However, that's not the end of it, at least for me. When my application actually did something and tried sending traces/segments to this now-finally-running xray-daemon, the daemon said:

2023-07-12T12:19:06+02:00 [Error] Sending segment batch failed with: SSOProviderInvalidToken: the SSO session has expired or is invalid
caused by: open C:\Users\xxxxxxx\.aws\sso\cache\6b818xxxxxxxxx.json: The system cannot find the file specified.

But on my computer, I had different files there:

$ ls sso/cache/
4915fxxxxxxxxxxxxxx.json  87bbbxxxxxxxxxxxx.json

I played a bit, and discovered that the hash-name of the file wanted by xray-daemon is related to the sso url from the profile (not the sso-session) from the ~/.aws/config file.

    https://xxxxxx.awsapps.com/start#/    -> looks for: 6b818xxxxxx.json
    https://xxxxxx.awsapps.com/start      -> looks for: b382dxxxxxx.json
    https://xxxxxx.awsapps.com/start#     -> looks for: 03350xxxxxx.json

So, for some unknown reason, the xray-daemon reads the config properly (well, almost, it ignores sso-session, and gets all properties from parent profile), and then it generates a hashname for the credentials file, but generates it differently than the aws sso login did.

I tried playing around with copying files. I had those 4915fxxxxxxxxxxxxxx.json 87bbbxxxxxxxxxxxx.json files, 4915 didn't work, but 87bb worked:

> cp ~/.aws/sso/cache/87bbbxxxxxxxxx.json ~/.aws/sso/cache/6b818xxxxxxxxx.json
> export AWS_PROFILE=my-profile
> xray_windows.exe -l dev -o -n eu-west-1
2023-07-12T12:52:33+02:00 [Info] Initializing AWS X-Ray daemon 3.3.7
2023-07-12T12:52:33+02:00 [Debug] Listening on UDP 127.0.0.1:2000
2023-07-12T12:52:33+02:00 [Info] Using buffer memory limit of 324 MB
2023-07-12T12:52:33+02:00 [Info] 5184 segment buffers allocated
2023-07-12T12:52:33+02:00 [Debug] Using proxy address:
2023-07-12T12:52:33+02:00 [Debug] Fetch region eu-west-1 from commandline/config file
2023-07-12T12:52:33+02:00 [Info] Using region: eu-west-1
2023-07-12T12:52:33+02:00 [Debug] ARN of the AWS resource running the daemon:
2023-07-12T12:52:33+02:00 [Debug] No hostname set for telemetry records
2023-07-12T12:52:33+02:00 [Debug] No Instance Id set for telemetry records
2023-07-12T12:52:33+02:00 [Debug] Using Endpoint: https://xray.eu-west-1.amazonaws.com
2023-07-12T12:52:33+02:00 [Debug] Telemetry initiated
2023-07-12T12:52:33+02:00 [Info] HTTP Proxy server using X-Ray Endpoint : https://xray.eu-west-1.amazonaws.com
2023-07-12T12:52:33+02:00 [Debug] Using Endpoint: https://xray.eu-west-1.amazonaws.com
2023-07-12T12:52:33+02:00 [Debug] Batch size: 50
2023-07-12T12:52:33+02:00 [Info] Starting proxy http server on 127.0.0.1:2000
2023-07-12T12:52:37+02:00 [Debug] processor: sending partial batch
2023-07-12T12:52:37+02:00 [Debug] processor: segment batch size: 1. capacity: 50
2023-07-12T12:52:38+02:00 [Info] Successfully sent batch of 1 segments (0.772 seconds)
2023-07-12T12:53:33+02:00 [Debug] Send 1 telemetry record(s)

So, it works now. Yay! It may look simple here, because it's trimmed down to the point, but it took me a lot of hours over several sessions fo trying to get it working (including running a linux version in a docker container, or trying the window-service version, etc -- all had the same initial behavior of ignoring the sso-session and only looking at the profile).

It seems the solution is 100% repeatable. If I observe the logs of xray-daemon and read from the error line what was the hash-name.json of the expected credential file, and I can copy the correct credential file so that expected file exists, and it works beautifuly.

But - I had 2 credential files. Here's how they looked like (anonymized ofc, and FORMATTED):

4915fxxxxxxxxxxxxxx.json:
{
  "clientId": "xxxxx some hash or token",
  "clientSecret": "xxxxx typical JWT token",
  "expiresAt": "2023-10-10T10:13:56Z",
  "scopes": ["sso:account:access"]
}
87bbbxxxxxxxxxxxx.json:
{
  "startUrl": "https://xxxxxx.awsapps.com/start#/",
  "region": "eu-west-1",
  "accessToken": "xxxxxx some hash or token",
  "expiresAt": "2023-07-12T11:14:06Z",
  "clientId": "xxxx some hash or token, THE SAME as in the other file",
  "clientSecret": "xxxxx typical JWT token, THE SAME as in the other file",
  "registrationExpiresAt": "2023-10-10T10:13:56Z",   ## note that it is the same as "expiresAt" in the other file
  "refreshToken": "xxxx some hash or token"
}

The xray daemon wanted the second file. That one having attributes like registrationExpiresAt or refreshToken. Judging by names and expiry dates, I guess that the first file are the 'profile registration' and that the second one (wanted by xray daemon) is 'current session', or something like that.

So, the full workaround was, for me:

HTH