search

aws / aws-xray-daemon

The AWS X-Ray daemon listens for traffic on UDP port 2000, gathers raw segment data, and relays it to the AWS X-Ray API.
Apache License 2.0
190 stars 68 forks source link

Windows could not start the AWSXrayDaemon service on Local Computer #16

Open abivolmv opened 5 years ago

abivolmv commented 5 years ago

I get this error when trying to setup and start xray daemon on local Windows 7:

Error 1067: The process terminated unexpectedly

Logs :

2018-10-10T09:55:41+02:00 [Info] Initializing AWS X-Ray daemon 3.0.0 2018-10-10T09:55:41+02:00 [Info] Using buffer memory limit of 1309 MB 2018-10-10T09:55:41+02:00 [Info] 20944 segment buffers allocated 2018-10-10T09:55:41+02:00 [Error] Unable to retrieve the region from the EC2 instance RequestError: send request failed caused by: Get http://169.254.169.254/latest/meta-data/placement/availability-zone: dial tcp 169.254.169.254:80: connectex: A socket operation was attempted to an unreachable network.

2018-10-10T09:55:41+02:00 [Error] Cannot fetch region variable from config file, environment variables and ec2 metadata.

This is my modified cfg.yaml :

# Maximum buffer size in MB (minimum 3). Choose 0 to use 1% of host memory.
TotalBufferSizeMB: 0
# Maximum number of concurrent calls to AWS X-Ray to upload segment documents.
Concurrency: 8
# Send segments to AWS X-Ray service in a specific region
Region: "eu-west-1"
# Change the X-Ray service endpoint to which the daemon sends segment documents.
Endpoint: ""
Socket:
  # Change the address and port on which the daemon listens for UDP packets containing segment documents.
  UDPAddress: "127.0.0.1:2000"
  # Change the address and port on which the daemon listens for HTTP requests to proxy to AWS X-Ray.
  TCPAddress: "127.0.0.1:2000"
Logging:
  LogRotation: true
  # Change the log level, from most verbose to least: dev, debug, info, warn, error, prod (default).
  LogLevel: "prod"
  # Output logs to the specified file path.
  LogPath: ""
# Turn on local mode to skip EC2 instance metadata check.
LocalMode: true
# Amazon Resource Name (ARN) of the AWS resource running the daemon.
ResourceARN: ""
# Assume an IAM role to upload segments to a different account.
RoleARN: ""
# Disable TLS certificate verification.
NoVerifySSL: true
# Upload segments to AWS X-Ray through a proxy.
ProxyAddress: ""
# Daemon configuration file format version.
Version: 2

I also have set the env var AWS_DEFAULT_REGION

yogiraj07 commented 5 years ago

Hi @abivolmv , From the error log, the daemon is not able to read region value. The daemon searches for the region value at from this location. Parsing logic : Link

From the error log, it looks like the configured cfg.yaml file is not read by the daemon.

Are you using executable for Windows service?

If yes, it is possible because of the user used to start the service. The daemon service looks for the file in the directory, where the service is running and not in the custom directory, where you are running the script.

Usually, the windows services run from C:\Windows\System32 location. You would need to place configured cfg.yaml file at this location and start the daemon service.

Alternatively, you can pass command line option -c with the absolute path to cfg.yaml file.

Note -c parameter in the below script.

sc.exe create AWSXRayDaemon binPath= "$daemonPath -f $daemonLogPath -c $configPath"

Please let me know if you have questions. Thanks, Yogi

abivolmv commented 5 years ago

Hi , I used the -c flag and it worked. But now I get this error (tell me if I should open a new issue for that) :

2018-10-11T09:22:12+02:00 [Info] Initializing AWS X-Ray daemon 3.0.0
2018-10-11T09:22:12+02:00 [Debug] Listening on UDP 127.0.0.1:2000
2018-10-11T09:22:12+02:00 [Info] Using buffer memory limit of 1309 MB
2018-10-11T09:22:12+02:00 [Info] 20944 segment buffers allocated
2018-10-11T09:22:12+02:00 [Debug] Using proxy address: 
2018-10-11T09:22:12+02:00 [Debug] Fetch region eu-west-1 from commandline/config file
2018-10-11T09:22:12+02:00 [Info] Using region: eu-west-1
2018-10-11T09:22:12+02:00 [Debug] ARN of the AWS resource running the daemon: 
2018-10-11T09:22:12+02:00 [Debug] No Metadata set for telemetry records
2018-10-11T09:22:12+02:00 [Debug] Using Endpoint: https://xray.eu-west-1.amazonaws.com
2018-10-11T09:22:12+02:00 [Debug] Telemetry initiated
2018-10-11T09:22:12+02:00 [Debug] Using Endpoint: https://xray.eu-west-1.amazonaws.com
2018-10-11T09:22:12+02:00 [Debug] Batch size: 50
2018-10-11T09:22:12+02:00 [Info] Starting proxy http server on 127.0.0.1:2000
2018-10-11T09:23:02+02:00 [Error] Unable to sign request: NoCredentialProviders: no valid providers in chain. Deprecated.
    For verbose messaging see aws.Config.CredentialsChainVerboseErrors
2018-10-11T09:23:03+02:00 [Debug] processor: sending partial batch
2018-10-11T09:23:03+02:00 [Debug] processor: segment batch size: 1. capacity: 50
2018-10-11T09:23:03+02:00 [Error] Sending segment batch failed with: NoCredentialProviders: no valid providers in chain. Deprecated.
    For verbose messaging see aws.Config.CredentialsChainVerboseErrors
2018-10-11T09:23:03+02:00 [Warn] Delaying sending of additional batches by 0 seconds
2018-10-11T09:23:12+02:00 [Debug] Failed to send telemetry 1 record(s). Re-queue records. NoCredentialProviders: no valid providers in chain. Deprecated.
    For verbose messaging see aws.Config.CredentialsChainVerboseErrors
2018-10-11T09:24:12+02:00 [Debug] Failed to send telemetry 2 record(s). Re-queue records. NoCredentialProviders: no valid providers in chain. Deprecated.
    For verbose messaging see aws.Config.CredentialsChainVerboseErrors
2018-10-11T09:25:13+02:00 [Debug] Failed to send telemetry 3 record(s). Re-queue records. NoCredentialProviders: no valid providers in chain. Deprecated.
    For verbose messaging see aws.Config.CredentialsChainVerboseErrors
2018-10-11T09:26:08+02:00 [Debug] Segment batch: done!

It works when I run the process exe directly in a cmd : xray_windows.exe -o -n eu-west-1 But I want it as a service.

yogiraj07 commented 5 years ago

Hi @abivolmv , The reason for above error is, the AWSXRayDaemon service is not able to access AWS Credentials while sending the segment. This may be due to user permissions.

Please make sure, your AWSXRayDaemon service has right permissions to access ~/.aws folder.

Alternatively, you can set environment variable for AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY, the daemon will automatically pick credentials.

Thanks, Yogi

abivolmv commented 5 years ago

Thanks for the hint : I added reading permission to LOCAL SERVICE user on ~/.aws and it is now running.

abivolmv commented 5 years ago

No , actually I was to happy to close the issue. After adding that permission the service is starting but when there is a batch to be sent the same error appears :

2018-10-12T11:52:30+02:00 [Error] Unable to sign request: NoCredentialProviders: no valid providers in chain. Deprecated.
    For verbose messaging see aws.Config.CredentialsChainVerboseErrors
2018-10-12T11:52:31+02:00 [Error] Sending segment batch failed with: NoCredentialProviders: no valid providers in chain. Deprecated.
    For verbose messaging see aws.Config.CredentialsChainVerboseErrors
2018-10-12T11:52:31+02:00 [Warn] Delaying sending of additional batches by 0 seconds
abivolmv commented 5 years ago

Setting the env vars worked. But is it possible to do it without the env vars ?

yogiraj07 commented 5 years ago

Hi @abivolmv , The X-Ray daemon service under the hood uses AWS Go SDK to fetch credentials : Link

The AWS CLI uses credentials at path > dir "%UserProfile%\.aws". Refer : https://docs.aws.amazon.com/sdk-for-go/v1/developer-guide/configuring-sdk.html

Another option can be adding your above directory path to PATH environment variable.

Thanks, Yogi

rbanerjiaws commented 5 years ago

I am getting the following which i snot exactly the same as the errors above. Any idea why? My credentials are at "%UserProfile%.aws". My client application in .Net Core picks them up just fine.

2018-10-18T18:28:02-04:00 [Debug] Failed to send telemetry 30 record(s). Re-queue records. NoCredentialProviders: no valid providers in chain. Deprecated.

yogiraj07 commented 5 years ago

Hi @rbanerjiaws , The reason for above error possibly is aws credentials are not available to the X-Ray daemon service.

For .NET Core application, AWS Dotnet sdk picks the credentials from %UserProfile%.aws location. However, for X-Ray daemon which runs as a service locally, requires aws credentials to be made available using above methods as discussed, if the service is not able to recognize the aws credentials automatically.

  1. Can you try setting the environment variables for aws credentials as discussed above.
  2. Telemetry records reports daemon statistics to X-Ray service at regular interval. After step 1, can you check from the daemon logs whether the daemon is able to send Segments and Telemetry records? (you can follow script mentioned in the thread)

Please let me know if you have more questions.

Thanks, Yogi

rbanerjiaws commented 5 years ago

It worked with setting environment variables. Just a minor note, the system environment variable needs to be set not the user's.

arcban02 commented 5 years ago

I have followed everything on this thread but still getting below error:

Error] Unable to sign request: NoCredentialProviders: no valid providers in chain. Deprecated. For verbose messaging see aws.Config.CredentialsChainVerboseErrors

awssandra commented 5 years ago

Hi, arcban02

Looks like a pretty common case: https://github.com/aws/aws-sdk-go/issues/992

As per previous comment: The AWS CLI uses credentials at path > dir "%UserProfile%.aws". Refer : https://docs.aws.amazon.com/sdk-for-go/v1/developer-guide/configuring-sdk.html

Can you post the daemon logs, I'd like to take a deeper look.

Can you confirm you have the ~/.aws folder, with a credentials file? Can you confirm the AWSXRayDaemon service has right permissions to access ~/.aws folder? Can you confirms the directory path above is on the PATH environment variable? Did you try setting the environment variables, AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY?

Thanks, Sandra

arcban02 commented 5 years ago

Hi Sandra,

Thanks for looking into it. Here are my daemon logs:

2019-07-29T10:53:22-07:00 [Info] Initializing AWS X-Ray daemon 3.1.0 2019-07-29T10:53:22-07:00 [Debug] Listening on UDP 127.0.0.1:2000 2019-07-29T10:53:22-07:00 [Info] Using buffer memory limit of 203 MB 2019-07-29T10:53:22-07:00 [Info] 3248 segment buffers allocated 2019-07-29T10:53:22-07:00 [Debug] Using proxy address: 2019-07-29T10:53:22-07:00 [Debug] Fetch region us-west-2 from commandline/config file 2019-07-29T10:53:22-07:00 [Info] STS Endpoint : https://sts.us-west-2.amazonaws.com 2019-07-29T10:53:42-07:00 [Info] Using region: us-west-2 2019-07-29T10:53:42-07:00 [Debug] ARN of the AWS resource running the daemon: 2019-07-29T10:53:42-07:00 [Debug] No Metadata set for telemetry records 2019-07-29T10:53:42-07:00 [Debug] Using Endpoint: https://xray.us-west-2.amazonaws.com 2019-07-29T10:53:42-07:00 [Debug] Telemetry initiated 2019-07-29T10:53:42-07:00 [Info] HTTP Proxy server using X-Ray Endpoint : https://xray.us-west-2.amazonaws.com 2019-07-29T10:53:42-07:00 [Debug] Using Endpoint: https://xray.us-west-2.amazonaws.com 2019-07-29T10:53:42-07:00 [Debug] Batch size: 50 2019-07-29T10:53:42-07:00 [Info] Starting proxy http server on 127.0.0.1:2000 2019-07-29T10:54:18-07:00 [Debug] Received request on HTTP Proxy server : /GetSamplingRules 2019-07-29T10:54:36-07:00 [Debug] processor: sending partial batch 2019-07-29T10:54:36-07:00 [Debug] processor: segment batch size: 1. capacity: 50 2019-07-29T10:54:38-07:00 [Error] Unable to sign request: NoCredentialProviders: no valid providers in chain. Deprecated. For verbose messaging see aws.Config.CredentialsChainVerboseErrors 2019-07-29T10:54:58-07:00 [Error] Sending segment batch failed with: NoCredentialProviders: no valid providers in chain. Deprecated. For verbose messaging see aws.Config.CredentialsChainVerboseErrors 2019-07-29T10:54:58-07:00 [Warn] Delaying sending of additional batches by 0 seconds 2019-07-29T10:55:19-07:00 [Debug] Failed to send telemetry 1 record(s). Re-queue records. NoCredentialProviders: no valid providers in chain. Deprecated. For verbose messaging see aws.Config.CredentialsChainVerboseErrors

I have the ~/.aws folder with credentials file.

Directory path is on PATH environment variable I have also set setting the environment variables, AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY However I am not sure how to confirm the AWSXRayDaemon service has right permissions to access ~/.aws folder Thanks.

awssandra commented 5 years ago

Hi arcban02,

If AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY are set, it shouldn't need to read in the credentials file. That's essentially an override - I'm really curious why that didn't work.

Confirm you have the following snippet/formats correct (and are set correctly in the env variables)

Example credentials file

[default]
aws_access_key_id = ******************** (20 characters, upper case)
aws_secret_access_key =  **************************************** (40 characters, uppercase and lowercase letters, symbols)

More on the credentials file format: https://docs.aws.amazon.com/sdk-for-go/v1/developer-guide/configuring-sdk.html#specifying-credentials

arcban02 commented 5 years ago

Yes I do.

awssandra commented 5 years ago

Let me see what other information I can dig up.

arcban02 commented 5 years ago

I think the daemon starts fine . The problem starts when I run a sample program that list buckets in S3. I do a Aws::AssumeRoleCredentials. I am expecting to see traces of that call captured by Xray in daemon log. That is when I see [Error] Sending segment batch failed with: NoCredentialProviders: no valid providers in chain. Deprecated.

awssandra commented 5 years ago

The root cause is still the same, as per the previous entry from user abivolmv:

2018-10-11T09:23:02+02:00 [Error] Unable to sign request: NoCredentialProviders: no valid providers in chain. Deprecated.
    For verbose messaging see aws.Config.CredentialsChainVerboseErrors
2018-10-11T09:23:03+02:00 [Debug] processor: sending partial batch
2018-10-11T09:23:03+02:00 [Debug] processor: segment batch size: 1. capacity: 50
2018-10-11T09:23:03+02:00 [Error] Sending segment batch failed with: NoCredentialProviders: no valid providers in chain. Deprecated.
    For verbose messaging see aws.Config.CredentialsChainVerboseErrors

I do not think it's related to the AssumeRole call. Does it work without the Assume role?

arcban02 commented 5 years ago

I think you are right.

toleabivol commented 5 years ago

@arcban02 did you set the system's or user's environment variables ? It must be set on system in order to work (don't ask me why). See @rbanerjiaws comment above.

arcban02 commented 5 years ago

Yes I have System environment variables set.

abivolmv commented 1 year ago

Is anyone looking into this ?

quetzalcoatl commented 1 year ago

I was able to finally run the xray daemon process locally on windows, on my development machine. It's a bit different, but maybe what I found will help you as well.

I used "xray-process" package instead of "xray-service". I suppose the "process" is meant to be used as a one-time instance, ran manually, as opposed to "service" which is meant to be installed as a windows service.

https://s3.us-east-2.amazonaws.com/aws-xray-assets.us-east-2/xray-daemon/aws-xray-daemon-windows-process-3.x.zip https://s3.dualstack.eu-west-1.amazonaws.com/aws-xray-assets.eu-west-1/xray-daemon/aws-xray-daemon-windows-service-3.x.zip

The "process" seems to work in a similar way. It also looks for a %USERPROFILE%/.aws folder and expects a config file there, similarly to what was mentioned earlier here in this thread.

I'm not using access keys. As a developer, I already have aws cli working fine, after aws sso login. I wanted a one-off xray daemon instance for debugging, and I wanted it to use my current authentication session.

It turned out that the current xray daemon doesn't support how the profile/sso/session are organized in .aws, but it could be adjusted manually, and then it ran with no issues.

Here are all the details: https://github.com/aws/aws-xray-daemon/issues/200#issuecomment-1632311913