Assume Role issue - when using Assume Role in Console application inside EC2 instance it is able to retrieve temporary credentials using metadata Service. but not able to connect to metadata service when same code used inside windows Service

APTest01 commented 1 year ago

Describe the bug

Assume Role issue - when using Assume Role in Console application inside EC2 instance it is able to retrieve temporary credentials using metadata Service. but not able to connect to metadata service when same code used inside windows Service , it again was able to connect to metadata service when fiddler was open. Please let me know what might be the issue . Below are the error and inner errors which is thrown when using inside windows service.

One or more errors occurred. Amazon.Runtime.AmazonServiceException: Unable to get IAM security credentials from EC2 Instance Metadata Service. at Amazon.Runtime.DefaultInstanceProfileAWSCredentials.FetchCredentials() at Amazon.Runtime.DefaultInstanceProfileAWSCredentials.GetCredentials() at Amazon.Runtime.DefaultInstanceProfileAWSCredentials.GetCredentialsAsync() at Amazon.Runtime.Internal.CredentialsRetriever.d7`1.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at Amazon.Runtime.Internal.RetryHandler.d101.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at Amazon.Runtime.Internal.RetryHandler.d__101.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Amazon.Runtime.Internal.CallbackHandler.d9`1.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Amazon.Runtime.Internal.CallbackHandler.d91.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Amazon.Runtime.Internal.ErrorCallbackHandler.d__51.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Amazon.Runtime.Internal.MetricsHandler

Expected Behavior

My use case is retrieving AWS Temporary Credentials using Metadata service inside EC2 instance . which is also the expected behaviour.

Current Behavior

Not able to retrieve Temporary Credentials when code is run inside windows service , but works completely fine when run in a Console Application . or able to run inside windows service when fiddler is open and running .

Reproduction Steps

AmazonSecurityTokenServiceClient client = new AmazonSecurityTokenServiceClient(); var response = client.AssumeRoleAsync(new AssumeRoleRequest { RoleArn = "rolearn", RoleSessionName = "sessionname" }); Amazon.SecurityToken.Model.Credentials credentials = response.Result.Credentials;

Possible Solution

No response

Additional Information/Context

No response

AWS .NET SDK and/or Package version used

AWSSDK.SecurityToken AWSSDK..Core

Targeted .NET Platform

.NET Framework 4.8

Operating System and version

Windows 10

ashishdhingra commented 1 year ago

Hi @APTest01,

Good afternoon.

I'm unsure why retrieving credentials from Windows Service application when running Fiddler might work as opposed to running the service normally. For getting the credentials, endpoint http://169.254.169.254/iam/security-credentials needs to be invoked. Fiddler is a web proxy and is intercepting requests and responses, acting as a middleman. It is quite possible that the security for your Windows Service is hardened (refer Restricting Service) which might be preventing code running under the context of Windows Service to access EC2 Instance Metadata. It might work with Fiddler since it is acting like a proxy.

You might want to check any security rules in Windows Firewall on EC2 instance for your Windows service.

Thanks, Ashish

APTest01 commented 1 year ago

hi @ashishdhingra I Tried totally turning off windows firewall altogether to test , but still getting the same error . ( Unable to get IAM security credentials from EC2 Instance Metadata Service.).

ashishdhingra commented 1 year ago

@APTest01 Good afternoon. Unfortunately, I'm unable to reproduce the issue. Below are the reproduction steps.

Using Visual Studio 2022, created a new Windows Service (.NET Framework) project named TestWindowsService.
- In Designer, added a new EventLog control named eventLogInstance from toolbox.
- Renamed the service code file to TestAwsWindowsService.cs and opened code view for the service to modify the code to below:
```
using Amazon.S3;
using System;
using System.Diagnostics;
using System.ServiceProcess;
using System.Timers;
```

namespace TestWindowsService { public partial class TestAwsWindowsService : ServiceBase { private int eventId = 1;

    public TestAwsWindowsService()
    {
        InitializeComponent();

        eventLogInstance = new EventLog();
        if (!EventLog.SourceExists("TestAwsWindowsServiceSource"))
        {
            EventLog.CreateEventSource("TestAwsWindowsServiceSource", "TestAwsWindowsServiceLog");
        }
        eventLogInstance.Source = "TestAwsWindowsServiceSource";
        eventLogInstance.Log = "TestAwsWindowsServiceLog";

    }

    protected override void OnStart(string[] args)
    {
        eventLogInstance.WriteEntry("In OnStart.");

        // Set up a timer that triggers every minute.
        Timer timer = new Timer
        {
            Interval = 60000 // 60 seconds
        };
        timer.Elapsed += new ElapsedEventHandler(this.OnTimer);
        timer.Start();

    }

    private void OnTimer(object sender, ElapsedEventArgs e)
    {
        eventLogInstance.WriteEntry("Monitoring AWS buckets", EventLogEntryType.Information, eventId++);
        try
        {
            using (AmazonS3Client client = new AmazonS3Client())
            {
                eventLogInstance.WriteEntry("Listing buckets...");
                var response = client.ListBuckets();
                foreach (var bucket in response.Buckets)
                {
                    eventLogInstance.WriteEntry(bucket.BucketName);
                }
            }
        }
        catch (Exception ex)
        {
            eventLogInstance.WriteEntry("Exception occurred: " + ex.Message);
            eventLogInstance.WriteEntry(ex.StackTrace);
            var tempEx = ex;

            while (ex.InnerException != null)
            {
                tempEx = ex.InnerException;
                eventLogInstance.WriteEntry("Exception occurred: " + tempEx.Message);
                eventLogInstance.WriteEntry(tempEx.StackTrace);
            }
        }
    }

    protected override void OnStop()
    {
        eventLogInstance.WriteEntry("In OnStop.");
    }
}

}


  - Rebuild the project.
- Launched a new Windows EC2 instance:
  - Ensured that IMDS is enabled (default setting).
  - Attached IAM role with the following permission policy `AmazonS3ReadOnlyAccess`.
  - Attached Key Pair at launch so that Windows **Administrator** password could be decrypted for connecting via remote desktop.
- Connected to EC2 instance using Remote Desktop.
  - Installed PowerShell 7.x on the EC2 instance.
  - Copied the `Debug` or `Release` folder build for the Windows service project to EC2 instance.
  - Opened PowerShell 7 prompt.
    - Installed new service using the command `New-Service -Name TestWindowsSevice -BinaryPathName <<absolute-path-to-windows-service-exe>>\Debug\TestWindowsService.exe`. This installs service to be executed under `Local System` account.
  - Started the Windows service using Windows Services console.
  - Opened Event Viewer and navigated to `Applications and Services Logs` > `TestAwsWindowsServiceLog`.
  **RESULT:** Diagnostic logging is logged mentioning successful listing of S3 buckets.

Please check:
- If you have IMDS enabled for your EC2 instance and attached proper permissions.
- I'm unsure why your service would work when Fiddler is running. It might also be a worthwhile checking the service run as user account, might be it is different from `Local System` account and has some restrictions to make HTTP calls. My hunch is that Fiddler is a web proxy that intercepts the HTTP requests. So when it is launched, it's intercepting AWS HTTP calls which run under current logged on user account (not the service run as user account).

Thanks,
Ashish

github-actions[bot] commented 1 year ago

This issue has not received a response in 5 days. If you want to keep this issue open, please just leave a comment below and auto-close will be canceled.

aws / aws-sdk-net