search

aws / amazon-ssm-agent

An agent to enable remote management of your EC2 instances, on-premises servers, or virtual machines (VMs).
https://aws.amazon.com/systems-manager/
Apache License 2.0
1.04k stars 323 forks source link

Error when calling AWS APIs - UpdateInstanceInformation #340

Closed eddelac closed 3 years ago

eddelac commented 3 years ago

Hi

I did a fresh installation of SSM on a VPS server, the hybrid activation works fine, but after it appears this on the error log: /var/log/amazon/ssm/amazon-ssm-agent.log

cat /var/log/amazon/ssm/amazon-ssm-agent.log
2021-01-07 23:29:33 INFO [ssm-agent-worker] [HealthCheck] increasing error count by 1
2021-01-07 23:29:33 ERROR [ssm-agent-worker] [HealthCheck] error when calling AWS APIs. error details - InvalidParameter: 1 validation error(s) found.
- minimum field size of 1, UpdateInstanceInformationInput.IPAddress.

It was working in the same VPS with a previous version,

details for /var/log/amazon/ssm/errors.log

cat /var/log/amazon/ssm/errors.log
2021-01-07 23:39:33 ERROR [HandleAwsError @ awserr.go.49] [ssm-agent-worker] [HealthCheck] error when calling AWS APIs. error details - InvalidParameter: 1 validation error(s) found.
- minimum field size of 1, UpdateInstanceInformationInput.IPAddress.

I don't know if the interface is being marked as loopback, the Public IP is set on venet0:0 (2 last octets replaced for X)

ifconfig
lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 660  bytes 62596 (62.5 KB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 660  bytes 62596 (62.5 KB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

venet0: flags=211<UP,BROADCAST,POINTOPOINT,RUNNING,NOARP>  mtu 1500
        inet 127.0.0.1  netmask 255.255.255.255  broadcast 0.0.0.0  destination 127.0.0.1
        inet6 ::2  prefixlen 128  scopeid 0x80<compat,global>
        inet6 2620:1e8:3:43::c82a  prefixlen 64  scopeid 0x0<global>
        unspec 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00  txqueuelen 0  (UNSPEC)
        RX packets 60527  bytes 124634267 (124.6 MB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 25997  bytes 2494642 (2.4 MB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

venet0:0: flags=211<UP,BROADCAST,POINTOPOINT,RUNNING,NOARP>  mtu 1500
        inet 172.110.X.X  netmask 255.255.255.255  broadcast 172.110.7.10  destination 172.110.7.10
        unspec 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00  txqueuelen 0  (UNSPEC)

It appears to be thrown by this function

amazon-ssm-agent/agent/platform/platform.go

The information of the Managed Instance appear like this on SSM

ActivationId                           : 571fe858-42cf-4d33-98cc-f88fxxxxxxx
AgentVersion                           :
AssociationOverview                    :
AssociationStatus                      :
ComputerName                           :
IamRole                                : service-role/AmazonEC2RunCommandRoleForManagedInstances
InstanceId                             : mi-08de742c340xxxxx
IPAddress                              :
IsLatestVersion                        : False
LastAssociationExecutionDate           : 1/1/0001 12:00:00 AM
LastPingDateTime                       : 1/1/0001 12:00:00 AM
LastSuccessfulAssociationExecutionDate : 1/1/0001 12:00:00 AM

Is there any workaround for this scenario?

ferkhat-aws commented 3 years ago

We filter out interfaces that are either not up or are a loopback/pointtopoint interfaces (check code here).

The three interfaces you have are either loopbacks or point to point, so they will all get filtered out. Thus, when calling UpdateInstanceInformation, there will be no IP and a validation error will be thrown.

eddelac commented 3 years ago

Is there a way to bypass this validation? This worked in the past. (Installed around 2018, working ok, server was wipped out.) AFIK the setup of the VPS server hasn't changed.

The only change has been in the latest available version of the client. Downgrading to a previous version might work?

Thanks for any advise you might have, I assume that this is an atypical network setup, but I have little control of the interfaces.

ferkhat-aws commented 3 years ago

Do you know which version of SSM Agent you were using before?

eddelac commented 3 years ago

Hi there,

I've installed the version (2.2.546.0) - Released 11 May 2018, and now the instance registers correctly.

As I said, the network setup hasn't changed, although the instance registers with the 127.0.0.1 as IP Address the run command works.

I want to use the remote session manager, so I would like to know how to bypass the validation on the first run of the new version if possible.

ferkhat-aws commented 3 years ago

Hello,

Since we cannot update the code to no longer filter out P2P interfaces as this could have backward compatibility issues, our suggestion would be for you to build SSM Agent for your use case by modifying this line.

The instructions on how to build the SSM Agent yourself can be found in the README.md file.

Thanks, Faho Shubladze

eddelac commented 3 years ago

Hello,

Since we cannot update the code to no longer filter out P2P interfaces as this could have backward compatibility issues, our suggestion would be for you to build SSM Agent for your use case by modifying this line.

The instructions on how to build the SSM Agent yourself can be found in the README.md file.

Thanks, Faho Shubladze

Thanks for the feedback Faho,

I would try to make a custom build to my scenario. Is great to know that the agent has such thorough support.

Regards, Edgar