Discovery API fails for DescribeInstances operation

[rayba]

With a valid AWS AK/SK the DescribeInstances operation consistently fails. The other BOTO3 calls work so it's something with this specific call.

server-list:
state: danger
msg: An error occurred (AuthFailure) when calling the DescribeInstances operation: AWS was not able to validate the provided access credentials At 2020-03-31 03:31:08.659259

When this happens, no servers show up in the UI. I believe a better behavior would be to show the last good result (from cache or Mongodb).

[Signorini]

Humm.

Before I start to deep investigate, can you:

• Do the clock is right on your host? This message error normally happens when it has a wrong clock configuration, docker uses the host timezone.

If yes can you try to use ntpdate on the host and then spin up again the discovery-maestro and discovery-maestro-workers https://stackoverflow.com/questions/24551592/how-to-make-sure-dockers-time-syncs-with-that-of-the-host

• Do the credentials have permission to Read Ec2 instances? I think is yes, but, you know, I need to ask

• I've finished a new release bumping all third libraries, including boto3, I starting to test the new version, and if everything is ok this version will be released in this week.

------------- //

When this happens, no servers show up in the UI. Is it worked before? In my mind, when a ran a sync command and works, then if for any reason the next sync command fails, the discovery service doesn't delete the old data. I'll check to be sure about it.

thanks

[rayba]

date is correct, it's an EC2 instance host. Also followed instructions from your link. That updated by a few ms.

The role does have describe_instances - however, to troubleshoot I assigned Administrator temporarily. Same result.

The interesting thing is that other boto3 calls are working. Will look for when the update is ready to try it. Contact me direct if you need more context on how this is deployed or the client use case.

[rayba]

You can close this issue, it was caused by a weird circumstance of running a local version at the same time as a cloud hosted one. Some services ran locally others on the cloud due to the way docker-compose was setup.