Working with AWS EC2MetadataCredentials Provider

[benswinburne]

When using the library on an EC2 instance (which has sufficient privileges through IAM) the library never seems to connect. As soon as the application starts I seem to end in an endless loop of the following.

{"level":50,"time":1566245540498,"pid":3823,"hostname":"ip-x","msg":"Error: Connection was closed abnormally","v":1}
{"level":40,"time":1566245540498,"pid":3823,"hostname":"ip-x","msg":"Connection to MQTT broker closed","v":1}
{"level":50,"time":1566245545502,"pid":3823,"hostname":"ip-x","msg":"Error: Connection was closed abnormally","v":1}
{"level":40,"time":1566245545502,"pid":3823,"hostname":"ip-x","msg":"Connection to MQTT broker closed","v":1}

I've used the same code on my local machine which resolves the credentials from the SharedIni file provider (Rather than the metadata service) and it works as expected.

I presume it's perhaps to do with the fact that the Metadata Credentials provider as async?

I've tried resolving the credentials myself rather than just passing in AWS.config.credentials which does work, but not without its own issue. For example

AWS.config.credentialProvider.resolve(
  (_, credentials): void => {
    if (credentials) {
      console.log(
        'Resolved AWS Credentials from',
        credentials.constructor.name
      );

      AWS.config.credentials = credentials;

      const client = new AWSMqttClient({
        expires: 86400,
        rejectUnauthorized: process.env.NODE_ENV === 'production',
        region: process.env.AWS_REGION,
        credentials: AWS.config.credentials,
        endpoint: endpoint,
        reconnectPeriod: Number(process.env.RECONNECT_WAIT) || 5000,
      });

      const error = (err) => console.error(err.toString());
      const close = () => console.warn('Connection to MQTT broker closed');
      const reconnect = () => console.info('Reconnecting to MQTT broker');
      const offline = () => console.warn('Connection to MQTT broker offline');
      const connect = () => console.log('Established connection with MQTT broker (AWS)');

      client.on('connect', connect);
      client.on('error', error);
      client.on('close', close);
      client.on('reconnect', reconnect);
      client.on('offline', offline);
    }
  }
);

This works both locally (SharedIni) and on EC2 (EC2Metadata) but fails to reconnect once the session expires. After a while, the library seems to send in an endless loop of the following.

Presumably I'm doing something not quite right as opposed to it being a bug/unexpected behaviour?

[kmamykin]

When using the library on an EC2 instance (which has sufficient privileges through IAM) the library never seems to connect. I've used the same code on my local machine which resolves the credentials from the SharedIni file provider (Rather than the metadata service) and it works as expected. I presume it's perhaps to do with the fact that the Metadata Credentials provider as async?

My hunch is that the EC2 instance does not have sufficient privileges or the role of the identity EC2 assumes does not have privileges, and I would further look into that. The only assumption AWSMqttClient class makes on the credentials object passed in, is that it has a function get (https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/Credentials.html#get-property) which is async, so that should work with EC2MetadataCredentials. See this line: https://github.com/kmamykin/aws-mqtt/blob/b373503d8371e2d6ab7d25d99af31cef585bdabc/src/urlSigner.js#L35

This works both locally (SharedIni) and on EC2 (EC2Metadata) but fails to reconnect once the session expires. After a while, the library seems to send in an endless loop of the following.

This is interesting because I haven't really tested the library as a long running Node process. My understanding is that the credential object should refresh itself once the session expires (https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/Credentials.html), but may be it is not happening.

It's notoriously hard to debug those access issues, I am afraid I don't have a quick fix there. More experimentation and debugging....