(TokenExchangeService): The authentication token has expired and does not update components work.

[DDemoNZ]

Description After deployment Greengrass receive an auth token using tokenExchangeRole with an expiration time of 3600 sec (1 hour). During another component work that works with a StreamManager, the token becomes expired after 1 hour and it does not update without component redeploy.

Expected As it was expected, the exchange token service will update the token as soon as it expires.

Actual Token remains expired without a new trigger - component redeployment (the component also has a HARD depend on TokenExchangeService). As a result, the component receives the error "Failed to upload data to S3. The provided token has expired."

Question Are there any ways to activate auto renewal for auth token using TokenExchangeService? Or any other possible solutions for this case?

[MikeDombo]

Hello,

The token is automatically refreshed when it expires and a a component requests new credentials. TES will request new credentials 5 minutes before expiration of the old token.

Please verify that your device's clock is correct. If the clock is in accurate, then Greengrass will not know that it needs to get updated credentials.

If you believe that your clock is accurate, please provide debug level logs from Greengrass showing the time that it gets the credentials and from the time that the S3 operation happens without credentials.

[roman-hizatulin-ge]

Hello. @MikeDombo Clocks are synchronized for now.

Also we log "AWS_CONTAINER_AUTHORIZATION_TOKEN" env in our application and we have the same token every time.

greengrass.log - removed StreamManager.log - removed

[MikeDombo]

@roman-hizatulin-ge is this related to the original question, I see that you are not the person who creates this issue?

The container auth token being the same is normal. This is not the AWS credential used to talk to AWS. It is only a token to allow you to request AWS credentials from Greengrass.

The Greengrass log file has no timestamps or log levels which will make it difficult to line up with any issues.

[roman-hizatulin-ge]

@MikeDombo "is this related to the original question" yes

[MikeDombo]

You said clocks are synchronized for now.

Was the clock correct previously?

Please provide all raw logs from /greengrass/v2/logs. Open a support case through the AWS Console if you want to share privately.

[roman-hizatulin-ge]

There was an hour before inside a docker container.

[MikeDombo]

There was an hour before inside a docker container.

I'm not sure what you mean by this. Are you saying that the system clock was wrong by 1 hour?

Or are you talking about the logs? Please provide all the logs in their raw format. The log files provided do not look like raw logs from greengrass, some are missing timestamps and context.

[roman-hizatulin-ge]

I'm not sure what you mean by this. Are you saying that the system clock was wrong by 1 hour?

Yes, you are right

Please provide all the logs in their raw format. The log files provided do not look like raw logs from greengrass, some are missing timestamps and context.

I'm collecting them

[MikeDombo]

I believe you have corrected the clock but just to make clear the requirements.

For the avoidance of doubt, the system clock can use any timezone as long as the time and timezone is set correctly. Logs are always written with UTC time. It is currently 14:28 UTC, so if the most recent logs have a different time than that, the clock is not set correctly.

[roman-hizatulin-ge]

Please look at these files (with timestamps)

greengrass.log - removed StreamManager.log - removed

[MikeDombo]

Thank you, I will look more in a little while.

I see at the end of the stream manager log that it has nothing to upload, please make sure that stream manager does have work to do so we can see if it succeeds or fails.

[roman-hizatulin-ge]

Please find "Failed to upload data to S3" in the SM.log

[MikeDombo]

Yes I see errors at 13:21 to 13:22. Uploads succeeded before that time. No uploads were attempted after that time. That's why I'm requesting that you give stream manager more work to do, to see if it will succeed given that the system clock is correct now.

[MikeDombo]

Did you change any of the default options of stream manager? It looks like it has loaded aws credentials from environment variables instead of loading them from greengrass, this should not happen. You said greengrass was in docker. Do you have aws credential environment variables passed into the container? You must remove those if you do.

[roman-hizatulin-ge]

Yes I see errors at 13:21 to 13:22. Uploads succeeded before that time. No uploads were attempted after that time. That's why I'm requesting that you give stream manager more work to do, to see if it will succeed given that the system clock is correct now.

Uploaded new files greengrass17.00.log - removed StramManager17.00.log - removed

[MikeDombo]

Based on the logs and your description, it looks like Stream Manager is getting credentials from environment variables instead of the Token Exchange Service. You could further validate this by looking in /proc/<pid of stream manager process>/environ for AWS_ACCESS_KEY_ID. These environment variables are not set by Greengrass, but may be inherited from your environment. You must remove those environment variables in order for Stream Manager to work properly by requesting live credentials from TES.

[roman-hizatulin-ge]

Based on the logs and your description, it looks like Stream Manager is getting credentials from environment variables instead of the Token Exchange Service. You could further validate this by looking in /proc/<pid of stream manager process>/environ for AWS_ACCESS_KEY_ID. These environment variables are not set by Greengrass, but may be inherited from your environment. You must remove those environment variables in order for Stream Manager to work properly by requesting live credentials from TES.

It seems you're right, we have the env var. Will try to unset

[MikeDombo]

Sounds like this issue should be resolved, closing since there have not been any updates.

Reopen as needed.

[roman-hizatulin-ge]

Sounds like this issue should be resolved, closing since there have not been any updates.

Reopen as needed.

Hello @MikeDombo Yes, the reason is the "AWS_ACCESS_KEY_ID" env var. We are working on a fix now. Thx for your big support. 🤝🏻

[roman-hizatulin-ge]

Hello. @MikeDombo could you please help with the next question: We need the "AWS_ACCESS_KEY_ID" variable during the green-grass installation to provision a thing and all components. After installation this variable should be unset (we've already tried that, but unsuccessfully). Do you know | have any workaround to achieve this?

[MikeDombo]

I believe you said that you are using Docker right?

1. I generally advise against using Docker as many people get confused about where their data is located and where exactly the processes are executing. You may not have this confusion, but I still will advise you to run Greengrass natively, not in a container.
2. You do not necessarily need to use automatic provisioning, which is the only thing that requires the AWS environment variables. You could use manual provisioning or fleet provisioning by claim.
3. How did you try to unset the variable? Environment variables are explicitly passed into docker containers, so if you run the docker container without passing in the environment variable then it cannot possibly be set in the container.
4. You could set the credentials in a credential file instead of environment variables. Then delete the file.

[MikeDombo]

Stream Manager v2.1.12 is just released now. This new version will always prefer the credentials from Greengrass instead of the default order which will use credentials from environment variables.

Other components, including components that you write yourself, will not have this credential order change so you should still work on a solution to remove the environment variable credentials.

[roman-hizatulin-ge]

Hello @MikeDombo. Sorry for late. Yes, we use a GG as a docker container. For now we have already updated a GG StreamManager to the latest version and chosen the latest option (4th - credential file) you proposed and it fixed our issue. Thanks a lot. 🤝🏻