Foundry fails to read static credentials for a S3-compatible path-style api

[LavCorps]

What happened?

Foundry fails to read static credentials provided at ~/.aws/credentials.

What ways of accessing Foundry can you encounter this issue in?

• [ ] Native App (Electron)
• [X] Chrome
• [ ] Firefox
• [ ] Safari
• [ ] Other

Reproduction Steps

This error is known to be fatal given the combination of the following factors:

• buckets is not defined in the S3 configuration json
• forcePathStyle is not defined in the S3 configuration json (with s3ForcePathStyle set defined instead)
• S3-compatible provider is a SeaweedFS server hosted under the same hostname on a different port
• static credentials are provided at ~/.aws/credentials

This error is logged but NOT fatal given the combination of the following factors:

• buckets is defined in the S3 configuration json
• forcePathStyle is defined in the S3 configuration json
• S3-compatible provider is a SeaweedFS server hosted under the same hostname on a different port
• static credentials are provided at ~/.aws/credentials

What core version are you reporting this for?

Version 11 Build 302

Relevant log output

• FATAL log output in Foundry:

  FoundryVTT | 2023-06-19 18:24:38 | [info] Configured AWS credentials using /srv/foundry/Config/s3.json
  FoundryVTT | 2023-06-19 18:24:38 | [info] Searching for allowed buckets for S3 storage.
  FoundryVTT | 2023-06-19 18:24:38 | [error] A fatal error occurred while trying to start the Foundry Virtual Tabletop server: The access key ID you provided does not exist in our records.
  InvalidAccessKeyId: A fatal error occurred while trying to start the Foundry Virtual Tabletop server: The access key ID you provided does not exist in our records.
  at throwDefaultError (/srv/foundry/Bin/resources/app/node_modules/@aws-sdk/smithy-client/dist-cjs/default-error-handler.js:8:22)
  at de_ListBucketsCommandError (/srv/foundry/Bin/resources/app/node_modules/@aws-sdk/client-s3/dist-cjs/protocols/Aws_restXml.js:4940:43)
  at processTicksAndRejections (node:internal/process/task_queues:96:5)
  at async /srv/foundry/Bin/resources/app/node_modules/@aws-sdk/middleware-serde/dist-cjs/deserializerMiddleware.js:7:24
  at async /srv/foundry/Bin/resources/app/node_modules/@aws-sdk/middleware-signing/dist-cjs/middleware.js:14:20
  at async /srv/foundry/Bin/resources/app/node_modules/@aws-sdk/middleware-retry/dist-cjs/retryMiddleware.js:27:46
  at async /srv/foundry/Bin/resources/app/node_modules/@aws-sdk/middleware-logger/dist-cjs/loggerMiddleware.js:7:26
  at async S3FileStorage.#t (file:///srv/foundry/Bin/resources/app/dist/files/s3.mjs:1:2574)
  at async S3FileStorage.identifyEndpoint (file:///srv/foundry/Bin/resources/app/dist/files/s3.mjs:1:3727)
  at async _initializeCriticalFunctions (file:///srv/foundry/Bin/resources/app/dist/init.mjs:1:2815)

• FATAL log output in SeaweedFS:

  I0619 18:24:38.104402 auth_credentials.go:197 could not find accessKey "foundry"

• NON-FATAL log output in FoundryVTT:

  FoundryVTT | 2023-06-19 19:08:13 | [info] Configured AWS credentials using /srv/foundry/Config/s3.json
  FoundryVTT | 2023-06-19 19:08:13 | [error] Failed to determine S3 endpoint: UnknownError

• NON-FATAL log output in SeaweedFS:

  I0619 19:08:13.180143 auth_credentials.go:197 could not find accessKey "foundry"

Bug Checklist

• [X] The issue occurs while all Modules are disabled

[LavCorps]

extra facts worth asserting:

• despite the could not find accessKey error, this access key IS defined and was able to be used by previous versions of foundry as well as s3 file browsing software
• this setup worked fine in foundry v10
• accessKey is defined as foundry

[luhahn]

is there a workaround? I'm getting the same error and it prevents foundry from starting

[Fyorl]

is there a workaround? I'm getting the same error and it prevents foundry from starting

The workaround should be to provide your credentials directly in your aws.json, there's an example in the KB: https://foundryvtt.com/article/aws-s3/

[luhahn]

yes, i did not notice, that aws.json format has changed. I used the old format. Everything is working fine now.

Sorry for the noise!

[FeistyViking]

also chiming in that we experienced the same problem with our instances and was due to the formatting change of the json file

[LavCorps]

the same problem as luhahn or the same problem as mine?

[FeistyViking]

LavCorps, was your problem not resolved when you updated to the new format for your json file?

8898

[LavCorps]

no, my problem is entirely seperate. it is intended to be able to read S3 credentials from ~/.aws/credentials (not a json file at all), but Foundry encounters an error using those credentials.

[FeistyViking]

ah I see. I wonder though if you are struggling with the same root problem though of needing to specify the region now that it is no longer inferred using just the credentials

[LavCorps]

no, region is unneeded for seaweedfs (and region is additionally specified in aws.json already as "none")

[FeistyViking]

ah I see. I apologize for jumping in on this thread then. My issue was specific to the formatting change of the json file, specifically needing to point to region in the config.

[FeistyViking]

I will note, something else I noticed though is that the aws.json file on my instances did get it's ownership changed. Not sure what caused that but I am assuming during the 10 to 11 update. Maybe worth checking your permissions of ~/.aws/credentials ?

[LavCorps]

~/.aws/credentials can be read from (note that foundry attempts to authenticate using the given access key), but appears unable to log in regardless. title may be misleading with this in mind, but as i'm unaware of the inner workings of how foundry's aws access works, it seemed the most appropriate.

[aaclayton]

We use the default Node.js credential provider chain described here: https://docs.aws.amazon.com/sdk-for-javascript/v3/developer-guide/setting-credentials-node.html

[Fyorl]

Not able to reproduce this. I am able to successfully authenticate with S3 with the following:

Config/s3.json

{
  "region": "eu-west-2",
  "forcePathStyle": true
}

~/.aws/credentials

[default]
aws_access_key_id = AKIAIOSFODNN7EXAMPLE
aws_secret_access_key = wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY

I don't have a seaweedFS instance to test with, so it's unclear where it would be failing. Since it worked in v10 I can only guess that either seaweedFS is not compatible with requests from v3 of the AWS SDK, or there are some additional configuration changes needed, but I'm not sure what they might be.

I have tested the S3 integration with Cloudflare R2 which is also an S3-compatible service with a custom endpoint, and that also works, so it's not a general problem related to non-official S3 services.

[LavCorps]

I would like to add additional context that SeaweedFS does work with Foundry V11 using credentials in s3.json, which would imply to me that it is compatible with v3 AWS SDK requests (unless there is some implicit request mutation I'm unaware of depending on credential location). Will be testing and reporting results here.

[LavCorps]

Ah. Testing has immediately revealed this appears to be a consequences of an AWS SDK update, and how it handles ~/.aws/credentials. V10 was able to parse aws_access_key_id and aws_secret_access_key if the value was wrapped in double quotes. Removing the double quotes in ~/.aws/credentials reveals that this issue specifically occurs under those circumstances, and that static credentials work fine otherwise.

Line 197 of auth_credentials.go in SeaweedFS does not include double quotes around the access key that could not be found:

glog.V(1).Infof("could not find accessKey %s", accessKey)

Which gives credence to the idea that where the AWS SDK ignored double quotes in ~/.aws/credentials before, it will now include them as part of the credentials. Further support comes from testing in SeaweedFS 3.53, attempting to authenticate with a non-existence access key that does not include double quotes:

I0707 16:32:51.142310 auth_credentials.go:197 could not find accessKey not-an-access-key

Additionally, possibly worth nothing that this testing took place under Foundry V11 Build 305, rather than the Build 302 mentioned in the original issue.

I suppose that reveal suggests this issue is:

1. unnecessary, as it has to do with AWS SDK.
2. open only because of a fault in my own testing. My sincerest apologies in this regard.

[Fyorl]

Thanks for the update, we were also not aware of this particular change to double-quote behaviour as part of the v3 SDK so it will help us advise others who may experience similar problems.