AWS endpoints other than amazonaws.com

[danopia]

Different types:

• [x] AWS Govcloud, AWS China (so-called AWS partitions)
• [x] dualstack endpoint for EC2, e.g. https://api.ec2.eu-west-1.aws/status
• [x] Localstack/minio (single origin endpoints, possibly unencrypted localhost)
• [ ] FIPS endpoints (some sort of different encryption)

[justinmchase]

I'm trying to point it to my local docker instance also, I'm running minio which exposes an S3 api.

[danopia]

Ok, so in a usecase where you'd like to give a specific URL when constructing the S3 client class, and path-based object access should then always be used (as opposed to the subdomain-style access that real S3 prefers)?

This is a bit different of a usecase than the original ones I listed, but also a bit simpler, so it should be pretty straightforward to add an escape hatch to enable this.

I understand that minio servers still have a region for signing purposes even though you won't be putting region into the URL. So the given base URL would effectively act like a global-endpoint override, except without forcing us-east-1.

[Scionax]

I came here to write exactly this. I need to be able to connect to a different provider of AWS as well.

It's fine if there's a system automatically in place to point things to amazonaws.com, but I definitely need an override or I'm locked out.

[danopia]

I need to be able to connect to a different provider of AWS as well.

Hi @Scionax are you also looking at a localstack setup (special localhost URLs for every service) or an alternative cloud (proper HTTPS URLs, just different ones from commercial AWS)?

I can put in an escape hatch for the first case, but I could also do better detection of the second case if that lines up with your use case.

[justinmchase]

The nodejs api that amazon publishes works with minio in localhost by just using the options endpoint and forcePathStyle. You have to specify a region also but it doesn't matter which one and minio gives you the creds when its launched and is compatible in most other ways with the sdk as is.

[danopia]

I've opened a PR which adds a fixedEndpoint option for the localstack/minio usecase, please feel free to give it a try now before I am ready to merge it and ship v0.5.0.

import { ApiFactory } from "https://raw.githubusercontent.com/cloudydeno/deno-aws_api/fixed-endpoints/lib/client/mod.ts";
import { S3 } from "https://aws-api.deno.dev/v0.1/services/s3.ts"; // or whatever

const s3 = new S3(new ApiFactory({
  fixedEndpoint: "http://localhost:9000",
}));

console.log("All buckets:", await s3.listBuckets()); // or whatever

I've done minimal testing with minio locally. (Might be cool for CI..)

[justinmchase]

Awesome, the only other thing is the forcePathStyle will essentially put the bucket name into the path rather than the subdomain, does this automatically go into path style mode when using a fixed endpoint? If so then that makes sense.

I think if you can just do even a single request to a bucket based endpoint then you're good to go. I don't have the environment up at the moment to test it myself but if this works then its good

import { ApiFactory } from "https://raw.githubusercontent.com/cloudydeno/deno-aws_api/fixed-endpoints/lib/client/mod.ts";
import { S3 } from "https://aws-api.deno.dev/v0.1/services/s3.ts"; // or whatever

const s3 = new S3(new ApiFactory({
  fixedEndpoint: "http://localhost:9000",
}));

await s3.createBucket({ Bucket: 'example' })
await s3.putObject({ Bucket: 'example', Body: 'Hello World!' })

Actually, looking at the code you're always using path style so it should be good!

[danopia]

Actually, looking at the code you're always using path style so it should be good!

You're looking at my generated code which doesn't get many API quirks baked in :) because the API definitions don't include them. My logic to upgrade S3 requests into virtual host style is here: https://github.com/cloudydeno/deno-aws_api/blob/6b267712cf00542603701a1fab333ede83707c6f/lib/client/client.ts#L67-L77

I already updated that heuristic to not enable when fixedEndpoint is passed, and tested basic object ops with Minio. Unfortunately the s3 example in this repo doesn't work with Minio as-is. It seems like there's some incompatibility around bucket configuration APIs, which makes sense since targeting minio wasn't a goal of this library. If issues come up with usage I'll accept them as separate tickets.

The other two cases that I mention in this issue's text will be handled via a different mechanism.

---

Apparently minio runs a public test instance so you can try this out without any local setup other than Deno.

const minio = new ApiFactory({
  fixedEndpoint: "https://play.min.io",
  credentials: {
    region: 'unused',
    awsAccessKeyId: 'Q3AM3UQ867SPQQA43P2F',
    awsSecretKey: 'zuf+tfteSlswRu7BJ86wekitnifILbZam1KYY3TG',
  },
}).makeNew(S3);