Various issues related to S3FileAdapter (including error in documentation)

We are trying to implement the S3FileAdapter (https://www.keystonejs.com/keystonejs/file-adapters/#s3fileadapter) and have encountered various issues. We are unable to get it to work correctly.

There seems to be an error in the documentation. When I click on 'create' in the AdminUI, the example given

const { S3Adapter } = require('@keystonejs/file-adapters');

const CF_DISTRIBUTION_ID = 'cloudfront-distribution-id';
const S3_PATH = 'uploads';

const fileAdapter = new S3Adapter({
  accessKeyId: 'ACCESS_KEY_ID',
  secretAccessKey: 'SECRET_ACCESS_KEY',
  region: 'us-west-2',
  bucket: 'bucket-name',
  folder: S3_PATH,
  publicUrl: ({ id, filename, _meta }) =>
    `https://${CF_DISTRIBUTION_ID}.cloudfront.net/${S3_PATH}/${filename}`,
  s3Options: {
    apiVersion: '2006-03-01',
  },
  uploadParams: ({ filename, id, mimetype, encoding }) => ({
    Metadata: {
      keystone_id: id,
    },
  }),
});

returns

{"level":50,"time":1588166644489,"pid":73682,"hostname":"AGOMA1002.local","name":"graphql","message":"Expected params.Metadata['keystone_id'] to be a string","locations":[{"line":2,"column":3}],"path":["createImage"],"uid":"ck9ldexon0003uqwk3d9mfa3h","name":"GraphQLError","errors":[{"code":"InvalidParameterType","time":{},"name":"Error","message":"Expected params.Metadata['keystone_id'] to be a string","stack":"ParamValidator.fail

in the logs (and an alert appears in the AdminUI saying 'Nested errors'). I traced out exactly what id was in uploadParams and it turns out to be an object, not a string. So the example given in the docs doesn't work. I had to change it to

uploadParams: ({ filename, id, mimetype, encoding }) => ({
    Metadata: {
      keystone_id: id.toString(),
    },
  }),

We don't want to include a public url, that is, my DevOp team wants to keep the images private. So we've commented out the public url (it's not clear from the documentation whether it's required or optional -- it would be helpful to have that info in the docs)

const { S3_KEY, S3_SECRET, S3_REGION, S3_BUCKET, S3_PATH } = process.env;

const imageFileAdapter = new S3Adapter({
    accessKeyId: S3_KEY, //S3_KEY,
    secretAccessKey: S3_SECRET,
    region: S3_REGION,
    bucket: S3_BUCKET,
    folder: S3_PATH,
    // publicUrl: ({ id, filename, _meta }) => `https://${process.env.CF_DISTRIBUTION_ID}.cloudfront.net/${process.env.S3_PATH}/${filename}`,
    s3Options: {
        apiVersion: '2006-03-01'
    },
    uploadParams: ({ filename, id, mimetype, encoding }) => ({
        Metadata: {
            keystone_id: id.toString(),
        },
    })
});

The image seems to upload correctly, that is, the AdminUI pops up an alert saying 'Saved successfully' after I click on 'Save Changes', but the AdminUI shows a broken image

if I get the URL of the image and try to load it I get

<Error>
<Code>NoSuchKey</Code>
<Message>The specified key does not exist.</Message>
<Key>public/images/5ea99101dcbf033a2394498a-node2.png</Key>
<RequestId>71478C01F92F489E</RequestId>
<HostId>
sgzWPRRoeZ6cyaMwPZrAUj35zwn2NMYOdpx8eVGLWTAPLcy3a9wtNz3sUTX8963UxesYrZtEO0w=
</HostId>
</Error>

This occurs even when we make the S3 images public.

Finally and perhaps most importantly, we don't see any records of calls to AWS in the Keystone logs, nor in the browser Network panel. AWS also has no records of our calling it. If we change the SECRET_KEY then it errors with an error that the key is wrong, so it's doing something, but we are really unclear as to why there seem to be not network calls to AWS.

So we are kind of dead in the water at the moment. As to the first error -- that of id being an object, not a string -- perhaps my 'fix' was incorrect and that's screwing things up. Any advice much welcome!

From a Slack conversation that helped solve much of the above, which I move here at request of @jesstelford

I ran into that issue last night: you'll need to call String(id) in the upload params resolver on the S3Adapter. Also make sure you're not including any / before whatever folder you specify - for some reason that was telling the adapter to create an empty, unnamed directory at the root level before creating another directory under it with the actual folder name

Also when specifying the CF Distribution ID (as laid out in the Keystone examples), you'll actually want the CF Domain, not the ID

Since we aren’t using the public url we aren’t using the CF Domain — is the public url optional? Even my DevOp, who’s an AWS guru, wasn’t sure whether it was or not in this context — that is, the AdminUI has access to the credentials, right, and could use those? If the images need to be public, we can do that too, it’s just he is trying to avoid anything public he says (of course that begs the question about how the front end will see them, but that’s another issue) (edited)

That's a good question, my thought was that the file adapter was meant to store references to files but not actually store and file blob data in the db, tbh I didn't try to get it working without a public url so I'm not sure. Unless the server is pre-fetching the images from AWS and then sending the blob data in the payload, my guess is that there needs to be some public reference in order for the front end (including the admin UI!) to see the images. It looked to me like only metadata was stored in the db. That said you also don't need to set up Cloud Front, you could serve directly from the S3 bucket, if you wanted, but that's definitely the less desirable option from a security and distribution standpoint.

keystonejs / keystone

Various issues related to S3FileAdapter (including error in documentation) #2858