S3.GetObject no longer returns the result as a string

[igilham]

Describe the bug I'm using the GetObjectCommand with an S3Client to pull a file down from S3. In v2 of the SDK I can write response.Body.toString('utf-8') to turn the response into a string. In v3 of the SDK response.Body is a complex object that does not seem to expose the result of reading from the socket.

It's not clear if the SDK's current behaviour is intentional, but the change in behaviour since v2 is significant and undocumented.

SDK version number 3.1.0

Is the issue in the browser/Node.js/ReactNative? Node.js

Details of the browser/Node.js/ReactNative version v12.18.0

To Reproduce (observed behavior)

import { GetObjectCommand, S3Client } from '@aws-sdk/client-s3';

export async function getFile() {
  const client = new S3Client({ region: 'eu-west-1' });
  const cmd = new GetObjectCommand({
    Bucket: 'my-bucket',
    Key: '/readme.txt',
  });
  const data = await client.send(cmd);

  console.log(data.Body.toString('utf-8'));
}

Expected behavior It should print the text of the file.

Additional context

data.Body is a complex object with circular references. Object.keys(data.Body) returns the following:

[
  "_readableState",
  "readable",
  "_events",
  "_eventsCount",
  "_maxListeners",
  "socket",
  "connection",
  "httpVersionMajor",
  "httpVersionMinor",
  "httpVersion",
  "complete",
  "headers",
  "rawHeaders",
  "trailers",
  "rawTrailers",
  "aborted",
  "upgrade",
  "url",
  "method",
  "statusCode",
  "statusMessage",
  "client",
  "_consuming",
  "_dumped",
  "req"
]

[kesavab]

Found an easy solution using transformToString if wanting to parse a JSON file in S3.

import { S3, GetObjectCommand } from '@aws-sdk/client-s3'

const s3 = new S3({});

const getObjectParams = {
  Bucket: 'my-bucket',
  Key: 'my-object',
};
const getObjectCommand = new GetObjectCommand(getObjectParams);
const s3Object = await s3.send(getObjectCommand);

const dataStr = await s3Object.Body?.transformToString();

let data;
if (dataStr) {
  data = JSON.parse(dataStr);
}

When I use this, I get error during parsing. Error is Uncaught SyntaxError SyntaxError: Unexpected token  in JSON at position 0 at eval (repl:1:6)

Call to transformToString returns the json file contents

[OlivierCuyp]

@kesavab This means your dataStr is not a valid JSON string. Did you try to log it with something like:

// ...
if (dataStr) {
 try {
   data = JSON.parse(dataStr);
 } catch (err) {
   console.log(err, dataStr);
 } 
}

[ghost]

What's the next step after getting the object string if I want to download the file? Should I convert it into a blob file and return it to the frontend?

[kmordan24]

Typescript errors if you try to pipe Body even though it is a stream: Property 'pipe' does not exist on type 'SdkStream<Readable | ReadableStream<any> | Blob | undefined>'.   Property 'pipe' does not exist on type 'ReadableStream<any> & SdkStreamMixin'.

const { Body } = await s3Client.send(new GetObjectCommand({ Bucket: bucket, Key: key }));

if (!Body) {
  throw new Error('....'');
}

Body.pipe(fs.createWriteStream(filePath))
    .on('error', (err: any) => reject(err))
    .on('close', () => resolve());

...

The only way around this is type casting - unless anyone has any other ideas.

Will there be a fix for this in a future release version?

[RanVaknin]

Hi all,

It is extremely hard for us to keep track of issues that lay at the end of the issue queue on Github.

Since the OP's concern was addressed , and additional concerns were also addressed I feel like we can close this.

If you still need help whether it be to report a bug, or to ask a question, please re-engage with us on a new thread / discussion. This makes it so we have more visibility to your concerns and can answer you in a timely manner.

Sorry for the inconvenience, Ran~

[jsancho]

Okay so after spending few hours I got it right. This way we can pipe our s3 response body into sharp and later use the .toBuffer() to push it to bucket.

  const getObj = new GetObjectCommand({
    Bucket,
    Key: objectKey,
  });

  const s3ImgRes = await s3Client.send(getObj);

  const sharpImg = sharp().resize({ width: 500 }).toFormat("webp");

  // pipe the body to sharp img
  s3ImgRes.Body.pipe(sharpImg);

  const putObj = new PutObjectCommand({
    Bucket,
    Key: `converted/${objectKey.replace(/[a-zA-Z]+$/, "webp")}`,
    Body: await sharpImg.toBuffer(),
  });

  await s3Client.send(putObj);

But AWS team please please you need update your docs, I know there is a lot to update but as developer its just so much struggle to use AWS services because of insufficient docs.

can't thank you enough for this workaround :)

[j0k3r]

I found an other solution depending on how you want to get your content from s3 when using Node.js >= v16.7.0 using stream/consumers:

import consumers from 'node:stream/consumers'
import { S3 } from '@aws-sdk/client-s3'

const s3 = new S3()
const { Body } = await s3.getObject({ Bucket: "your-bucket", Key: "your-object-key" })
// as a buffer
const buffer = await consumers.buffer(Body)
// as a text
const text = await consumers.text(Body)
// as a json
const json = await consumers.json(Body)

[github-actions[bot]]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs and link to relevant comments in this thread.

[yenfryherrerafeliz]

Hi people, I just want to let you all know that we have introduced a new feature that allows us to convert the body of a bucket's object to a string by doing the following:

import {S3Client, GetObjectCommand} from "@aws-sdk/client-s3";

const client = new S3Client({
    region: 'us-east-2'
});
const response = await client.send(new GetObjectCommand({
    Bucket: process.env.TEST_BUCKET,
    Key: process.env.TEST_KEY
}));
console.log(await response.Body.transformToString('utf-8'));

This feature is available after version 3.357.0

Thanks!