`cloud.Bucket` API for downloading objects as files

[Chriscbr]

Use Case

Expose an API that downloads objects to disk as files (instead of as return value(s)) so that they do not have to be kept in memory.

Proposed Solution

No response

Implementation Notes

No response

Component

SDK

Community Notes

• Please vote by adding a 👍 reaction to the issue to help us prioritize.
• If you are interested to work on this issue, please leave a comment.
• If this issue is labeled needs-discussion, it means the spec has not been finalized yet. Please reach out on the #dev channel in the Wing Slack.

[Chriscbr]

Here is one idea for a possible API:

bring cloud;
bring fs;

let bucket = new cloud.Bucket();

inflight () => {
  let tmp = fs.mkdtemp();
  bucket.download("object.txt", fs.join(tmp, "object.txt"));
  // ... do other things with the file
}

A possible issue could be that the Wing simulator doesn't provide file system isolation for inflight functions, so if the user tries saving the code to a non-temporary directory, they'll run into issues (multiple tests will try writing to the same place etc.). We can either try to provide file system isolation for simulator code, or we could design the API to prevent this kind of mistake:

inflight () => {
  let localFile = bucket.download("object.txt"); // it gives you a path to a file in a temporary directory
}

[eladb]

I think it would also be useful to be able to download a "directory" (in buckets it's represented as a prefix).

I would also make sure we document the requirement to use multi-part/recovering downloads so that huge files can be downloaded using this API.

[ekeren]

I would add a requirement to make sure you can download the file into a writableStream in JS it is:

s3.getObject(params).createReadStream().pipe(file)

This supports things like:

const { createGzip } = require('zlib');
const gzip = createGzip();
const params = { Bucket: 'mybucket', Key: 'mykey' };
const file = fs.createWriteStream('localfile.gz');

s3.getObject(params).createReadStream().pipe(gzip).pipe(file);

And Also:

s3.getObject(params).createReadStream().pipe(process.stdout);

[github-actions[bot]]

Hi,

This issue hasn't seen activity in 90 days. Therefore, we are marking this issue as stale for now. It will be closed after 7 days. Feel free to re-open this issue when there's an update or relevant information to be added. Thanks!