ZJONSSON / parquetjs

fully asynchronous, pure JavaScript implementation of the Parquet file format
MIT License
34 stars 61 forks source link

Feature: Read from S3 using v3 of the aws js sdk - ParquetReader.openS3v3 #62

Closed entitycs closed 3 years ago

entitycs commented 3 years ago

Allow developers to use version 3 of the aws s3 client. It follows in the spirit of parquetjs-lite to enable usage of the lighter version of the aws s3 sdk. Pretty straightforward implementation.

There could be better interface/architectural approaches to the solution. Note, however, that the command inputs are readonly (also not modifiable by the middleware stacks) - hence the approach taken to pass the class modules themselves along with the input parameters as opposed to passing objects.

The following is appended to the README, under the existing example, and is sharing that version's params variable.

//v3 example const {S3Client, HeadObjectCommand, GetObjectCommand} = require('@aws-sdk/client-s3'); const v3client = new S3Client({region:"us-east-1"}); let commands = { headObjectCommand: HeadObjectCommand, getObjectCommand: GetObjectCommand }; let v3reader = await parquet.ParquetReader.openS3v3(v3client, commands, params);