search

vkvam / fpipe

A framework for processing file-like objects with pipes
https://github.com/vkvam/fpipe
MIT License
9 stars 0 forks source link

S3 improvements #19

Open vkvam opened 4 years ago

vkvam commented 4 years ago

Consider passing arguments to S3 gen and S3Writer as boto3 compatible ones.

FileMeta could have a context concept, where a value is re-formatted and named according to the the context it is used.

Path = boto3 Key Version = boto3 Version

This could be in conflict with pathname_resolver in S3/Local generators. This should anyways be bound in some way to what we are expecting. Let's say S3 expects something that creates Path, and the default is getting the path from the source file.

vkvam commented 4 years ago

S3 generator has a pretty large and unreadable process() method

vkvam commented 4 years ago

When creating an s3 object there is a huge number of parameters.

  1. Difficult to set all as FileData
  2. Implicit reuse of FileData from previous Generators is dangerous (e.g. ACL). Consider a max depth to search for needed FileData and prefixing s3 FileData with S3. So S3Key instead of Path. Another option is to require a whitelisting of all parameters that can be used from a parent generator.