File asset usability improvements

[peterdemartini]

IMPORTANT:

• [ ] When file_per_slice: false is not respected with ldjson, generates file with foobar/0.0.ldson
• [ ] Extension shouldn't care if the dot prefix exists or not, generates file with foobar/0.0ldjson
• [ ] S3 sender should chunk files

USABILITY:

• [ ] No more global configuration
  • sender and reader config should be separated
  • formats should have their own config values
  • config values that aren't used in the APIs should not be specified
• [ ] Add sensible defaults at the API level
• [x] Format should indicate the generate the extension, if extension is not provided
  • Like format ldson should be ldjson
  • if compression is set should be .ldjson.gz or ldjson.lz4

[peterdemartini]

Also we should add more jsdoc especially for configuration values that have a specific reason to exist that may not be obvious, like line_delimeter should have "Useful for overriding the line delimiter for making windows compatible files with carriage returns".

[peterdemartini]

The S3Reader should be named S3Fetcher since it doesn't do both fetching and slicing.

Also the file names in ./packages/file-asset-apis/src/s3-api should not have the -api suffix since it is inconsistently used and unnecessary.

[jsnoble]

All readers export a function to create slicers, currently the hdfs does not but it will shortly

[peterdemartini]

But that seems like a helper function to create something else that does the slicing. Seems kind of inconsistent.

[peterdemartini]

Also regarding the multi-part upload we need make sure the implementation doesn't single create a gigantic string or buffer and then split it up since that would still make us hit the max string or buffer limit of 1GB. So it should be split up before serializing.