Is your feature request related to a problem? Please describe.
The Load functionality included in the web app needs to support file stream loading. This is to facilitate large datasets and cloud infrastructure.
Describe the solution you'd like
The Web App will optionally allow the user to select their stream producer location through the admin interface and upload JSON/JSONL and CSV datasets to it. This will be done through websockets since that is the most mature stream writable interface available through a browser as of 2/1/2021.
There is a new "poc-api-server" that inherits functionality from the "api-rest-server" but extends it's architecture to support loading to SQS queues through Websockets from webclient data streams. There are new endpoints in the OAS spec.
Describe alternatives you've considered
There was an exploratory spike done on evaluating the newer Stream Writer/Fetch. It is too immature at this time, Stream Readers are widely supported but Writers are barely supported.
arawinters
commented
3 years ago
Is your feature request related to a problem? Please describe. The Load functionality included in the web app needs to support file stream loading. This is to facilitate large datasets and cloud infrastructure.
Describe the solution you'd like The Web App will optionally allow the user to select their stream producer location through the admin interface and upload JSON/JSONL and CSV datasets to it. This will be done through websockets since that is the most mature stream writable interface available through a browser as of 2/1/2021.
There is a new "poc-api-server" that inherits functionality from the "api-rest-server" but extends it's architecture to support loading to SQS queues through Websockets from webclient data streams. There are new endpoints in the OAS spec.
see:
Describe alternatives you've considered There was an exploratory spike done on evaluating the newer Stream Writer/Fetch. It is too immature at this time, Stream Readers are widely supported but Writers are barely supported.