I'd like to have a stream plugin that allows streaming files from on Blob to another, for examples S3 -> MinIO, S3 -> S3, Azure -> S3 etc. For starters no need to support sinking into Parquet or other formats, basically just move bytes from one storage to another. YAML spec can be something like:
I'd like to have a stream plugin that allows streaming files from on Blob to another, for examples S3 -> MinIO, S3 -> S3, Azure -> S3 etc. For starters no need to support sinking into Parquet or other formats, basically just move bytes from one storage to another. YAML spec can be something like: