SneaksAndData / arcane-framework

Akka.NET-based framework for data streaming services using the Arcane Kubernetes Operator
Apache License 2.0
5 stars 2 forks source link

Add BlobStorage to BlobStorage stream #62

Closed george-zubrienko closed 4 months ago

george-zubrienko commented 4 months ago

I'd like to have a stream plugin that allows streaming files from on Blob to another, for examples S3 -> MinIO, S3 -> S3, Azure -> S3 etc. For starters no need to support sinking into Parquet or other formats, basically just move bytes from one storage to another. YAML spec can be something like:

spec:
  blobSource:
    s3:
      bucket: mybucket
      prefix: myprefix/data
    azure:
       container: my-container
       prefix: myprefix/data
  blobTarget:
    s3:
      bucket: mybucket
      prefix: myprefix/data
    azure:
       container: my-container
       prefix: myprefix/data