Binary data implementation

[lukas-valenta]

Description

In this pull request is introduced new "primitive" type BinaryData.

export interface IBinaryData {
  peek(size?: number): Promise<Buffer | undefined>;
  getAllData(): Promise<Buffer>;
  chunkBy(chunkSize: number): AsyncIterable<Buffer>;
  toStream(): Readable;
}

If BinaryData instance gets to be used as body or value in FormData, it is streamed. If BinaryData should be returned as outcome, an UnexpectedError is returned. To solve it, call getAllData() to read stream into Buffer. Later streamed results will be added.

Limitations

1. Asynchronous operations on BinaryData Current limitation for use in Comlink Map is that peek, read and getAllData methods are asynchronous. So, they can't be directly chained with function on Buffer. Most likely toString().

To overcome this limitation, It must be first set into variable, so the interpreter resolves the Promise:

map Example {
  firstBytes = input.binary.peek(10)
  data = input.binary.read(10)
  remainingData = input.binary.getAllData()

  map result {
    firstBytes = firstBytes
    data = data
   remainingData = remainingData
  }
}

Opaque data type In profile, the type must be omitted for now. Later we may add it to Comlink Profile as new Primitive.

name = "example"
version = "1.0.0"

usecase Example {
  input {
    binary
  }
}

Comlink Map examples

Send file in body:

map SendFile {
  http POST "/resource" {
    request "application/octet-stream" {
      body = input.file
    }

    response 200 "text/plain" {
      map result body
    }
  }
}

Send file as FormData field:

map SendFile {
  http POST "/resource" {
    request "multipart/form-data" {
      body {
        field = input.field
        file1 = input.file1
        file2 = input.file2
      }
    }

    response 200 "text/plain" {
      map result body
    }
  }
}

Send file by chunks:

profile = "example@1.0"
provider = "localhost"

map ChunkedUpload {
  uploadResults = call foreach(chunk of input.file.chunkBy(10)) Upload(chunk = chunk)

  return map result uploaded
}

operation Upload {
  http POST "/upload/{args.uploadId}" {
    request "application/octet-stream" {
      body = args.chunk
    }

    response 200 "text/plain" {
      return true
    }

    response 500 "text/plain" {
      return false
    }
  }
}

Read first bytes of file and then send it in body:

map SendFile {
  firstTen = input.file.peek(10)

  contentType = firstTen.toString('ascii').toLowerCase().includes('pdf') ? 'application/pdf' : 'application/octet-stream'

  http POST "/resource" {
    request {
      headers {
        'content-type' = contentType
      }

      body = input.file
    }

    response 200 "text/plain" {
      map result body
    }
  }
}

OneSDK use example

From user perspective it is used as followed:

import { SuperfaceClient, BinaryData } from '@superfaceai/one-sdk';

const client = new SuperfaceClient();
const profile = await client.getProfile('example');

// for files
const result = await profile.getUseCase('Example').perform({
  binary: BinaryData.fromPath(pathToFile, { mimetype: 'text/plain', filename: 'example.txt' });
});

// or for Readable streams
const stream = getReadableStream();
const result = await profile.getUseCase('Example').perform({
  binary: BinaryData.fromStream(stream);
});

Motivation and Context

To allow the passing of binary data with capabilities like async loading, streaming and chunking.

Types of changes

• [ ] Bug fix (non-breaking change which fixes an issue)
• [x] New feature (non-breaking change which adds functionality)
• [ ] Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

• [ ] I have updated the documentation accordingly. If you are changing code related to user secrets you need to really make sure that security documentation is correct.
• [x] I have read the CONTRIBUTING document.
• [x] I haven't repeated the code. (DRY)
• [x] I have added tests to cover my changes.
• [x] All new and existing tests passed.