puppetlabs-toy-chest / wash

Wide Area SHell: a cloud-native shell for bringing remote infrastructure to your terminal.
https://puppetlabs.github.io/wash
Apache License 2.0
180 stars 29 forks source link

Clearer interface for Read primitives #619

Closed MikaelSmith closed 4 years ago

MikaelSmith commented 4 years ago

Reads from FUSE always operate on a length and offset within a file. This makes the simplest interface to adapt the io.ReaderAt interface. It also maps well to some modes of object access like AWS S3 and Google Cloud Storage APIs, or using dd to access parts of a file.

However other things we want to read don't have a pre-determined size, and we have to download the entire thing to find out the size. In those cases we want to save the content when we download it for later block reads. We want to download that data as soon as the file is accessed (Open is the earliest clear access) so we can update the size attribute of the file. Release could be used to free the buffered data.

So we want to provide a basic Read function for block access

type BlockReadable struct {
  Read(ctx context.Context, size int64, offset int64) ([]byte, error)
}

Files where you need to download the content to know the size should implement

type Readable struct {
  Read(context.Context) ([]byte, error)
}

External plugins have matching semantics

// Block-readable entries
<plugin_script> read <path> <state> <size> <offset> # data written to stdout
// All other readable entries
<plugin_script> read <path> <state> # data written to stdout

Here, we still preserve the methods array, but we add the following semantics for read:

read prefetching is not supported for block-readable entries.

ekinanp commented 4 years ago

Note that the [read, <block_readable>] tuple is a specialized case of the more general [<method>, <signature>] tuple. The latter could be useful for overloaded actions; overloading's a useful way to capture a common pattern that could be implemented in different ways.

ekinanp commented 4 years ago

Updated the issue to change BlockReadable's signature to size, offset b/c the latter results in a less wordy comment ("read size bits of content starting at offset")

ekinanp commented 4 years ago

https://github.com/puppetlabs/wash/pull/643 was the last PR needed for this work.