inventory_add with s3 accessor fails with a 416 error and attempts to read beyond file size.

[fr4gment]

Using the inventory_add function with s3 and trying to pull a text file of about 120B, I receive the following error

inventory_add: operation error S3: GetObject, https response error StatusCode: 416, RequestID: 17D6BDE74195C262, HostID: dd9025bab4ad464b049177c95eb6ebf374d3b3fd1af9251148b658df7ac2e3e8, api error InvalidRange: The requested range 'bytes=1048576-2097151' is not satisfiable

Using the AWS SDK to pull from s3, the s3manager.download function (when not specifying a range) EXPECTS a http 416 response to identify the EOF. However, since the velociraptor s3 reader handles the range values, the download function branches into a downloadRange, which does not handle the 416 error and emits it back to the calling function.

The code in the utils.Copy function (which is what inventory_add uses) also does not handle the 416 error (it's expecting an io.EOF error) and instead returns and passes it to the calling function.

            n, err = src.Read(*buff)
            if err != nil && err != io.EOF {
                return offset, err
            }

Notably, the upload function seems to work in this regard, as it too receives a 416 at the end of the download from s3 server but does not error out. Looking through the code I can see that the upload function seems to inadvertently handle the error because during the 416, 0 bytes will have been able to be read.

            n, err := reader.Read(buf) 
            if n == 0 || err == io.EOF {
                break loop
            }

Additionally, the read_file function appears to have the same issue as the inventory_add function when using the s3 accessor. If you attempt to read the entire file (without a length parameter) the function returns a 416 error.

To reproduce:

Set up an s3 bucket using something like aws or minio. Add a file to the s3 bucket. Using server side vql, run something like the following

LET S3_CREDENTIALS <= dict(
    endpoint='<s3 url>',
    credentials_key='<key>',
    credentials_secret='<secret>',
    region="<region>")

SELECT inventory_add(tool="newTool", file="/<path>/<to>/<file>", accessor=s3)

and you should receive the 416 error message in the VQL log.

To Fix

• Perform an LSTAT on the file and do not request data beyond the size of the file.

• Ideally, the s3manager within the aws sdk handles the 416 error as it does when not providing the download function with a range. Since that is an external dependency, consider having the s3 reader handle the 416 error (which is a non-standard error for an io.Reader to emit) and produce an io.EOF error instead.

[scudette]

Thanks for reporting