BLOB Store

[varnerac]

BLOB Store

The Issue

JSON is great for managing structured text. It's terrible for managing binary large objects (BLOBs).

• Base64 is inefficient
• Most JSON libraries decode the entire structure into memory without a SAX-like/streaming implementation

HTTP solutions have mechanisms in place to make dealing with large downloads easier:

• Range Requests for resumable downloads
• existing libraries handle POST from and GET to disk
• caching

Most databases aren't designed to handle BLOBs and structured text in the same object. STIX 2.0 recognizes this to some extent, providing a URL alternative to Base64-encoded binaries in the url field of the Artifact Object. A custom property could choose to use a similar scheme. However, how to handle this URL is left as an exercise for the implementor:

It is incumbent on object creators to ensure that the URL is accessible for downstream consumers.

The requirements around this URL-based BLOB offloading may include:

• Access controls similar to the STIX 2.x objects that reference them
• We find and authenticate to the URL the same way can retrieve the object, i.e. TAXII 2.x.
• TAXII 2.x implementations MAY verify the existence of a value at the URL before inserting the associated object. It makes no sense to add an Artifact with no value.

If we do not provide implementors a standardized mechanism to handle large BLOBs, we raise the barrier to entry. Proof of concept codebases may blow-up. Without a standardized way to upload BLOBs, it becomes impossible to upload an Artifact that exceeds an API Root's max_content_length. Clients will need to write custom JSON code to stream values larger than available RAM, etc.

STIX Patterning Language is another potential gotcha for large STIX/JSON objects. I don't see an existing way to offload a block of STIX Patterning Language that includes a massive Base64 binary. This could trip up Pattern Language lexers/parsers dealing with large BLOBs that exceed available memory.

The Proposed Solution

TAXII 2.x servers MAY implement the BLOB endpoint.

POST <api-root>/collections/<id>/blobs/
GET <api-root>/collections/<id>/blobs/<blob-id>/

Read/Write access to BLOBs will be governed by the existing can_read and can_write attributes on Collections.

I have deliberately chosen blob over a term like artifact because a custom property could reference a BLOB.

Sponsorship

NineFX will implement a prototype endpoint to support this.

[varnerac]

I guess we need a DELETE on this too.

[JasonKeirstead]

From my perspective, I do not see why this should be part of TAXII. TAXII is not supposed to be a database API, nor should it be trying to be the one-and-only API that a security product use.

[varnerac]

I think this should be optional. I don't think every server needs to implement it.

[jordan2175]

We know that organizations will want to, for example, share malware samples via TAXII. Doing this in STIX is really kind of ugly, and we know that. It would be nice if TAXII allowed you to deliver this. In fact, we have talked about needing this sort of feature for some time.

[JasonKeirstead]

@jordan2175 I think it is very weird to assume someone wants to POST and GET raw malware samples without any STIX, via TAXII. That use case makes no sense to me at all. All of the use cases around sharing malware with TAXII include the context in a STIX document. Otherwise you've just make a really complicated API around an object store.

TAXII is not an FTP server or a web object store. Object stores already exist, we should not be trying to re-invent the wheel here.

[varnerac]

The BLOBs would not be posted without STIX. They'd be tied via the url field of STIX Artifact objects in the TAXII server.