humio / issues

Issue Tracker for Humio
4 stars 2 forks source link

Structured data expectations should be formally specified (per parser, as needed) #110

Open atz opened 4 years ago

atz commented 4 years ago

For building custom code that sends structured data to humio, there should be a single language-independent machine-verifiable governing specification of what is or is not valid data. This would be used on our client side to test validity (i.e. to know we aren't sending invalid data), both in development and even in production.

Recommendation: use JSON-schema.

Why bother with formal description? It is not sufficient to say "well, our code should be able to parse it". The spec should not be an artifact of implementation, it should be a design contract that we can use to establish conformance across various languages, using different libraries, etc.

By versioning and publishing the spec independently of your implementation, you also signal concretely (and verifiably) what changes that callers need to care about. It allows you say things like "Version X of Humio's json parser supports version 16 and 17 of the json-parser spec, but version 15 has been obsoleted". Those differences are literally visible as diffs.

Such a spec would resolve outstanding difficulty we have knowing what our target document outputs are supposed to look like.