Check if field is existent (but ignore its value)

[timsn]

It would be nice to have the option to check if a field is existent after logstash has processed a data point.

As I understand it, it's currently only possible check for equal values or to ignore the field completly. But especially for newly created fields which are getting populated by some sort of mapping it could be helpful to have a check if such field was created at all (so if it was mapped to something).

Another idea is to have the possiblity to check if a field matches a regular expression. So that you don't have to know the exact value of the field while writing the test but you can make sure that it at least matches a well defined pattern.

[magnusbaeck]

Yes, this would definitely be useful. Have you given any thoughts as to how this could be expressed in the testcase file? Field values with special meanings that LFV interprets, like

"foo": {
  "_predicate": "matches-regexp",
  "regexp": "^\\d+$"
}

to make sure the foo field is a string containing an integer? This would require that we stop relying (solely) on an external diff engine and instead diff the events ourselves. That would be a pretty substantial change.

[magnusbaeck]

Instead of inventing our own language we should provide an ability to use jq to evaluate the JSON blob produced by Logstash, i.e. instead of diffing it against a JSON blob from the testcase file we'd use a jq expression from the testcase file and evaluate against the Logstash output.