More comprehensive test suite and data

[marshallmain]

We now have a number of features in the tooling focused on custom schemas and subsets that don't have great test coverage - partly due to the pain of creating suitable test data that uses these new features. It would be nice to make a test data fixture that can be used in multiple tests without having to manually create data each time. We should also improve the test coverage of these features. The endpoint-app-team repo https://github.com/elastic/endpoint-app-team has custom schemas and subsets that would make a good starting point for a test fixture since they use the tooling features extensively.

[webmat]

Agreed, I'd love to improve the test coverage for the generator. There's two ways of doing this that I think we could tackle.

The most beneficial one might be to do integration tests where none of the Python code is called directly. Instead shell commands are issued to test each of the CLI flags, output to a temp directory, and the tests confirm the results are as expected. Which files were generated, do they include/exclude what was expected and so on.

I also think we could do more in the unit tests, perhaps sharing basic test fixtures. However I tend to be cautious there, and start by being explicit in each test, then generalize fixtures only when there's obvious commonalities.

[webmat]

@marshallmain and @jonathan-buttner, I'd appreciate if you could drop a few test cases in this issue, that you'd like the ECS tooling to support, and have integration tests for.

Just a one sentence explanation per test case.

Here's a few that come to mind right away:

• it should be possible to merge different field sets to ECS
• it should be possible to insert custom fields inside existing ECS field sets
• it should be possible to insert additional nesting locations to an existing ECS field set

So I'm looking for any additional ones, including subtle variations of the above if necessary.

Thanks!

[jonathan-buttner]

@webmat Here are some test cases that might be useful:

• it should be possible to retrieve a subset of the ECS and custom fields
• it should be possible to disable (mark as not indexed by ES) ECS and custom fields in the template
• it should not be possible to redefine an ECS field with a custom field
• it should not be possible to define multiple definitions for a custom field

[webmat]

Your first one is interesting. I recently hit that specifically: I was adding custom fields, and wanted to only include a few ECS fields. I was surprised that I had to specifically whitelist my custom fields via --subset.

We do have to pick which one overrides the other, and having subset override custom makes the feature ultimately more flexible. But this initially surprised me.