endgameinc / eqllib

MIT License
158 stars 46 forks source link

Normalization fails for optimizing unique_count pipe #32

Closed rw-access closed 4 years ago

rw-access commented 4 years ago

Describe the bug

I automatically call optimize on all BaseNodes. But only expressions need to be optimized. There was an issue where | unique_count wouldn't get optimized correctly. The bug could probably be fixed in eql directly or here. I found a workaround here, so this works too.

To Reproduce

Try to normalize a query with | unique_count in it.

Expected behavior

Not an exception

Screenshots

Got this traceback

Traceback (most recent call last):
  File "/Users/rwolf/.conda/envs/eqllib/lib/python3.7/unittest/case.py", line 59, in testPartExecutor
    yield
  File "/Users/rwolf/.conda/envs/eqllib/lib/python3.7/unittest/case.py", line 615, in run
    testMethod()
  File "/Users/rwolf/Development/eqllib/tests/test_normalization.py", line 119, in test_unmodified_unique_count
    normalizer.normalize_ast(original)
  File "/Users/rwolf/Development/eqllib/eqllib/normalization.py", line 244, in normalize_ast
    return QueryNormalizer(self).walk(node)
  File "/Users/rwolf/Development/eqllib/eqllib/normalization.py", line 33, in walk
    node = super(QueryNormalizer, self).walk(node, *args, **kwargs)
  File "/Users/rwolf/.conda/envs/eqllib/lib/python3.7/site-packages/eql/walkers.py", line 225, in walk
    slots = [self.walk(v, *args, **kwargs) for name, v in node.iter_slots()]
  File "/Users/rwolf/.conda/envs/eqllib/lib/python3.7/site-packages/eql/walkers.py", line 225, in <listcomp>
    slots = [self.walk(v, *args, **kwargs) for name, v in node.iter_slots()]
  File "/Users/rwolf/Development/eqllib/eqllib/normalization.py", line 33, in walk
    node = super(QueryNormalizer, self).walk(node, *args, **kwargs)
  File "/Users/rwolf/.conda/envs/eqllib/lib/python3.7/site-packages/eql/walkers.py", line 216, in walk
    rv = self.autowalk(node, *args, **kwargs)
  File "/Users/rwolf/.conda/envs/eqllib/lib/python3.7/site-packages/eql/walkers.py", line 178, in autowalk
    return [self.walk(n, *args, **kwargs) for n in node]
  File "/Users/rwolf/.conda/envs/eqllib/lib/python3.7/site-packages/eql/walkers.py", line 178, in <listcomp>
    return [self.walk(n, *args, **kwargs) for n in node]
  File "/Users/rwolf/Development/eqllib/eqllib/normalization.py", line 35, in walk
    node = node.optimize()
  File "/Users/rwolf/.conda/envs/eqllib/lib/python3.7/site-packages/eql/ast.py", line 100, in optimize
    return Optimizer(recursive=recursive).walk(self)
  File "/Users/rwolf/.conda/envs/eqllib/lib/python3.7/site-packages/eql/optimizer.py", line 24, in walk
    return Walker.walk(self, node, *args, **kwargs)
  File "/Users/rwolf/.conda/envs/eqllib/lib/python3.7/site-packages/eql/walkers.py", line 188, in walk
    rv = self.autowalk(node, *args, **kwargs)
  File "/Users/rwolf/.conda/envs/eqllib/lib/python3.7/site-packages/eql/walkers.py", line 184, in autowalk
    return dict({self.walk(k, *args, **kwargs): self.walk(v, *args, **kwargs) for k, v in node.items()})
  File "/Users/rwolf/.conda/envs/eqllib/lib/python3.7/contextlib.py", line 119, in __exit__
    next(self.gen)
  File "/Users/rwolf/.conda/envs/eqllib/lib/python3.7/site-packages/eql/walkers.py", line 172, in set_context
    exit_method(node)
  File "/Users/rwolf/.conda/envs/eqllib/lib/python3.7/site-packages/eql/walkers.py", line 116, in _exit_pipe_command
    output_schemas = node.output_schemas([NodeInfo(a, TypeHint.Unknown) for a in node.arguments], incoming_schema)
  File "/Users/rwolf/.conda/envs/eqllib/lib/python3.7/site-packages/eql/pipes.py", line 129, in output_schemas
    first_event_type, = event_schemas[0].schema.keys()
IndexError: list index out of range