This pull request introduces a optimization to run policy queries in parallel. This could be useful for running in large scale of tests (200k+ targets for example).
The change is to parallelize the policy queries, while running queries in fixed numbers of worker. Since rego rule evaluation is a CPU bound task, the current implementation uses a default workers count of runtime.GOMAXPROCS(0).
To verify the change, this pull request includes a perf test for running tests on 20000 targets, here is the test results:
env
without change
with change
local (Apple M1 pro)
16.379325125s (CPU=10 files=200, rules=100)
10.400924167s (CPU=10 files=200, rules=100)
github ci
1m17.09322013s (CPU=4 files=200, rules=100)
40.099342258s (CPU=4 files=200, rules=100)
As a comparison, the results for running 200k+ targets from my local environment:
This pull request introduces a optimization to run policy queries in parallel. This could be useful for running in large scale of tests (200k+ targets for example).
The change is to parallelize the policy queries, while running queries in fixed numbers of worker. Since rego rule evaluation is a CPU bound task, the current implementation uses a default workers count of
runtime.GOMAXPROCS(0).To verify the change, this pull request includes a perf test for running tests on 20000 targets, here is the test results:
16.379325125s (CPU=10 files=200, rules=100)10.400924167s (CPU=10 files=200, rules=100)1m17.09322013s (CPU=4 files=200, rules=100)40.099342258s (CPU=4 files=200, rules=100)As a comparison, the results for running 200k+ targets from my local environment:
As we can see from above results, this change can reduce the test execution time by 100%~200%.