search

grapl-security / grapl

Graph platform for Detection and Response
https://www.graplsecurity.com
Apache License 2.0
686 stars 78 forks source link

Provision Scylla during `grapl-provision` #2074

Closed wimax-grapl closed 2 years ago

wimax-grapl commented 2 years ago

Which issue does this PR correspond to?

This is a followup to https://github.com/grapl-security/grapl/pull/2037 and a stab at https://github.com/grapl-security/issue-tracker/issues/1042

What changes does this PR make to Grapl? Why?

this PR is a stab at greatly simplifying the scylla provision process by doing it during grapl-provision.

as we currently see in these logs: https://grapl-buildkite-artifacts-ecfff1a.s3.amazonaws.com/0183f6a6-741e-4fc3-99b4-bd57fff4b7b4/test_artifacts/2022-10-20T18-55-21/grapl-graph-db/scylla-provisioner.stdout.log

the "if already provisioned, do nothing" behavior Colin has in there doesn't at all if we invoke the RPC 8 times all at once from different tests!

worse, we are seeing some flaky weird errors around scylla:

Error: encountered protocol error status: Unknown, message: "Property query failed: DbError(DbError(ServerError, \"Can't find a column family with UUID a1e331b0-5153-11ed-8789-ba1d40420f58: std::out_of_range (_Map_base::at)\"))"
Error: encountered protocol error status: Unknown, message: "Property query failed: DbError(DbError(Invalid, \"unconfigured table edges\"))"

both of these are from this build: https://buildkite.com/grapl/grapl-verify/builds/4865

How were these changes tested?

CI right here right now

codecov[bot] commented 2 years ago

Codecov Report

Base: 39.42% // Head: 39.20% // Decreases project coverage by -0.21% :warning:

Coverage data is based on head (e15482e) compared to base (3109f73). Patch coverage: 0.00% of modified lines in pull request are covered.

Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #2074 +/- ## ========================================== - Coverage 39.42% 39.20% -0.22% ========================================== Files 434 436 +2 Lines 10148 10207 +59 ========================================== + Hits 4001 4002 +1 - Misses 6147 6205 +58 ``` | [Impacted Files](https://codecov.io/gh/grapl-security/grapl/pull/2074?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=grapl-security) | Coverage Δ | | |---|---|---| | [pulumi/grapl/\_\_main\_\_.py](https://codecov.io/gh/grapl-security/grapl/pull/2074/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=grapl-security#diff-cHVsdW1pL2dyYXBsL19fbWFpbl9fLnB5) | `0.00% <ø> (ø)` | | | [src/python/provisioner/provisioner/app.py](https://codecov.io/gh/grapl-security/grapl/pull/2074/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=grapl-security#diff-c3JjL3B5dGhvbi9wcm92aXNpb25lci9wcm92aXNpb25lci9hcHAucHk=) | `0.00% <0.00%> (ø)` | | | [...hon\_proto/api/scylla\_provisioner/v1beta1/client.py](https://codecov.io/gh/grapl-security/grapl/pull/2074/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=grapl-security#diff-c3JjL3B5dGhvbi9weXRob24tcHJvdG8vcHl0aG9uX3Byb3RvL2FwaS9zY3lsbGFfcHJvdmlzaW9uZXIvdjFiZXRhMS9jbGllbnQucHk=) | `0.00% <0.00%> (ø)` | | | [...n\_proto/api/scylla\_provisioner/v1beta1/messages.py](https://codecov.io/gh/grapl-security/grapl/pull/2074/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=grapl-security#diff-c3JjL3B5dGhvbi9weXRob24tcHJvdG8vcHl0aG9uX3Byb3RvL2FwaS9zY3lsbGFfcHJvdmlzaW9uZXIvdjFiZXRhMS9tZXNzYWdlcy5weQ==) | `0.00% <0.00%> (ø)` | | | [src/python/python-proto/python\_proto/client.py](https://codecov.io/gh/grapl-security/grapl/pull/2074/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=grapl-security#diff-c3JjL3B5dGhvbi9weXRob24tcHJvdG8vcHl0aG9uX3Byb3RvL2NsaWVudC5weQ==) | `0.00% <0.00%> (ø)` | | | [src/rust/e2e-tests/src/test\_utils/context.rs](https://codecov.io/gh/grapl-security/grapl/pull/2074/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=grapl-security#diff-c3JjL3J1c3QvZTJlLXRlc3RzL3NyYy90ZXN0X3V0aWxzL2NvbnRleHQucnM=) | `0.00% <ø> (ø)` | | | [src/rust/sysmon-parser/src/util.rs](https://codecov.io/gh/grapl-security/grapl/pull/2074/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=grapl-security#diff-c3JjL3J1c3Qvc3lzbW9uLXBhcnNlci9zcmMvdXRpbC5ycw==) | `42.28% <0.00%> (+0.67%)` | :arrow_up: | Help us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=grapl-security). Have a feature suggestion? [Share it here.](https://app.codecov.io/gh/feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=grapl-security)

:umbrella: View full report at Codecov.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.