nodestream-proj / nodestream

A Declarative framework for Building, Maintaining, and Analyzing Graph Data
https://nodestream-proj.github.io/docs/
Apache License 2.0
36 stars 11 forks source link

Improve Neo4j Query Performance By Using MERGE #272

Closed zprobst closed 6 months ago

zprobst commented 6 months ago

Due to some internal analysis done by intuit, while this query does perform better is certain situations, in others it can cause memory consumption spikes in the 10s of GBs on the cluster and can cause stability and consistency issues. This PR approaches using the more obvious MERGE path and leaving it to the database to execute this more in a stable manor. This PR also adds several settings (currently undocumented) that allow for some tweaking of the underlying batch performance.

NOTE: Docs excluded from this PR pending the docs revamp.

codecov[bot] commented 6 months ago

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 96.02%. Comparing base (94f09f8) to head (62f94cd).

Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #272 +/- ## ======================================= Coverage 96.01% 96.02% ======================================= Files 133 133 Lines 3968 3976 +8 ======================================= + Hits 3810 3818 +8 Misses 158 158 ``` | [Flag](https://app.codecov.io/gh/nodestream-proj/nodestream/pull/272/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=nodestream-proj) | Coverage Δ | | |---|---|---| | [3.10-macos-latest](https://app.codecov.io/gh/nodestream-proj/nodestream/pull/272/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=nodestream-proj) | `96.02% <100.00%> (+<0.01%)` | :arrow_up: | | [3.10-ubuntu-latest](https://app.codecov.io/gh/nodestream-proj/nodestream/pull/272/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=nodestream-proj) | `96.02% <100.00%> (+<0.01%)` | :arrow_up: | | [3.10-windows-latest](https://app.codecov.io/gh/nodestream-proj/nodestream/pull/272/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=nodestream-proj) | `95.93% <100.00%> (+<0.01%)` | :arrow_up: | | [3.11-macos-latest](https://app.codecov.io/gh/nodestream-proj/nodestream/pull/272/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=nodestream-proj) | `96.02% <100.00%> (+<0.01%)` | :arrow_up: | | [3.11-ubuntu-latest](https://app.codecov.io/gh/nodestream-proj/nodestream/pull/272/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=nodestream-proj) | `96.02% <100.00%> (+<0.01%)` | :arrow_up: | | [3.11-windows-latest](https://app.codecov.io/gh/nodestream-proj/nodestream/pull/272/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=nodestream-proj) | `95.93% <100.00%> (+<0.01%)` | :arrow_up: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=nodestream-proj#carryforward-flags-in-the-pull-request-comment) to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.