[CHORE] connect: Optimize plans in connect

[colin-ho]

1. Optimize logical plans before executing them in connect.
2. Run the native executor on a spawned task on the connect runtime instead of a dedicated thread. The native executor already spawns it's own thread, and it also returns an awaitable receiver, so we should be able to just receive from the connect runtime.

[codspeed-hq[bot]]

CodSpeed Performance Report

Merging #3378 will degrade performances by 23.5%

_{Comparing colin/optimize-connect-plan (76d8a7f) with main (d9d916b)}

Summary

❌ 2 regressions
✅ 15 untouched benchmarks

:warning: Please fix the performance issues or acknowledge them on CodSpeed.

Benchmarks breakdown

	Benchmark	main	colin/optimize-connect-plan	Change
❌	test_count[1 Small File]	3.5 ms	3.9 ms	-10.7%
❌	test_iter_rows_first_row[100 Small Files]	184.3 ms	240.9 ms	-23.5%

[codecov[bot]]

Codecov Report

Attention: Patch coverage is 89.70588% with 14 lines in your changes missing coverage. Please review.

Project coverage is 77.43%. Comparing base (5fee192) to head (76d8a7f). Report is 2 commits behind head on main.

Files with missing lines	Patch %	Lines
src/daft-logical-plan/src/builder.rs	74.28%	9 Missing :warning:
src/daft-local-execution/src/run.rs	96.34%	3 Missing :warning:
src/daft-connect/src/op/execute/root.rs	86.66%	2 Missing :warning:

Additional details and impacted files

[](https://app.codecov.io/gh/Eventual-Inc/Daft/pull/3378?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Eventual-Inc) ```diff @@ Coverage Diff @@ ## main #3378 +/- ## ========================================== + Coverage 77.41% 77.43% +0.02% ========================================== Files 678 678 Lines 83680 83307 -373 ========================================== - Hits 64783 64511 -272 + Misses 18897 18796 -101 ``` | [Files with missing lines](https://app.codecov.io/gh/Eventual-Inc/Daft/pull/3378?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Eventual-Inc) | Coverage Δ | | |---|---|---| | [src/daft-local-execution/src/channel.rs](https://app.codecov.io/gh/Eventual-Inc/Daft/pull/3378?src=pr&el=tree&filepath=src%2Fdaft-local-execution%2Fsrc%2Fchannel.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Eventual-Inc#diff-c3JjL2RhZnQtbG9jYWwtZXhlY3V0aW9uL3NyYy9jaGFubmVsLnJz) | `98.24% <100.00%> (+0.09%)` | :arrow_up: | | [src/daft-local-execution/src/lib.rs](https://app.codecov.io/gh/Eventual-Inc/Daft/pull/3378?src=pr&el=tree&filepath=src%2Fdaft-local-execution%2Fsrc%2Flib.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Eventual-Inc#diff-c3JjL2RhZnQtbG9jYWwtZXhlY3V0aW9uL3NyYy9saWIucnM=) | `95.29% <100.00%> (ø)` | | | [src/daft-connect/src/op/execute/root.rs](https://app.codecov.io/gh/Eventual-Inc/Daft/pull/3378?src=pr&el=tree&filepath=src%2Fdaft-connect%2Fsrc%2Fop%2Fexecute%2Froot.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Eventual-Inc#diff-c3JjL2RhZnQtY29ubmVjdC9zcmMvb3AvZXhlY3V0ZS9yb290LnJz) | `95.65% <86.66%> (-2.27%)` | :arrow_down: | | [src/daft-local-execution/src/run.rs](https://app.codecov.io/gh/Eventual-Inc/Daft/pull/3378?src=pr&el=tree&filepath=src%2Fdaft-local-execution%2Fsrc%2Frun.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Eventual-Inc#diff-c3JjL2RhZnQtbG9jYWwtZXhlY3V0aW9uL3NyYy9ydW4ucnM=) | `89.47% <96.34%> (+2.14%)` | :arrow_up: | | [src/daft-logical-plan/src/builder.rs](https://app.codecov.io/gh/Eventual-Inc/Daft/pull/3378?src=pr&el=tree&filepath=src%2Fdaft-logical-plan%2Fsrc%2Fbuilder.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Eventual-Inc#diff-c3JjL2RhZnQtbG9naWNhbC1wbGFuL3NyYy9idWlsZGVyLnJz) | `91.66% <74.28%> (+0.07%)` | :arrow_up: | ... and [10 files with indirect coverage changes](https://app.codecov.io/gh/Eventual-Inc/Daft/pull/3378/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Eventual-Inc)

---

🚨 Try these New Features:

• Flaky Tests Detection - Detect and resolve failed and flaky tests

[graphite-app[bot]]

Graphite Automations

"Request reviewers once CI passes" took an action on this PR • (11/21/24)

1 reviewer was added to this PR based on Andrew Gazelka's automation.