[FEAT] Add initial Spark Connect support

[andrewgazelka]

Implements basic Spark Connect functionality in Daft with the following:

• Add daft-connect crate to handle Spark Connect protocol
• Implement configuration management via Spark Connect API
• Add Python bindings with connect_start() function
• Add integration tests for config operations

Currently supports:

• Basic session management
• Config operations (Set, Get, GetWithDefault, GetOption, GetAll, Unset)
• Error handling and status reporting

Notable changes:

• New dependency on spark-connect protocol
• Added tracing for debugging and monitoring
• Integration with existing Daft infrastructure

Some operations like execute_plan, analyze_plan, and artifact handling are currently unimplemented and will return appropriate error messages.

[codspeed-hq[bot]]

CodSpeed Performance Report

Merging #3261 will degrade performances by 34.58%

_{Comparing andrew/connect-config (a3c6037) with main (f290f40)}

Summary

⚡ 1 improvements ❌ 2 regressions ✅ 14 untouched benchmarks

:warning: Please fix the performance issues or acknowledge them on CodSpeed.

Benchmarks breakdown

	Benchmark	main	andrew/connect-config	Change
❌	test_count[1 Small File]	3.3 ms	4 ms	-17.44%
❌	test_iter_rows_first_row[100 Small Files]	177.3 ms	271 ms	-34.58%
⚡	test_show[100 Small Files]	51.1 ms	42.1 ms	+21.5%

[codecov[bot]]

Codecov Report

Attention: Patch coverage is 69.48052% with 94 lines in your changes missing coverage. Please review.

Project coverage is 77.78%. Comparing base (f290f40) to head (a3c6037). Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
src/daft-connect/src/lib.rs	58.67%	50 Missing :warning:
src/daft-connect/src/main.rs	0.00%	33 Missing :warning:
src/daft-table/src/lib.rs	0.00%	8 Missing :warning:
src/daft-connect/src/config.rs	95.00%	3 Missing :warning:

Additional details and impacted files

[](https://app.codecov.io/gh/Eventual-Inc/Daft/pull/3261?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Eventual-Inc) ```diff @@ Coverage Diff @@ ## main #3261 +/- ## ========================================== + Coverage 76.07% 77.78% +1.70% ========================================== Files 644 650 +6 Lines 81581 80350 -1231 ========================================== + Hits 62065 62501 +436 + Misses 19516 17849 -1667 ``` | [Files with missing lines](https://app.codecov.io/gh/Eventual-Inc/Daft/pull/3261?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Eventual-Inc) | Coverage Δ | | |---|---|---| | [src/daft-connect/src/session.rs](https://app.codecov.io/gh/Eventual-Inc/Daft/pull/3261?src=pr&el=tree&filepath=src%2Fdaft-connect%2Fsrc%2Fsession.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Eventual-Inc#diff-c3JjL2RhZnQtY29ubmVjdC9zcmMvc2Vzc2lvbi5ycw==) | `100.00% <100.00%> (ø)` | | | [src/daft-connect/src/util.rs](https://app.codecov.io/gh/Eventual-Inc/Daft/pull/3261?src=pr&el=tree&filepath=src%2Fdaft-connect%2Fsrc%2Futil.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Eventual-Inc#diff-c3JjL2RhZnQtY29ubmVjdC9zcmMvdXRpbC5ycw==) | `100.00% <100.00%> (ø)` | | | [src/daft-local-execution/src/lib.rs](https://app.codecov.io/gh/Eventual-Inc/Daft/pull/3261?src=pr&el=tree&filepath=src%2Fdaft-local-execution%2Fsrc%2Flib.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Eventual-Inc#diff-c3JjL2RhZnQtbG9jYWwtZXhlY3V0aW9uL3NyYy9saWIucnM=) | `92.85% <ø> (ø)` | | | [src/lib.rs](https://app.codecov.io/gh/Eventual-Inc/Daft/pull/3261?src=pr&el=tree&filepath=src%2Flib.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Eventual-Inc#diff-c3JjL2xpYi5ycw==) | `95.77% <100.00%> (+0.06%)` | :arrow_up: | | [src/daft-connect/src/config.rs](https://app.codecov.io/gh/Eventual-Inc/Daft/pull/3261?src=pr&el=tree&filepath=src%2Fdaft-connect%2Fsrc%2Fconfig.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Eventual-Inc#diff-c3JjL2RhZnQtY29ubmVjdC9zcmMvY29uZmlnLnJz) | `95.00% <95.00%> (ø)` | | | [src/daft-table/src/lib.rs](https://app.codecov.io/gh/Eventual-Inc/Daft/pull/3261?src=pr&el=tree&filepath=src%2Fdaft-table%2Fsrc%2Flib.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Eventual-Inc#diff-c3JjL2RhZnQtdGFibGUvc3JjL2xpYi5ycw==) | `88.59% <0.00%> (-1.17%)` | :arrow_down: | | [src/daft-connect/src/main.rs](https://app.codecov.io/gh/Eventual-Inc/Daft/pull/3261?src=pr&el=tree&filepath=src%2Fdaft-connect%2Fsrc%2Fmain.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Eventual-Inc#diff-c3JjL2RhZnQtY29ubmVjdC9zcmMvbWFpbi5ycw==) | `0.00% <0.00%> (ø)` | | | [src/daft-connect/src/lib.rs](https://app.codecov.io/gh/Eventual-Inc/Daft/pull/3261?src=pr&el=tree&filepath=src%2Fdaft-connect%2Fsrc%2Flib.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Eventual-Inc#diff-c3JjL2RhZnQtY29ubmVjdC9zcmMvbGliLnJz) | `58.67% <58.67%> (ø)` | | ... and [27 files with indirect coverage changes](https://app.codecov.io/gh/Eventual-Inc/Daft/pull/3261/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Eventual-Inc)