[FEATURE] Support Spark 3.5

[dai-chen]

Is your feature request related to a problem?

Support Spark 3.5.1

What solution would you like?

Upgrade Spark version to 3.5.1 and make sure no functional or performance regression.

What alternatives have you considered?

• [x] Cut feature branch for Spark 3.5.1
• [x] Made code changes and fix broken UT/IT
  • [x] https://github.com/opensearch-project/opensearch-spark/pull/351
  • [x] https://github.com/opensearch-project/opensearch-spark/pull/354
  • [ ] #443
• [ ] End-to-end integration test
  • [ ] Testing with OpenSearch and EMR-S Spark
  • [ ] Testing with Observability Integrations
• [ ] Performance test
  • [x] Direct SparkSQL query performance
  • [ ] Flint index building performance, including skipping/CV/MV
  • [ ] Query acceleration performance, including skipping/CV
• [ ] Release
  • [ ] Prepare final PR for changes in feature branch [TBD: refactor to support multi Spark version]
  • [x] Review and merge to main branch first
  • [x] Bump Flint main branch version to 0.6
  • [x] Create 0.5 branch
  • [ ] Build the latest artifact

Do you have any additional context?

N/A

[dai-chen]

Some issues and IT failure were tracked and fixed in https://github.com/opensearch-project/opensearch-spark/issues/331.

[dai-chen]

Cut feature branch for Spark 3.5.1 from previous 3.4.1 branch: https://github.com/opensearch-project/opensearch-spark/tree/spark-3.5.1.

Note that both feature branches are only for testing which may miss later commits from 0.4 branch. Eventually we should raise a consolidate PR to main branch first and then back port to 0.4 branch to release.