Spark 3.2

[echeipesh]

Pushing for Spark 3.2

• [x] get tests green
• [x] update to spark-testing-base for more speed
• [x] get code review and/or cleanup the changes
• [x] figure out what to do with GeoTrellis layer data source
• [x] update python package
  • ~cleanup dependencies: disentangle the release wheel from pweave~
  • ~grok the python environment, consider introducing poetry instead of conda~
• [x] run some jobs in spark 3.2 environment

[pomadchin]

I'm going to bump the min Spark version in GT as well https://github.com/locationtech/geotrellis/pull/3389 Thogh it should not affect this PR much.

[echeipesh]

@pomadchin It'd be nice to get 3.6.3 out with spark 3.2.0 and then 3.7.0 with CES3 and spark 3.2.1. That'd allow this to sync with frameless 0.11 and later with 0.12+.

[pomadchin]

@echeipesh 🚀 https://github.com/locationtech/geotrellis/pull/3471 + I'm dropping Spark 3.0 over there https://github.com/typelevel/frameless/pull/637 (frameless will be published for 3.1.x, 3.2.x, and 3.3.x)

[pomadchin]

CI is dead? https://app.circleci.com/pipelines/github/locationtech/rasterframes?utm_campaign=workflow-failed&utm_medium=email&utm_source=notification

[echeipesh]

I've been looking at the python side of the project enough to form strong enough opinions of it. I think it's actually going to need some work and it doesn't make sense to do it in this PR.

• make the build PEP 517 compatible
• ditch use of conda for poetry (we're not building a conda package here anyway)
• separate the the pweave deps from the release deps
• decide if shipping jars with .whl still makes sense
  • it was a nice attempt at nicety but unless you start the process from ipython instead of pyspark it is just dead weight.
• pyrasterframes python files are in the sbt project structure
  • Kind of the only benefit of this is to couple the production of assembly jar with the building of the .whl
  • Not sure the weirdness of this justifies the setup

So that's a lot of moving stuff around potentially and it would be much cleaner to make that it's own PR.