Palantir Spark Diff on v3.0.2

jdcasale commented 3 years ago

This PR will rebase palantir/spark on top of upstream 3.0.2. It has the Palantir essentials, a few changes not merged upstream, and a few cherry-picks that weren't released yet. Details on that below. General suggestion is to look at each commit in isolation, as it should represent an atomic change that can be reviewed independent of the other changes. This is a big diff, (sorry) but we don't really have a better way to manage this change.

This code has already been tested internally on f-s and downstream products.

What happens to our master

The diff you're looking at is comparing against a branch tracking upstream's v3.0.2 tag. If you approve, we'll apply this set of Palantir commits on top of v3.0.2, and then do a 3-way merge with master where we throw away master's changes but keep its commit history (like here).

Changes

Obviously we have Spark 3 now instead of being mostly Spark 2. While fixing merge conflicts we tried to change as little as we could. No tests got dropped and we're not expecting any behavioural changes.

However, we dropped some "logical items" listed in our FORK.md. That's either because they were taken upstream, or because we no longer need the change internally. Below is the annotated list with what we did with the respective item. Items not listed there were probably dropped.

Items listed in FORK.md

Differences with upstream

SPARK-21195 - Kept: Automatically register new metrics from sources and wire default registry
SPARK-20952 - Kept: ParquetFileFormat should forward TaskContext to its forkjoinpool
SPARK-20001 / SPARK-13587 - Kept: Support PythonRunner executing inside a Conda env (and R)
SPARK-17059 - Dropped: (fixed with DSv2) ~Allow FileFormat to specify partition pruning strategy via splits~
SPARK-24345 - Dropped: (merged upstream) ~Improve ParseError stop location when offending symbol is a token~
SPARK-23795 - Dropped: (no longer needed) ~Make AbstractLauncher#self() protected~
SPARK-18079 - Kept: CollectLimitExec.executeToIterator should perform per-partition limits
SPARK-15777 - Kept: Catalog federation changes
SPARK-17091 - Dropped: (merged upstream) ~Better pushdown for IN expressions in parquet via UserDefinedPredicate~
SPARK-26626 - Kept: Limited the maximum size of repeatedly substituted aliases
Kept: SafeLogging implemented in a bunch of files

Additions

All these are kept:

Gradle plugin to easily create custom docker images for use with k8s
Filter rLibDir by exists so that daemon.R references the correct file (#460)
Add pre-installed conda configuration and use to find rlib directory (#700)
Supports Arrow-serialization of Python 2 strings (#678)
SPARK-26200 - Allow setting HADOOP_CONF_DIR as a spark config
K8s local file mounting
K8s local deploy mode

Reverts

These were breaking changes we had reverted. Most of these were later undone upstream, so we no longer need to track reverts.

SPARK-25908 - ~Removal of monotonicall_increasing_id, toDegree, toRadians, approxCountDistinct, unionAll~
SPARK-25862 - ~Removal of unboundedPreceding, unboundedFollowing, currentRow~
SPARK-26127 - ~Removal of deprecated setters from tree regression and classification models~
SPARK-25867 - ~Removal of KMeans computeCost~
SPARK-26216 - ~Change to UserDefinedFunction type~
- SPARK-26323 - ~Scala UDF null checking~
- SPARK-26580 - ~Bring back scala 2.11 behaviour of primitive types null behaviour~
SPARK-26133 - ~Old OneHotEncoder~ (accepting a break)
SPARK-11215 - ~StringIndexer multi column support~ (accepting a break here from 3.1)
SPARK-26616 - ~No document frequency in IDFModel~ (not a break)

jdcasale commented 3 years ago

Will squash the last few docs commits together before merging (don't want to do it now because it breaks review comment links)