Closed jangorecki closed 6 years ago
To match other tools behavior, otherwise it should be marked as cache=FALSE as Impala and Presto were marked before. also to not suffer from writing to HDD it should persist only to memory, by default it uses RAM and HDD. https://spark.apache.org/docs/2.3.1/api/python/pyspark.sql.html#pyspark.sql.DataFrame.persist
cache=FALSE
To match other tools behavior, otherwise it should be marked as
cache=FALSEas Impala and Presto were marked before. also to not suffer from writing to HDD it should persist only to memory, by default it uses RAM and HDD. https://spark.apache.org/docs/2.3.1/api/python/pyspark.sql.html#pyspark.sql.DataFrame.persist