Closed khajaasmath786 closed 6 years ago
The simplest way to hint usage of indices is to A) either create a dataframe using df.index(precision) or B) directly specify in the join as x.join(y index precision)
In both A and B the index will be created on the fly for the join
Hi,
I am bit confused with the posts on where to use the index factor on the data frames.
spark.read.format("magellan") .option("magellan.index", "true") .option("magellan.index.precision", "25") .load(s"$path")
OR PolygonDataframe.index(30) after loading the dataframe from GEOJSON file
points.join(polygons).where($"point" within $"polygon") // or
points.join(polygons index 30).where($"point" within $"polygon") do I need to still add index 30 as above after indicating it step 1 at the time of loading ?
can I simply use points.join(polygons).where($"point" within $"polygon") after loading initial dataframe asPolygonDataframe.index(30) ? will it still consider indexs in this case?