search

fingltd / 4mc

4mc - splittable lz4 and zstd in hadoop/spark/flink
Other
108 stars 36 forks source link

Create DataSet using SparkSession #26

Open santhoshtangudu opened 7 years ago

santhoshtangudu commented 7 years ago

Hi, We have 4mc format files in my Hadoop cluster. We are trying to read these files and create DataSet (instead of creating RDD and then DataSet) in spark-2.0. Can you please us to do the same?

carlomedas commented 7 years ago

For sure there are several methods to achieve that, and to be honest I'm not sure I'm giving you best solution. Unfortunately at this time I don't have time to dig deeper, but: what about, you load the RDD using a SQLContext, which could be pre-filtered etc etc, then you create a bean class that can be used to quickly map to Dataset by leveraging convention over configuration, like e.g: Dataset devices = sqlContext.createDataFrame(ratingsRDD, DeviceEntry.class);