gzip 으로 압축하여 쓰기 - Githubissues

Gunju-Ko / TIL

Today I Learn

0 stars 0 forks source link

gzip 으로 압축하여 쓰기 #6

Open Gunju-Ko opened 3 years ago

Gunju-Ko commented 3 years ago

gzip으로 압축하여 쓰기

df.write.option("compression","gzip").csv("path")

출처 : https://stackoverflow.com/questions/40163996/how-to-save-a-dataframe-as-compressed-gzipped-csv

Gunju-Ko commented 3 years ago

rdd를 gzip으로 압축하여 쓰기

rdd.saveAsTextFile("/user/cloudera/sfpd.2", classOf[org.apache.hadoop.io.compress.GzipCodec])

출처 : https://sites.google.com/a/einext.com/einext_original/apache-spark/compress-output-files-in-spark