search

luminousmen / luminousmen.com

2 stars 0 forks source link

https://luminousmen.com/post/azure-blob-storage-with-pyspark #10

Closed utterances-bot closed 3 weeks ago

utterances-bot commented 4 years ago

Azure Blob Storage with Pyspark - Blog | luminousmen

Azure Blob Storage is a Microsoft solution for storing objects in the cloud. It is optimized for storing large amounts of data and can be easily accessed by your Python/spark application

https://luminousmen.com/post/azure-blob-storage-with-pyspark

pmarct commented 4 years ago

session.write doesn't seem to be valid. Is there any way to write/send a parquet to the blob with pyspark?

luminousmen commented 4 years ago

@palani-thangaraj, it is really not valid. Where did you find it in the article?

I guess what you trying to achive is writing to azure blob and write should be called from DataFrame but not session. Something like sdf.write