Hello! Excuse for troubling. I read with that driver 150k very small files. I will get dataframe on 150k partitions and that df not good for a large number of operations. df.coalesce work so long. Can I choose partitions manually on read? Thank you in advice!
This isn't related to this library. You just need to coalesce(). That itself is not an expensive operation. But you will start with a partition per (small) file in Spark.
Hello! Excuse for troubling. I read with that driver 150k very small files. I will get dataframe on 150k partitions and that df not good for a large number of operations. df.coalesce work so long. Can I choose partitions manually on read? Thank you in advice!