Write large dataframes to a directory

[ssimeonov]

Is it possible to write large dataframes the typical way they are handled in big data: one directory per dataframe? For example, it doesn't make sense to write a dataframe with 8+ billion rows to a single CSV file.

[samuel-pt]

Currently we are writing in a single file. We can add a parameter to write the results into multiple files.

[ssimeonov]

That would be very helpful since the current approach only works for small data.