Give a feature to avoid loading all subsetted cohort in memory

[strayMat]

Feature type

utility

Description

We would like to avoid loading all data in memory persisting HiveData. Especially usefull when processing big cohort, where the actual code will blow memory.

Several proposition/directions for amelioration:

• Keep local storage but use garbage collector to avoid surcharging the memory. It does not sovle the issue but it scale to bigger cohort.
• Use HDFS instead of local storage to gain distributed and out-of-memory processing (as described in the aphp wiki). This is a bit APHP specific though.
• Write directly from spark to local file system by prefixing the path by "file://" in a spark.write.mode("overwrite").parquet("file://"+path) It would be the better solution in my opinion. However , actually the local spark does not have the write to write to local file (tested on cse210038 and the cse diabetes of Theo Jolivet).

[strayMat]

https://github.com/aphp/eds-scikit/pull/37 has implemented something close to solution 1. It still collect the data but dataframe by dataframe and parquet fragments by parquet fragments.