Closed Thomzoy closed 1 year ago
Patch coverage: 100.00
% and project coverage change: +0.01
:tada:
Comparison is base (
6ec366e
) 91.88% compared to head (666b6ad
) 91.90%.
:mega: This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more
:umbrella: View full report at Codecov.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.
This PR will probably fix https://github.com/aphp/eds-scikit/issues/36
Hi, thanks for the PR!
The main point I'd like to address is that I don't quite see the benefit of using a class since the same result could be achieved with a
write_batches_to_hdfs()
utils function.To do so,
__init__
,get_pd_fragment
, andget_pd_table
could be merged into one single function.What do you think?
Yep good call !
Description
The current method to save a subcohort locally relies on the
toPandas
method of PySpark, which isn't efficient and prone to errors.This PR implements a
pyarrow
connector which reads the parquet file directlyChecklist