ddotta / parquetize

R package that allows to convert databases of different formats to parquet format
https://ddotta.github.io/parquetize/
62 stars 4 forks source link

evol: add an option to check arguments passed by user #41

Open nbc opened 1 year ago

nbc commented 1 year ago

Hi @ddotta,

A first implementation to allow arguments checking.

We can do the same thing to check the returned dataset. It will be even simpler, the function will just take the dataset object.

Regards

ddotta commented 1 year ago

I ❤️ the idea @nbc! The same logic with package withr could be used to disallow missing values in partitioning variables in returned dataset?

nbc commented 1 year ago

I heart the idea @nbc! The same logic with package withr could be used to disallow missing values in partitioning variables in returned dataset?

There's two way to do it :

check the returned dataset object

We use the same method to check the returned object and warn/fail if it's not ok for us.

This way is more flexible as we can check if there's NA and the number of partition files for example.

really check the values of partioning columns for NA

We use something like this :

if (any(is.na(data[partitioning])))) cli_abort("bla bla bla")

I prefer the first way but we should perhaps wait for the talk to choose no ?

I will implement the first way anyway as soon as I've time.

codecov[bot] commented 1 year ago

Codecov Report

Patch coverage: 96.66% and project coverage change: -0.25 :warning:

Comparison is base (84d4d57) 99.71% compared to head (dd8eef2) 99.47%.

Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #41 +/- ## ========================================== - Coverage 99.71% 99.47% -0.25% ========================================== Files 16 17 +1 Lines 352 379 +27 ========================================== + Hits 351 377 +26 - Misses 1 2 +1 ``` | [Impacted Files](https://app.codecov.io/gh/ddotta/parquetize/pull/41?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Damien+Dotta) | Coverage Δ | | |---|---|---| | [R/options.R](https://app.codecov.io/gh/ddotta/parquetize/pull/41?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Damien+Dotta#diff-Ui9vcHRpb25zLlI=) | `83.33% <83.33%> (ø)` | | | [R/dbi\_to\_parquet.R](https://app.codecov.io/gh/ddotta/parquetize/pull/41?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Damien+Dotta#diff-Ui9kYmlfdG9fcGFycXVldC5S) | `100.00% <100.00%> (ø)` | | | [R/write\_parquet\_at\_once.R](https://app.codecov.io/gh/ddotta/parquetize/pull/41?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Damien+Dotta#diff-Ui93cml0ZV9wYXJxdWV0X2F0X29uY2UuUg==) | `100.00% <100.00%> (ø)` | | | [R/write\_parquet\_by\_chunk.R](https://app.codecov.io/gh/ddotta/parquetize/pull/41?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Damien+Dotta#diff-Ui93cml0ZV9wYXJxdWV0X2J5X2NodW5rLlI=) | `100.00% <100.00%> (ø)` | |

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.