-
Does `sparklyr` need a function which is analogous to `doParallel::clusterEvalQ`?
I am trying to parallelize some database queries, where each worker would pull a single day's worth of data from th…
-
Hello,
Within a `html_notebook`, I am trying to use the `tidy=TRUE` option to retain blank spaces in the following code chunk,
``` r
{r load_packages, echo=TRUE, message=FALSE, warning=FALSE, tidy=…
-
### Feature description
Spark3 is currently in RC. Will there be support for Spark3 in the next release version (v8) or will we have to wait for v9 ?
more precisely, do you guys plan to start sup…
-
I would like to write a hive table (to allow other users read the table) from a sparkdataframe using sparklyr
I'm using an Amazon EMR in a project and I found that sparklyr works fine but I need w…
-
Problem:
When copying a data frame into spark, sdf_copy_to (or spark?) is treating characters or factors that use "NA" as NA. At least in R, "NA" is not the same as NA. Thus the copy in spark may …
-
I am writing Spark SQL by sparklyr in Rnotebook
```{r}
year = "2018-01-09"
````
```{sql connection=sc}
select * from tbl_name where partition = ${year}
```
it seems ${} language sugar doe…
-
I am working in Sparklyr 1.4.0 and trying to distribute a packed Conda R environment to all workers. Here is my config:
```
config
-
This functionality is not implemented in sparklyr yet
-
Hi all, when I try connecting to Spark (tried both local and yarn master):
```
library(sparklyr)
library(dplyr)
Sys.setenv(HADOOP_CONF_DIR = '/etc/hadoop/conf')
Sys.setenv(YARN_CONF_DIR = '/e…
-
Things to mention:
- ft_robust_scaler (https://github.com/sparklyr/sparklyr/pull/2254), courtesy of Maciej (zero323)
- bug fix related to `dplyr::inner_join` (https://github.com/sparklyr/sparklyr/…