-
[![mitm_build](https://user-images.githubusercontent.com/1323708/59226671-90645200-8ba1-11e9-8ab3-39292bef99e9.jpeg)](https://medium.com/@jonathan.leitschuh/want-to-take-over-the-java-ecosystem-all-yo…
-
1. registered AWS account and created a user and generated access_id and secret_key and chose a region for "aws configure" command, I passed ec2 instance installation. It took time to get everything i…
-
I'm trying to do hyper-parameter tuning in scala, using GridCV. However I create my pipeline and everything, I fit my dataset to the pipeline, it fits properly.
Then I add some `paramGrid` and I go…
-
When I read a parquet file on s3 from sparklyr context like this:
spark_read_parquet(sc, name = "parquet_test", path = "s3a://"), it throws me an error which is:
Caused by: java.io.IOException: Cou…
-
Got error ImportError: No module named 'pyspark' when running python ch02/pyspark_mongodb.py
The other examples in chapter 2 ran fine.
-
I am trying to run the GATK4 workflow on Cromwell using the Spark backend. The WDL is from the Broad's GitHub repo:
https://github.com/broadinstitute/gatk4-data-processing/blob/master/processing-for…
-
The code is throwing java.lang.IllegalArgumentException. Could you please shed some light on the workaround.
Below is the **usage**
val df = spark.read.
format("com.springml.spark.s…
-
I'm running XGBoost on an AWS EC2 instance with 8 cores and 32g memory. My dataset is around 30g distributed into 60 .csv files. I wrote my codes in Scala and used distributed XGBoost with Spark to do…
-
I've created a cluster using the AWS Console but now would like to set things up to launch it using CLI. When I attempt to take the code generated from the CLI Exporter (see below), receive the follow…
-
I have a RDD KV pair need to insert into a redis server
my code is like
```
//init pair rdd
sql.sparkSession.sparkContext.toRedisKV(rdd)
//other stuff...using jedis but not spark-redis
```
then…