-
When running PageRank on a cluster, sometimes I hit a NoSuchElementException that's caused somewhere in VertexSetRDD. Full stack trace and command below. The line numbers may be slightly off due to de…
-
2020-04-30 20:37:04,644 [task-result-getter-0] WARN -[org.apache.spark.scheduler.TaskSetManager]-[WARN]org.apache.spark.internal.Logging$class.logWarning(Logging.scala:66)- Lost task 0.0 in stage 0.0…
-
**Describe the problem you faced**
I'm running an application that reads from 4 medium-sized (few hundred GB) Hudi MoR tables which are compacted weekly.
When running incremental queries to loo…
-
Hi,
I am having trouble creating the spark cluster with a custom spark version.
I am doing:
``` bash
ec2/spark-ec2 --key-pair= --identity-file= --region=eu-west-1 --zone=eu-west-1a --vpc-id= --subn…
-
I am told that if I can set fs.s3a.endpoint=s3.ca-central-1.amazonaws.com I can support the S3 V4 signature in the new regions so saveAsText file won't fail.
However, could someone please chime in…
-
I started instances successfully with the ./spark-ec2 script but at the end I don't get the information like http://master-hostname:8080 - I assume because I get `No valid Tachyon version found; Tachy…
ddsky updated
6 years ago
-
Done:
- Configure and install Kubernetes on an EC2 instance
- Create a K8s master and 1 node setup
- Submit a sample job
- Create YAML file to submit the spark job
- Run ocr_service.py on a pod,…
-
./spark-standalone/setup.sh: line 22: /root/spark/sbin/stop-all.sh: No such file or directory
./spark-standalone/setup.sh: line 27: /root/spark/sbin/start-master.sh: No such file or directory
./spar…
-
This would be very, very nice in 2016/2017. Or at least provide some instructions in README on how to do so.
-
Using spark 2.3, v0.0.51/metorikku-standalone.jar
/home/ec2-user/spark_home/bin/spark-submit --master local[*] --conf "spark.sql.parquet.writeLegacyFormat=true" --class com.yotpo.metorikku.Metorikk…