Closed zero88 closed 5 years ago
Thank for reporting!
Could you check, if there is any additional information in logs
directory? There should be a log file that starts from local-worker
.
I added more in description.
Btw, worker
is using which log4j.properties
? I would like to redirect all to console
log
Could you also check if mist-worker.jar
exists in $MIST_HOME directory on the node where Mist was run?
I've just tested your setup on my machine and it works.
Actually, logs in local-worker
file are stdout from spark-submit
that is used to submit mist-worker
.
This is command run in worker
has mist-worker.jar
. It is tricky because I copy manually from Mist
to worker
.
mist_1 | 2018-11-09 13:26:42,772 INFO io.hydrosphere.mist.master.execution.workers.starter.LocalSparkSubmit Try submit local worker k8s-master_e0642e8b-a3af-41ee-bc1b-faafb363341e_1, cmd: /opt/spark/bin/spark-submit --conf spark.eventLog.enabled=true --conf spark.shuffle.service.enabled=true --conf spark.submit.deployMode=cluster --conf spark.master=spark://spark-master:7077 --conf spark.eventLog.dir=/data/spark/events --conf spark.dynamicAllocation.enabled=true --conf spark.eventLog.compress=true --class io.hydrosphere.mist.worker.Worker /opt/mist/mist-worker.jar --master 172.22.0.4:2551 --name k8s-master_e0642e8b-a3af-41ee-bc1b-faafb363341e_1
spark-master_1 | 2018-11-09 13:26:45,455 INFO org.apache.spark.deploy.master.Master Driver submitted org.apache.spark.deploy.worker.DriverWrapper
spark-master_1 | 2018-11-09 13:26:45,458 INFO org.apache.spark.deploy.master.Master Launching driver driver-20181109132645-0000 on worker worker-20181109120654-c6e613739ff2-9099
spark-worker_1 | 2018-11-09 13:26:45,509 INFO org.apache.spark.deploy.worker.Worker Asked to launch driver driver-20181109132645-0000
spark-worker_1 | 2018-11-09 13:26:45,529 INFO org.apache.spark.deploy.worker.DriverRunner Copying user jar file:/opt/mist/mist-worker.jar to /opt/spark/work/driver-20181109132645-0000/mist-worker.jar
spark-worker_1 | 2018-11-09 13:26:45,570 INFO org.apache.spark.util.Utils Copying /opt/mist/mist-worker.jar to /opt/spark/work/driver-20181109132645-0000/mist-worker.jar
spark-worker_1 | 2018-11-09 13:26:45,813 INFO org.apache.spark.deploy.worker.DriverRunner Launch Command: "/usr/lib/jvm/java-1.8-openjdk/bin/java" "-cp" "/opt/spark/conf/:/opt/spark/jars/*" "-Xmx1024M" "-Dspark.app.name=io.hydrosphere.mist.worker.Worker" "-Dspark.eventLog.enabled=true" "-Dspark.driver.supervise=false" "-Dspark.shuffle.service.enabled=true" "-Dspark.submit.deployMode=cluster" "-Dspark.jars=file:/opt/mist/mist-worker.jar" "-Dspark.master=spark://spark-master:7077" "-Dspark.eventLog.dir=/data/spark/events" "-Dspark.eventLog.compress=true" "-Dspark.dynamicAllocation.enabled=true" "-Dspark.rpc.askTimeout=10s" "org.apache.spark.deploy.worker.DriverWrapper" "spark://Worker@c6e613739ff2:9099" "/opt/spark/work/driver-20181109132645-0000/mist-worker.jar" "io.hydrosphere.mist.worker.Worker" "--master" "172.22.0.4:2551" "--name" "k8s-master_e0642e8b-a3af-41ee-bc1b-faafb363341e_1"
Just remind, worker
in my failed case has nothing related to Mist
, just plain spark
, no mist-worker.jar
. Does mist
support this case?
The strange thing is worker
is used 2 jars with different locations but same content:
-Dspark.jars=file:/opt/mist/mist-worker.jar
and /opt/spark/work/driver-20181109132645-0000/mist-worker.jar
Actually, logs in local-worker file are stdout from spark-submit that is used to submit mist-worker.
We have 2 logs: one is job-[which id?]
and loca-worker-[which id?]
, these ids doesn't match each other. Is is correct?
Back to my question:
If it is from spark-submit
but I didn't set any configuration and it should be in console
as Spark
default behavior. Any configuration to redirect log to console
instead of file
Finally, I found out what's going on. I expected that in standalone mode, even for cluster mode, it is enough to provide a local path to mist-worker.jar
for spark-submit
.
This problem is fixed in #529
About logs. These files job-$id
and local-worker-$id
are creating by mist. Here local-worker
is stdout from a process that mist runs when it runs a new worker. job-$id
are logs that mist collets from a spark-driver application during job invocation. Both of them are used for debugging purposes.
Here you may found more information: glossary and invocation
I've just merged #529 into master. @zero-88, it would be great if you could build mist from the master branch and get feedback if it solved your problem or not. Feel free to reopen this issue in case if it won't help.
Thank @dos65 I will try today and let you know. Do you intend to release hotfix version?
I will try today and let you know. Do you intend to release hotfix version?
Great! If it works, I'll publish 1.1.1
Awesome, works like a charm. Please make a new release version when you have a time. Thanks
@zero-88 Done!
Context
Spark Master
(withoutmist
) +Worker
(withoutmist
) +Mist Master
2.4.0
1.1.0
Log
Job log
Local worker log
Suspicious Seem hard code when using
$MIST_HOME
for folder path tomist_worker.jar
on sparkworker