Open dos65 opened 6 years ago
Hi @dos65 i was trying out this patch, but on starting job with required context, its failing with
ERROR 2020-05-09T11:59:33.324 [d5400064-2e35-413b-8b17-32261d930310] FailedEvent with Error:
java.lang.RuntimeException: Context is broken
at io.hydrosphere.mist.master.execution.JobActor$$anonfun$io$hydrosphere$mist$master$execution$JobActor$$initial$1.applyOrElse(JobActor.scala:59)
at akka.actor.Actor$class.aroundReceive(Actor.scala:517)
at io.hydrosphere.mist.master.execution.JobActor.akka$actor$Timers$$super$aroundReceive(JobActor.scala:24)
at akka.actor.Timers$class.aroundReceive(Timers.scala:44)
at io.hydrosphere.mist.master.execution.JobActor.aroundReceive(JobActor.scala:24)
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:527)
at akka.actor.ActorCell.invoke(ActorCell.scala:496)
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257)
at akka.dispatch.Mailbox.run(Mailbox.scala:224)
at akka.dispatch.Mailbox.exec(Mailbox.scala:234)
at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Caused by: java.lang.RuntimeException: Unknown settings name default_emr for ctx root_emr_ctx
at io.hydrosphere.mist.master.execution.ClustersService$$anon$1.start(ClustersService.scala:46)
at io.hydrosphere.mist.master.execution.ExecutionService$$anonfun$2$$anonfun$3.apply(ExecutionService.scala:119)
at io.hydrosphere.mist.master.execution.ExecutionService$$anonfun$2$$anonfun$3.apply(ExecutionService.scala:119)
at io.hydrosphere.mist.master.execution.ContextFrontend.io$hydrosphere$mist$master$execution$ContextFrontend$$gotoStartingConnector(ContextFrontend.scala:345)
at io.hydrosphere.mist.master.execution.ContextFrontend$$anonfun$awaitRequest$1.applyOrElse(ContextFrontend.scala:70)
at akka.actor.Actor$class.aroundReceive(Actor.scala:517)
at io.hydrosphere.mist.master.execution.ContextFrontend.akka$actor$Timers$$super$aroundReceive(ContextFrontend.scala:39)
at akka.actor.Timers$class.aroundReceive(Timers.scala:44)
at io.hydrosphere.mist.master.execution.ContextFrontend.aroundReceive(ContextFrontend.scala:39)
... 9 more
I have kept the conf folder same as present in the release, just modified the context to pick up emr_ctx
context instead of default.
Any pointers?
Could you please show your context.conf here? I think there might be some issues with that.
Hi @blvp Thanks for reply.
I am using following config:
model = Context
name = emr_ctx
data {
launchData {
type = "aws-emr",
launcherSettingsName = "default_emr"
releaseLabel = "emr-5.17.0"
instances = [
{
instanceGroupType = "master"
instanceType = "m4.large"
name = "my-master"
instanceCount = 1
market = "onDemand"
},
{
instanceGroupType = "core"
instanceType = "m4.large"
name = "my-core"
instanceCount = 2
market = "spot"
bidPrice = "0.035"
}
]
}
maxJobs = 1
maxConnFailures = 1
workerMode = "exclusive"
precreated = false
sparkConf {
spark.executor.instances = 2
spark.submit.deployMode = "cluster"
spark.master = "yarn"
spark.executor.memory = "1G"
spark.executor.cores = 2
}
downtime = "1200s"
streamingDuration = "1s"
}
and using 20func.conf as follows:
model = Function
name = hello-mist-scala
data {
path = hello-mist-scala_0.0.1.jar
class-name = "HelloMist$"
# context = default
context = emr_ctx
# context = emr_autoscale_ctx
}
This is creating the function correctly and picking up the required context as well but fails on triggering the function with above error.
Thank you. I will keep you in the loop. Also, could you please join gitter, so we communicate there istead of PR :)
Yes @blvp! Thanks
Hey @blvp @dos65 Did you guys get a chance to look at this error?
This is working implementation of built-in integration with EMR. Cluster configuration may be placed into context settings(examples) and mist can dynamically spawn EMR-clusters.
This requires introduces new mini-submodules:
agent
- simple application for running on EMR master node to guarantee shutdown cluster in case if mist instance was stoppedaws-init-setup
- performs initial AWS environment setup(creates if necessary roles, security groups, ssh keys) + generates mist config with this informationThis branch is ready for preview and already published here + cloudformation template.
What's left:
launcher-settings
section and allow contexts to use themmanual-launcher
- the similar like we have for workers