VeritoneAlpha / spark-job-rest

Other
33 stars 20 forks source link

Substitutions in job config cause peculiar behavior #19

Open Sitin opened 9 years ago

Sitin commented 9 years ago

If you start job with config that contains substitutions (like ${PWD} which is allowed by TypeSafe.Config) it will immediately cause job failure but REST endpoint will return successful submit status and job wan't be shown at jobs list.

2015-06-13 05:57:31 JobActor [INFO] Sending RunJob message to actor ActorSelection[Anchor(akka.tcp://S-test-context-1@127.0.0.1:11008/), Path(/user/A-test-context-1)]
2015-06-13 05:57:31 LoggingOutputStream$ [INFO] 260)
        at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
        at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
        at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

[E
2015-06-13 05:57:31 LoggingOutputStream$ [INFO] RROR] [06/13/2015 05:57:31.229] [ManagerSystem-akka.remote.default-remote-dispatcher-6] [akka.tcp://ManagerSystem@127.0.0.1:4042/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2FS-test-context-1%40127.0.0.1%3A11008-7/endpointWriter] Transient
2015-06-13 05:57:31 LoggingOutputStream$ [INFO]  association error (association remains live)
java.io.NotSerializableException: tried to serialize a value with unresolved substitutions, need to Config#resolve() first, see API docs
        at com.typesafe.config.impl.SerializedConfigValue.writeExternal(Serializ

We should either mark job as failed or (better) reject during submit.