search

cloudera-labs / envelope

Build configuration-driven ETL pipelines on Apache Spark
Apache License 2.0
158 stars 89 forks source link

Envelope stuck waiting for dependencies that do not exists #45

Open siddharthniruban opened 4 years ago

siddharthniruban commented 4 years ago

I was running Spark Streaming job in 'yarn client mode'. Job was running fine for sometime and after that job got failed because of Envelope pipeline stuck due to steps waiting for dependencies that do not exists.

jeremybeard commented 4 years ago

First thing is to make sure you are using the latest release 0.7.2 which solves some bugs with that error. If you are getting the error on 0.7.2 then likely you have a step that specifies one or more dependencies that are not in the pipeline.