search

tony-framework / TonY

TonY is a framework to natively run deep learning frameworks on Apache Hadoop.
https://tony-project.ai
Other
708 stars 164 forks source link

Make job fail when partial tasks' pre-dependent tasks finished and exceeds the waiting timeout #621

Closed zuston closed 2 years ago

zuston commented 2 years ago

This PR is to solve the problems of tf workers hang when chief has been finished and other bugs.

For example, tensorflow estimator training job will include some roles of ps/worker/evaluator/chief. Actually, due to the bug of tensorflow or misusing the estimator api, sometimes evaluator will hang. So if we use the configuration as follows, when evaluator is still running after timeout and chief and workers are all finished, the mechanism of dependency group timeout will make job failed.

Conf as follows, the evaluator will be alive 3600(sec) after workers and chief are all finished.

tony.application.group.A = worker,chief
tony.application.dependency.evaluator.timeout.after.A = 3600

So, this PR introduce the conf of tony.application.group.{GROUP_NAME}={JOB_TYPES} and tony.application.dependency.{JOB_TYPE}.timeout.after.{GROUP_NAME} = 3600

Besides, due to tensorflow bug, sometimes the workers will hang after the chief finished. We could use above conf to solve it. The worker will be alive 3600(sec) after chief finished.

tony.application.group.A = chief
tony.application.dependency.worker.timeout.after.A = 3600
zuston commented 2 years ago

Refer to #610

oliverhu commented 2 years ago

@zuston thanks.. can you update READ.me as well? also provide some examples and context in description?

zuston commented 2 years ago

@oliverhu Updated.

zuston commented 2 years ago

Updating configurations in wiki will be done after this PR merged @oliverhu