Summary:
In the case when job failed and ReadFromAggregators was used, there is a chance that executors will try to read result before onStageCompleted function is called. In this case, reading task will read from the old table and get Table has reached its quota of 1 reader(s) error .
I was not able to reproduce this scenario neither locally neither on real spark cluster. Changed table name to be unique for each stage attempt. As a result of this change, this error shoudn't appear in any case. Design doc/spec:
Docs impact: none
Preliminary Reviewer(s):
Final Reviewer:
Test Plan:
Summary: In the case when job failed and
ReadFromAggregatorswas used, there is a chance that executors will try to read result before onStageCompleted function is called. In this case, reading task will read from the old table and getTable has reached its quota of 1 reader(s)error . I was not able to reproduce this scenario neither locally neither on real spark cluster. Changed table name to be unique for each stage attempt. As a result of this change, this error shoudn't appear in any case. Design doc/spec: Docs impact: none Preliminary Reviewer(s): Final Reviewer: Test Plan:Reviewers: pmishchenko-ua, adam CC:
Jira issues: PLAT-6477