sematic-ai / sematic

An open-source ML pipeline development platform
Other
975 stars 59 forks source link

Runner doesn't take full advantage of parallelism #1120

Closed augray closed 7 months ago

augray commented 7 months ago

Sometimes when there are parallel branches in a Sematic run (particularly with deep nesting of the branches, it seems), Sematic doesn't always recognize that it can start on a run from one branch while it waits for run(s) in another branch to complete. We should fix this, to make the overall pipeline runtime lower in such cases.