search

allegroai / clearml

ClearML - Auto-Magical CI/CD to streamline your AI workload. Experiment Management, Data Management, Pipeline, Orchestration, Scheduling & Serving in one MLOps/LLMOps solution
https://clear.ml/docs
Apache License 2.0
5.69k stars 655 forks source link

Pipeline from Decorators: ROR TypeError :: UI fails to render pipeline steps #988

Closed grimmJ04 closed 1 year ago

grimmJ04 commented 1 year ago

Describe the bug

Pipeline graph disappears when executing a step that depends on the output of multiple other pipeline steps. Note that the pipeline still finishes as expected, without errors. The only problem is that the graph disappears.

To reproduce

Here is a small project to reproduce: pipeline-from-decorators.zip

I run the pipeline from the project root, setting the variable PYTHONPATH=./.

When the execution of the pipeline reaches step k_fold_model_trainer, the pipeline graph vanishes.

image

The error message copied from console:

ROR TypeError: Cannot read properties of undefined (reading 'filter') ``` at ngneat-dag.js:64:57 at Array.forEach () at ngneat-dag.js:63:35 at Array.forEach () at l (ngneat-dag.js:50:14) at ngneat-dag.js:70:13 at Array.forEach () at l (ngneat-dag.js:50:14) at ngneat-dag.js:70:13 at Array.forEach () Eh @ core.mjs:6485 handleError @ core.mjs:6532 next @ core.mjs:26027 next @ Subscriber.js:91 _next @ Subscriber.js:60 next @ Subscriber.js:31 (anonymous) @ Subject.js:34 s @ errorContext.js:19 next @ Subject.js:27 emit @ core.mjs:22423 (anonymous) @ core.mjs:25514 invoke @ zone.js:372 run @ zone.js:134 runOutsideAngular @ core.mjs:25387 onHandleError @ core.mjs:25514 handleError @ zone.js:376 runTask @ zone.js:181 invokeTask @ zone.js:487 invoke @ zone.js:476 m.args. @ zone.js:2541 setTimeout (async) _ @ zone.js:2543 scheduleTask @ zone.js:393 onScheduleTask @ zone.js:283 scheduleTask @ zone.js:386 scheduleTask @ zone.js:221 scheduleMacroTask @ zone.js:244 Me @ zone.js:679 (anonymous) @ zone.js:2585 r. @ zone.js:975 setTimeout @ timeoutProvider.js:7 s @ reportUnhandledError.js:4 $ @ Subscriber.js:158 next @ Subscriber.js:94 _next @ Subscriber.js:60 next @ Subscriber.js:31 (anonymous) @ distinctUntilChanged.js:14 _next @ OperatorSubscriber.js:13 next @ Subscriber.js:31 (anonymous) @ map.js:7 _next @ OperatorSubscriber.js:13 next @ Subscriber.js:31 (anonymous) @ map.js:7 _next @ OperatorSubscriber.js:13 next @ Subscriber.js:31 (anonymous) @ Subject.js:34 s @ errorContext.js:19 next @ Subject.js:27 next @ ReplaySubject.js:22 (anonymous) @ ngrx-store-devtools.mjs:808 next @ Subscriber.js:91 _next @ Subscriber.js:60 next @ Subscriber.js:31 (anonymous) @ scanInternals.js:14 _next @ OperatorSubscriber.js:13 next @ Subscriber.js:31 (anonymous) @ withLatestFrom.js:26 _next @ OperatorSubscriber.js:13 next @ Subscriber.js:31 (anonymous) @ observeOn.js:6 (anonymous) @ executeSchedule.js:3 _execute @ AsyncAction.js:53 execute @ QueueAction.js:20 flush @ AsyncScheduler.js:18 schedule @ QueueAction.js:14 schedule @ Scheduler.js:8 e @ executeSchedule.js:2 (anonymous) @ observeOn.js:6 _next @ OperatorSubscriber.js:13 next @ Subscriber.js:31 subscribe.we @ mergeInternals.js:25 _next @ OperatorSubscriber.js:13 next @ Subscriber.js:31 (anonymous) @ map.js:7 _next @ OperatorSubscriber.js:13 next @ Subscriber.js:31 subscribe.we @ mergeInternals.js:25 _next @ OperatorSubscriber.js:13 next @ Subscriber.js:31 (anonymous) @ filter.js:6 _next @ OperatorSubscriber.js:13 next @ Subscriber.js:31 (anonymous) @ Subject.js:34 s @ errorContext.js:19 next @ Subject.js:27 next @ BehaviorSubject.js:24 next @ ngrx-store.mjs:133 next @ ngrx-store.mjs:449 next @ Subscriber.js:91 _next @ Subscriber.js:60 next @ Subscriber.js:31 subscribe.we @ mergeInternals.js:25 _next @ OperatorSubscriber.js:13 next @ Subscriber.js:31 subscribe.we @ mergeInternals.js:25 _next @ OperatorSubscriber.js:13 next @ Subscriber.js:31 v @ Notification.js:68 (anonymous) @ dematerialize.js:6 _next @ OperatorSubscriber.js:13 next @ Subscriber.js:31 (anonymous) @ filter.js:6 _next @ OperatorSubscriber.js:13 next @ Subscriber.js:31 (anonymous) @ map.js:7 _next @ OperatorSubscriber.js:13 next @ Subscriber.js:31 _next @ Subscriber.js:60 next @ Subscriber.js:31 subscribe.we @ mergeInternals.js:25 _next @ OperatorSubscriber.js:13 next @ Subscriber.js:31 (anonymous) @ map.js:7 _next @ OperatorSubscriber.js:13 next @ Subscriber.js:31 (anonymous) @ materialize.js:7 _next @ OperatorSubscriber.js:13 next @ Subscriber.js:31 _next @ Subscriber.js:60 next @ Subscriber.js:31 subscribe.S @ switchMap.js:14 _next @ OperatorSubscriber.js:13 next @ Subscriber.js:31 _next @ Subscriber.js:60 next @ Subscriber.js:31 subscribe.we @ mergeInternals.js:25 _next @ OperatorSubscriber.js:13 next @ Subscriber.js:31 (anonymous) @ innerFrom.js:51 _trySubscribe @ Observable.js:37 (anonymous) @ Observable.js:31 s @ errorContext.js:19 subscribe @ Observable.js:22 me @ mergeInternals.js:19 Q @ mergeInternals.js:14 _next @ OperatorSubscriber.js:13 next @ Subscriber.js:31 (anonymous) @ withLatestFrom.js:26 _next @ OperatorSubscriber.js:13 next @ Subscriber.js:31 (anonymous) @ map.js:7 _next @ OperatorSubscriber.js:13 next @ Subscriber.js:31 (anonymous) @ map.js:7 _next @ OperatorSubscriber.js:13 next @ Subscriber.js:31 (anonymous) @ filter.js:6 _next @ OperatorSubscriber.js:13 next @ Subscriber.js:31 subscribe.we @ mergeInternals.js:25 _next @ OperatorSubscriber.js:13 next @ Subscriber.js:31 _next @ Subscriber.js:60 next @ Subscriber.js:31 Mt @ http.mjs:1818 invokeTask @ zone.js:406 onInvokeTask @ core.mjs:25475 invokeTask @ zone.js:405 runTask @ zone.js:178 invokeTask @ zone.js:487 S @ zone.js:1600 D @ zone.js:1637 load (async) B.p @ zone.js:1752 scheduleTask @ zone.js:393 onScheduleTask @ zone.js:283 scheduleTask @ zone.js:386 scheduleTask @ zone.js:221 scheduleEventTask @ zone.js:247 (anonymous) @ zone.js:1907 (anonymous) @ http.mjs:1904 _trySubscribe @ Observable.js:37 (anonymous) @ Observable.js:31 s @ errorContext.js:19 subscribe @ Observable.js:22 (anonymous) @ catchError.js:9 (anonymous) @ lift.js:10 (anonymous) @ Observable.js:26 s @ errorContext.js:19 subscribe @ Observable.js:22 me @ mergeInternals.js:19 Q @ mergeInternals.js:14 _next @ OperatorSubscriber.js:13 next @ Subscriber.js:31 (anonymous) @ innerFrom.js:51 _trySubscribe @ Observable.js:37 (anonymous) @ Observable.js:31 s @ errorContext.js:19 subscribe @ Observable.js:22 d @ mergeInternals.js:50 (anonymous) @ mergeMap.js:13 (anonymous) @ lift.js:10 (anonymous) @ Observable.js:26 s @ errorContext.js:19 subscribe @ Observable.js:22 (anonymous) @ filter.js:6 (anonymous) @ lift.js:10 (anonymous) @ Observable.js:26 s @ errorContext.js:19 subscribe @ Observable.js:22 (anonymous) @ map.js:6 (anonymous) @ lift.js:10 (anonymous) @ Observable.js:26 s @ errorContext.js:19 subscribe @ Observable.js:22 (anonymous) @ map.js:6 (anonymous) @ lift.js:10 (anonymous) @ Observable.js:26 s @ errorContext.js:19 subscribe @ Observable.js:22 (anonymous) @ withLatestFrom.js:23 (anonymous) @ lift.js:10 (anonymous) @ Observable.js:26 s @ errorContext.js:19 subscribe @ Observable.js:22 d @ mergeInternals.js:50 (anonymous) @ mergeMap.js:13 (anonymous) @ lift.js:10 (anonymous) @ Observable.js:26 s @ errorContext.js:19 subscribe @ Observable.js:22 (anonymous) @ catchError.js:9 (anonymous) @ lift.js:10 (anonymous) @ Observable.js:26 s @ errorContext.js:19 subscribe @ Observable.js:22 f.subscribe.v @ switchMap.js:14 _next @ OperatorSubscriber.js:13 next @ Subscriber.js:31 (anonymous) @ withLatestFrom.js:26 _next @ OperatorSubscriber.js:13 next @ Subscriber.js:31 subscribe.S @ switchMap.js:14 _next @ OperatorSubscriber.js:13 next @ Subscriber.js:31 (anonymous) @ map.js:7 _next @ OperatorSubscriber.js:13 next @ Subscriber.js:31 (anonymous) @ take.js:12 _next @ OperatorSubscriber.js:13 next @ Subscriber.js:31 (anonymous) @ filter.js:6 _next @ OperatorSubscriber.js:13 next @ Subscriber.js:31 (anonymous) @ distinctUntilChanged.js:14 _next @ OperatorSubscriber.js:13 next @ Subscriber.js:31 (anonymous) @ map.js:7 _next @ OperatorSubscriber.js:13 next @ Subscriber.js:31 (anonymous) @ map.js:7 _next @ OperatorSubscriber.js:13 next @ Subscriber.js:31 _subscribe @ ReplaySubject.js:31 _trySubscribe @ Observable.js:37 _trySubscribe @ Subject.js:74 (anonymous) @ Observable.js:31 s @ errorContext.js:19 subscribe @ Observable.js:22 _subscribe @ Observable.js:64 (anonymous) @ Observable.js:29 s @ errorContext.js:19 subscribe @ Observable.js:22 (anonymous) @ map.js:6 (anonymous) @ lift.js:10 (anonymous) @ Observable.js:26 s @ errorContext.js:19 subscribe @ Observable.js:22 _subscribe @ Observable.js:64 (anonymous) @ Observable.js:29 s @ errorContext.js:19 subscribe @ Observable.js:22 (anonymous) @ map.js:6 (anonymous) @ lift.js:10 (anonymous) @ Observable.js:26 s @ errorContext.js:19 subscribe @ Observable.js:22 (anonymous) @ distinctUntilChanged.js:9 (anonymous) @ lift.js:10 (anonymous) @ Observable.js:26 s @ errorContext.js:19 subscribe @ Observable.js:22 (anonymous) @ filter.js:6 (anonymous) @ lift.js:10 (anonymous) @ Observable.js:26 s @ errorContext.js:19 subscribe @ Observable.js:22 (anonymous) @ take.js:10 (anonymous) @ lift.js:10 (anonymous) @ Observable.js:26 s @ errorContext.js:19 subscribe @ Observable.js:22 (anonymous) @ map.js:6 (anonymous) @ lift.js:10 (anonymous) @ Observable.js:26 s @ errorContext.js:19 subscribe @ Observable.js:22 f.subscribe.v @ switchMap.js:14 _next @ OperatorSubscriber.js:13 next @ Subscriber.js:31 (anonymous) @ filter.js:6 _next @ OperatorSubscriber.js:13 next @ Subscriber.js:31 (anonymous) @ Subject.js:34 s @ errorContext.js:19 next @ Subject.js:27 (anonymous) @ ngrx-store-devtools.mjs:811 next @ Subscriber.js:91 _next @ Subscriber.js:60 next @ Subscriber.js:31 (anonymous) @ scanInternals.js:14 _next @ OperatorSubscriber.js:13 next @ Subscriber.js:31 (anonymous) @ withLatestFrom.js:26 _next @ OperatorSubscriber.js:13 next @ Subscriber.js:31 (anonymous) @ observeOn.js:6 (anonymous) @ executeSchedule.js:3 _execute @ AsyncAction.js:53 execute @ QueueAction.js:20 flush @ AsyncScheduler.js:18 schedule @ QueueAction.js:14 schedule @ Scheduler.js:8 e @ executeSchedule.js:2 (anonymous) @ observeOn.js:6 _next @ OperatorSubscriber.js:13 next @ Subscriber.js:31 subscribe.we @ mergeInternals.js:25 _next @ OperatorSubscriber.js:13 next @ Subscriber.js:31 (anonymous) @ map.js:7 _next @ OperatorSubscriber.js:13 next @ Subscriber.js:31 subscribe.we @ mergeInternals.js:25 _next @ OperatorSubscriber.js:13 next @ Subscriber.js:31 (anonymous) @ filter.js:6 _next @ OperatorSubscriber.js:13 next @ Subscriber.js:31 (anonymous) @ Subject.js:34 s @ errorContext.js:19 next @ Subject.js:27 next @ BehaviorSubject.js:24 next @ ngrx-store.mjs:133 next @ ngrx-store.mjs:449 next @ Subscriber.js:91 _next @ Subscriber.js:60 next @ Subscriber.js:31 subscribe.we @ mergeInternals.js:25 _next @ OperatorSubscriber.js:13 next @ Subscriber.js:31 subscribe.we @ mergeInternals.js:25 _next @ OperatorSubscriber.js:13 next @ Subscriber.js:31 v @ Notification.js:68 (anonymous) @ dematerialize.js:6 _next @ OperatorSubscriber.js:13 next @ Subscriber.js:31 (anonymous) @ filter.js:6 _next @ OperatorSubscriber.js:13 next @ Subscriber.js:31 (anonymous) @ map.js:7 _next @ OperatorSubscriber.js:13 load (async) B.p @ zone.js:1752 scheduleTask @ zone.js:393 onScheduleTask @ zone.js:283 scheduleTask @ zone.js:386 scheduleTask @ zone.js:221 scheduleEventTask @ zone.js:247 (anonymous) @ zone.js:1907 (anonymous) @ http.mjs:1904 _trySubscribe @ Observable.js:37 (anonymous) @ Observable.js:31 s @ errorContext.js:19 subscribe @ Observable.js:22 (anonymous) @ catchError.js:9 (anonymous) @ lift.js:10 (anonymous) @ Observable.js:26 s @ errorContext.js:19 subscribe @ Observable.js:22 me @ mergeInternals.js:19 Q @ mergeInternals.js:14 _next @ OperatorSubscriber.js:13 next @ Subscriber.js:31 (anonymous) @ innerFrom.js:51 _trySubscribe @ Observable.js:37 (anonymous) @ Observable.js:31 s @ errorContext.js:19 subscribe @ Observable.js:22 d @ mergeInternals.js:50 (anonymous) @ mergeMap.js:13 (anonymous) @ lift.js:10 (anonymous) @ Observable.js:26 s @ errorContext.js:19 subscribe @ Observable.js:22 (anonymous) @ filter.js:6 (anonymous) @ lift.js:10 (anonymous) @ Observable.js:26 s @ errorContext.js:19 subscribe @ Observable.js:22 (anonymous) @ map.js:6 (anonymous) @ lift.js:10 (anonymous) @ Observable.js:26 s @ errorContext.js:19 subscribe @ Observable.js:22 (anonymous) @ map.js:6 (anonymous) @ lift.js:10 (anonymous) @ Observable.js:26 s @ errorContext.js:19 subscribe @ Observable.js:22 f.subscribe.v @ switchMap.js:14 _next @ OperatorSubscriber.js:13 next @ Subscriber.js:31 (anonymous) @ withLatestFrom.js:26 _next @ OperatorSubscriber.js:13 next @ Subscriber.js:31 (anonymous) @ innerFrom.js:51 _trySubscribe @ Observable.js:37 (anonymous) @ Observable.js:31 s @ errorContext.js:19 subscribe @ Observable.js:22 (anonymous) @ withLatestFrom.js:23 (anonymous) @ lift.js:10 (anonymous) @ Observable.js:26 s @ errorContext.js:19 subscribe @ Observable.js:22 (anonymous) @ switchMap.js:10 (anonymous) @ lift.js:10 (anonymous) @ Observable.js:26 s @ errorContext.js:19 subscribe @ Observable.js:22 d @ mergeInternals.js:50 (anonymous) @ mergeMap.js:13 (anonymous) @ lift.js:10 (anonymous) @ Observable.js:26 s @ errorContext.js:19 subscribe @ Observable.js:22 (anonymous) @ catchError.js:9 (anonymous) @ lift.js:10 (anonymous) @ Observable.js:26 s @ errorContext.js:19 subscribe @ Observable.js:22 f.subscribe.v @ switchMap.js:14 _next @ OperatorSubscriber.js:13 next @ Subscriber.js:31 (anonymous) @ withLatestFrom.js:26 _next @ OperatorSubscriber.js:13 next @ Subscriber.js:31 (anonymous) @ filter.js:6 _next @ OperatorSubscriber.js:13 next @ Subscriber.js:31 (anonymous) @ Subject.js:34 s @ errorContext.js:19 next @ Subject.js:27 (anonymous) @ ngrx-store-devtools.mjs:811 next @ Subscriber.js:91 _next @ Subscriber.js:60 next @ Subscriber.js:31 (anonymous) @ scanInternals.js:14 _next @ OperatorSubscriber.js:13 next @ Subscriber.js:31 (anonymous) @ withLatestFrom.js:26 _next @ OperatorSubscriber.js:13 next @ Subscriber.js:31 (anonymous) @ observeOn.js:6 (anonymous) @ executeSchedule.js:3 _execute @ AsyncAction.js:53 execute @ QueueAction.js:20 flush @ AsyncScheduler.js:18 schedule @ QueueAction.js:14 schedule @ Scheduler.js:8 e @ executeSchedule.js:2 (anonymous) @ observeOn.js:6 _next @ OperatorSubscriber.js:13 next @ Subscriber.js:31 subscribe.we @ mergeInternals.js:25 _next @ OperatorSubscriber.js:13 next @ Subscriber.js:31 (anonymous) @ map.js:7 _next @ OperatorSubscriber.js:13 next @ Subscriber.js:31 subscribe.we @ mergeInternals.js:25 _next @ OperatorSubscriber.js:13 next @ Subscriber.js:31 (anonymous) @ filter.js:6 _next @ OperatorSubscriber.js:13 next @ Subscriber.js:31 (anonymous) @ Subject.js:34 s @ errorContext.js:19 next @ Subject.js:27 next @ BehaviorSubject.js:24 next @ ngrx-store.mjs:133 dispatch @ ngrx-store.mjs:446 refreshList @ experiments.component.ts:603 (anonymous) @ base-entity-page.ts:127 next @ Subscriber.js:91 _next @ Subscriber.js:60 next @ Subscriber.js:31 (anonymous) @ map.js:7 _next @ OperatorSubscriber.js:13 next @ Subscriber.js:31 (anonymous) @ filter.js:6 _next @ OperatorSubscriber.js:13 next @ Subscriber.js:31 (anonymous) @ withLatestFrom.js:26 _next @ OperatorSubscriber.js:13 next @ Subscriber.js:31 (anonymous) @ Subject.js:34 s @ errorContext.js:19 next @ Subject.js:27 (anonymous) @ refresh.service.ts:27 next @ Subscriber.js:91 _next @ Subscriber.js:60 next @ Subscriber.js:31 (anonymous) @ filter.js:6 _next @ OperatorSubscriber.js:13 next @ Subscriber.js:31 (anonymous) @ withLatestFrom.js:26 _next @ OperatorSubscriber.js:13 next @ Subscriber.js:31 (anonymous) @ timer.js:23 _execute @ AsyncAction.js:53 execute @ AsyncAction.js:41 flush @ AsyncScheduler.js:18 S. @ zone.js:2561 invokeTask @ zone.js:406 onInvokeTask @ core.mjs:25475 invokeTask @ zone.js:405 runTask @ zone.js:178 invokeTask @ zone.js:487 invoke @ zone.js:476 m.args. @ zone.js:2541 ```

Expected behaviour

The pipeline steps should render correctly, like seen in the documentation, or in the image below, before the k_fold_model_trainer step.

image

Environment

A similar problem can be found here https://github.com/allegroai/clearml/issues/618, however no matter how I try to organize my imports, the problem still persists.

ainoam commented 1 year ago

Thanks for reporting @grimmJ04.

Do note you're server version is almost a year old... does the problem persist on an up-to date server?

grimmJ04 commented 1 year ago

No, the issue seems to be gone on version 1.10.0-357.