[Feature Request][Go SDK]: Rewrite the `dot` runner in terms of a Portable pipeline.

lostluck commented 1 year ago

What would you like to happen?

The Go SDK's dot runner current generates a pipeline graph based on the Go SDK internal structures.

https://github.com/apache/beam/blob/master/sdks/go/pkg/beam/core/util/dot/dot.go#L89

If it were rewritten in terms of the portable pipeline proto (https://github.com/apache/beam/blob/master/model/pipeline/src/main/proto/org/apache/beam/model/pipeline/v1/beam_runner_api.proto#L91) then it could be used to render Cross Language pipelines and be used as a basis for the stand alone Prism Runner's web UI.

That would sufficiently allow #25896 to be closed.

Issue Priority

Priority: 3 (nice-to-have improvement)

Issue Components

[ ] Component: Python SDK
[ ] Component: Java SDK
[X] Component: Go SDK
[ ] Component: Typescript SDK
[ ] Component: IO connector
[ ] Component: Beam examples
[ ] Component: Beam playground
[ ] Component: Beam katas
[ ] Component: Website
[ ] Component: Spark Runner
[ ] Component: Flink Runner
[ ] Component: Samza Runner
[ ] Component: Twister2 Runner
[ ] Component: Hazelcast Jet Runner
[ ] Component: Google Cloud Dataflow Runner

mls3odp commented 1 month ago

.take-issue

mls3odp commented 4 weeks ago

Does this mean to use Pipeline proto as the input of Execute function in dot.go?

Or modify the Render function to transforming edges and nodes into the Pipeline proto and writing the corresponding information (e.g. Pipeline::Components containing transforms and pcolllections) as texts? CMIIW thanks!

lostluck commented 4 weeks ago

In this case it means rendering the pipeline structure from the Pipeline Proto instead of from the raw Go structures.

This would let it take any pipeline proto graph and produce a dot graph for it.

So this means building the pipeline proto with the existing code to do so (see how the Universal runner is doing it) and then building the visual graph based on the information only from the proto.

mls3odp commented 4 weeks ago

Gotcha! So we need to convert beam.Pipeline to Pipeline proto like https://github.com/apache/beam/blob/master/sdks/go/pkg/beam/runners/universal/universal.go#L64-L91 in https://github.com/apache/beam/blob/master/sdks/go/pkg/beam/runners/dot/dot.go#L40.

apache / beam