hansva
commented
1 month ago Our workflows and pipelines are saved as an XML representation. We are slowly taking steps to make this more readable to humans (save them as json or yaml). But projects and the needed metadata to run them are perfectly possible under source control.
For the actual execution on Dataflow/Spark/Flink you also only need to upload our JAR once and can then execute pipelines using only the needed metadata.
Example using a Flex template on Dataflow https://hop.apache.org//manual/latest/pipeline/beam/dataflowPipeline/google-dataflow-pipeline.html
What would you like to happen?
From my investigations of Hop over Beam, it appears to be a No-Code Workflow / Pipeline builder that requires use of a GUI. Would be great to have a markup syntax that captures the steps that are generated as code - this would enable import/export, meta code generation and source control.
For reference, Apache Beam has a YAML API https://beam.apache.org/documentation/sdks/yaml/
Issue Priority
Priority: 2
Issue Component
Component: API