search

woodpecker-ci / woodpecker

Woodpecker is a simple, yet powerful CI/CD engine with great extensibility.
https://woodpecker-ci.org
Apache License 2.0
4.26k stars 369 forks source link

Conversion extension or pipeline modification "on the fly"? #4341

Open nmapx opened 4 hours ago

nmapx commented 4 hours ago

Clear and concise description of the problem

I'm currently moving from Drone to Woodpecker and while discovering differences I've noticed there is no such thing as conversion extension here. I guess it was introduced into Drone when Woodpecker was already there but I'm not sure. Maybe you could clarify more, or if it was removed why so? In my opinion it's a very useful and specific feature that allows more than just static pipelines. I'm talking here about "paved roads" / "golden paths" approach which allows creating complex yet very easy to maintain pipelines.

Suggested solution

Well, maybe it doesn't need to be conversion extension thing just like it is in Drone but something that would allow modifying pipeline syntax on the fly. This way we could easily introduce support for other languages (Starlark & Jsonnet) - it would eventually be converted into Yaml using this approach.

Alternative

No response

Additional context

No response

Validations

zc-devs commented 4 hours ago

https://woodpecker-ci.org/docs/administration/advanced/external-configuration-api https://github.com/woodpecker-ci/woodpecker/pull/1396 https://github.com/woodpecker-ci/woodpecker/discussions/3277 https://github.com/woodpecker-ci/woodpecker/discussions/2254

nmapx commented 3 hours ago

@zc-devs Thanks for the links, it clarified a lot! External configuration API seems to fit the problem, but only partially. If I get it correctly it's doing some pre-processing but still sending yaml syntax to the external API. There is no way to initialize the repository with pipeline in different syntax than yaml. I get the point that Woodpecker works on yaml files at the end, but for the pre-processing (in order to make sense of it) the source workflow should be in different lang like jsonnet or starlark (second in my case but it doesn't matter). There is Jsonnet addon mentioned in one of the issues but I wouldn't consider it stable - it seems it doesn't work for private repositories (which is usually the case I think) and UX of it is just... "not good enough"?

zc-devs commented 3 hours ago

Before the run or restart of any pipeline Woodpecker will make a POST request to an external HTTP API sending the current repository, build information and all current config files retrieved from the repository

still sending yaml syntax to the external API. There is no way to initialize the repository with pipeline in different syntax than yaml

I'm not a ~real welder~ user of External API :) If I remember correctly, WP indeed looks up only for yaml and yml config files in a repository. But I don't know whether it reads/parses them or not. You can try to save jsonnet content in a file with yaml extension 😆 I hope WP will send them as is to the external service. Then there you could process it, finally convert into WP YAML syntax and send back.