Add dummy backend

[6543]

create a pipeline backend to mock exec for integration testing

TODO:

• [x] document mock backend in development docs -> https://woodpecker-ci-woodpecker-pr-3820.surge.sh/docs/next/development/testing

close #3503

[6543]

mock found already first error :dagger:

build and use cli exec with this config:

steps:
  - name: echo
    image: dummy
    commands: echo ja

we get:

8:38PM DBG cli/common/hooks.go:38 > Checking for updates ...
8:38PM DBG cli/update/updater.go:25 > Current version: next-1f21033432
8:38PM DBG cli/update/updater.go:28 > Skipping update check for development & next versions
8:38PM DBG cli/common/hooks.go:49 > No update required
8:38PM FTL cmd/cli/main.go:27 > error running cli error="[bad_habit] Please set an event filter for all steps or the whole workflow on all items of the when block"

I expect the cli to not fail just lock the error - as error type is bad_habit

[6543]

if we would have this framework merged ... the bugfix for #3821 could add an regression test for this exact case ...

[woodpecker-bot]

Deployment of preview was torn down

[6543]

@anbraten with this we could in the long term also catch ui glitches like: https://cloud.obermui.de/s/gcZfzD7SDLxBs6i/download/vid-20240622-220313.mkv

(in the main branch, we don't get notivied via eventstream when services are stoped ...)

[6543]

[codecov[bot]]

Codecov Report

Attention: Patch coverage is 73.38129% with 37 lines in your changes missing coverage. Please review.

Project coverage is 28.35%. Comparing base (b8b6efb) to head (bc821eb). Report is 6 commits behind head on main.

:exclamation: Current head bc821eb differs from pull request most recent head 98ae630

Please upload reports for the commit 98ae630 to get more accurate results.

Files	Patch %	Lines
pipeline/backend/dummy/dummy.go	74.45%	27 Missing and 8 partials :warning:
cli/exec/exec.go	0.00%	1 Missing :warning:
cmd/agent/main.go	0.00%	1 Missing :warning:

Additional details and impacted files

```diff @@ Coverage Diff @@ ## main #3820 +/- ## ========================================== + Coverage 28.25% 28.35% +0.10% ========================================== Files 363 364 +1 Lines 25114 25213 +99 ========================================== + Hits 7095 7149 +54 - Misses 17486 17531 +45 Partials 533 533 ```

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

[qwerty287]

Adding testing stuff into production binaries isn't a good idea I'd say.

Would it be possible to use a go build flag to define whether you want to have this included?

[6543]

Well I personally do see value in have it shipped in the normal cli and agent binarys ... but as of now if i realy would need it i can compile stuff myselfe.

will do ...

[qwerty287]

How you implemented it now it's kinda useless. Then you can also directly add the dummy backend. We don't need a dummy for the dummy 😄

Wouldn't it be possible to just not add it to the backend list if the build option is not set?

[6543]

@qwerty287 I dont get the problem ?!?

yes this does not help us right now at all but I have to do the init groundwork at some point

[qwerty287]

I don't want to have a dummy_noop backend. I'd like to completely remove this from the final binary.

The dummy_noop is just useless - it's a dummy of the dummy. If there's no easy way to completely exclude it, rather add dummy to the final binary. I can also check later how it can be excluded.

[6543]

ah that's what you want ... well it's posible but we have to alter how we register backend engines into cli and agent ...

[6543]

moved unrelated changes int new pull: https://github.com/woodpecker-ci/woodpecker/pull/3831

[anbraten]

Started using the dummy backend for some unit (almost integration) testing:

Works quite nice already. Would like to set all behavior options using env vars instead of using the name. Some additional options I tried:

• FAIL_START_STEP=true, FAIL_TAIL_STEP=true, etc (so we can let any of the backend functions fail)
• STEP_EXIT_CODE=123 So we can pass a special exit code
• STEP_OOM_KILLED=true to say one step failed due to an oom error

The runner_test.go I have used:

package agent

import (
    "context"
    "fmt"
    "testing"
    "time"

    "github.com/stretchr/testify/assert"
    "github.com/stretchr/testify/mock"

    "go.woodpecker-ci.org/woodpecker/v2/pipeline/backend/dummy"
    "go.woodpecker-ci.org/woodpecker/v2/pipeline/backend/types"
    "go.woodpecker-ci.org/woodpecker/v2/pipeline/rpc"
    "go.woodpecker-ci.org/woodpecker/v2/pipeline/rpc/mocks"
)

type peery struct {
    *mocks.Peer
}

func (p *peery) Done(ctx context.Context, id string, state rpc.WorkflowState) error {
    return nil
}

func TestRunnerCanceledState(t *testing.T) {
    backend := dummy.New()
    _peer := mocks.NewPeer(t)

    peer := &peery{_peer}

    hostname := "dummy"
    filter := rpc.Filter{
        Labels: map[string]string{
            "hostname": hostname,
            "platform": "test",
            "backend":  backend.Name(),
            "repo":     "*", // allow all repos by default
        },
    }
    state := &State{
        Metadata: map[string]Info{},
        Polling:  1, // max workflows to poll
        Running:  0,
    }
    r := NewRunner(peer, filter, hostname, state, &backend)
    ctx, cancel := context.WithCancel(context.Background())

    workflow := &rpc.Workflow{
        ID: "1",
        Config: &types.Config{
            Stages: []*types.Stage{
                {
                    Steps: []*types.Step{
                        {

                            Name: "test",
                            Environment: map[string]string{
                                "SLEEP": "10s",
                            },
                            Commands: []string{
                                "echo 'hello world'",
                            },
                            OnSuccess: true,
                        },
                    },
                },
            },
        },
        Timeout: 1, // 1 minute
    }

    peer.On("Next", mock.Anything, filter).Return(workflow, nil).Once()
    peer.On("Init", mock.Anything, "1", mock.MatchedBy(func(state rpc.WorkflowState) bool {
        return state.Started != 0 && state.Finished == 0 && state.Error == ""
    })).Return(nil)
    peer.On("Done", mock.Anything, "1", mock.MatchedBy(func(state rpc.WorkflowState) bool {
        return state.Started != 0 && state.Finished != 0 && state.Error == ""
    })).Return(nil)
    peer.On("Log", mock.Anything, mock.Anything).Return(nil)
    peer.On("Wait", mock.Anything, "1").Return(nil)
    peer.On("Update", mock.Anything, "1", mock.Anything).Return(nil)
    peer.On("Extend", mock.Anything, "1").Return(nil).Maybe()

    go func() {
        time.Sleep(100 * time.Millisecond)
        fmt.Println("canceling ...")
        cancel()
    }()

    err := r.Run(ctx)
    assert.NoError(t, err)
}

[6543]

Started using the dummy backend for some unit (almost integration) testing:

jep not for this pull :)