End-to-end test combining: plugin actions + template + scaffolder project and/or e2e golden path

[cmoulliard]

TODO

Explore if we could automate some scenario like

• e2e test combining: plugin actions + template + scaffolded project
• e2e test of a Quarkus scaffolded application (example TODO app) and its golden path: ArgoCD => Tekton pipelines => ... => Application runs

[cmoulliard]

Here is the transcript of a recent discussion on Quarkus zulip around that topic

Ioannis Canellos said:

Testing the templates e2e is something that does make sense to me. However, I would rather keeping things as simple as possible as we are still pretty volatile. So, it might make sense to start with an e2e that checks that template is rendered correctly. The rest are nice to have and of lower priority.

Charles Moulliard replied

It is easier to test if an application generated from a backstage template runs than validating the generated resources.

The idea to check if a curl POST http://localhost:7007/api/scaffolder/v2/tasks works is interesting but is no so simple to achieve. Why ?

• The backend don't report errors occurring post execution of the template
• The steps of the template can imply to access a system like: github, argocd, etc and by consequence such systems should be available to avoid that the template execution fails
• The resources have been generated properly but the application scaffolder will fail when running on cluster for reasons x.y.z

Remarks:

• We could use playwright to perform some e2e tests on the plugin or template but again creating such scenario able to mimic human interactions with the UI will be time/effort consuming. Some examples are available
  • https://github.com/backstage/backstage/blob/master/packages/app/e2e-tests/HomePage.test.ts
  • https://github.com/janus-idp/backstage-plugins/blob/main/plugins/kiali/tests/kialiHelper.ts The project backstage only include a few playwright UI tests !
• Template DryRun mode could help us (as reported here: https://github.com/backstage/backstage/issues/14280) as it allows to generate the resources without saving them under a git repository in a n automated way using a simple client BUT some steps of the template execution will be skipped as dry-run mode is not supported by all the plugins. See screenshot

So the question is: What do we want to control/verify here ?

@iocanel

[cmoulliard]

Ioannis Canellos said:

As QShift is not going to be a product of its own, but something that could be added to RHDH, I am not sure if it makes sense to test installing QShift, at least not before its possible to do that on top of RHDH. So, I would skip that for now.

Charles Moulliard replied:

We have to test at least one golden path like the Quarkus TODO application to be sure that what we develop end to end is working : plugin - actions + template + resources generated by backstage or backastage + Quarkus CLI and this is why we need a cluster provisioned with: argocd + tekton + kubevirt + gitea + etc

The skeleton of job already exists and could be integrated top of backstage-playground !

Remark: I don't think that janus-idp/RHDP test end to end what they ship except if QE play some scenario. To be checked

@iocanel

[aureamunoz]

If we can have an automatized way to verify that everything is ok and no broken, I vote for it. But I don't see the complexity of putting in place this end2end job so it could be too much effort

[cmoulliard]

Such a test case could help us to test an action and what it has been generated:

// https://github.com/janus-idp/backstage-plugins/blob/main/plugins/scaffolder-annotator-action/src/actions/annotator/annotator.test.ts#L61-L122
...
  it('should call action to annotate catalog-info.yaml', async () => {
    const catalogInfoYaml = await fs.readFile(
      resolveSafeChildPath(workspacePath, './catalog-info.yaml'),
      'utf8',
    );
    const action = createAnnotatorAction(
      'catalog:test-annotate',
      'Creates a new `catalog:test-annotate` Scaffolder action to annotate catalog-info.yaml with labels and annotations.',
      '',
      {},
    );

    const logger = getVoidLogger();
    jest.spyOn(logger, 'info');

    const mockContext = {
      workspacePath,
      logger,
      logStream: new PassThrough(),
      input: {
        labels: {
          label1: 'value1',
          label2: 'value2',
          label3: 'value3',
        },
        annotations: {
          annotation1: 'value1',
          annotation2: 'value2',
          annotation3: 'value3',
        },
      },
      output: jest.fn(),
      createTemporaryDirectory() {
        // Usage of createMockDirectory is recommended for testing of filesystem operations
        throw new Error('Not implemented');
      },
    };

    await action.handler(mockContext as any);

    let entity: { [key: string]: any } = yaml.parse(catalogInfoYaml);
    entity = {
      ...entity,
      metadata: {
        ...entity.metadata,
        labels: {
          ...entity.metadata.labels,
          ...mockContext.input.labels,
        },
        annotations: {
          ...entity.metadata.annotations,
          ...mockContext.input.annotations,
        },
      },
    };

    expect(logger.info).toHaveBeenCalledWith('Annotating your object');
    expect(mockContext.output).toHaveBeenCalledWith(
      'annotatedObject',
      yaml.stringify(entity),
    );
  });

So is allows to:

• Set the input parameters (aka what we do in a template)
• Call an action
• Get from the temporary folder what it has been generated
• Verify conditions x.y.z on the file(s) generated

[cmoulliard]

The following job (which still need improvements) is working end to end : https://github.com/ch007m/test-github-action/actions/runs/9514046915/workflow

I will create a PR to integrate it on this project

FYI: @aureamunoz @iocanel

[aureamunoz]

Great! kudos @cmoulliard !