Feature Request: support for parallel testrun

ngraf commented 3 years ago

When it comes to parallel test execution, there is currently no way to make it happen without logical conflicts.

Problem #1 Given you run npx codceptjs run-workers 2 Then you are not able to to make sure I.vrtStart is only executed once.

Problem #2 Given you run npx codceptjs run-workers 2 Then the helper is not shared across workers And you will end with I.vrtTrack failing in the worker where I.vrtStart was not executed, because I.vrtStart is needed from code point of view, but not possible because of Problem #1

... to make it short: Using this helper in parallel is close to impossible or I am just missing the right documentation how to make it possible.

pashidlos commented 3 years ago

@ngraf thanks for your request could you clarify what are the versions in docker compose? it should be no problem to start build with the same ciBuildId several times from multiple threads all the results should be aggregated in one build

are you having any specific error?

ngraf commented 3 years ago

could you clarify what are the versions in docker compose?

Versions are: visualregressiontracker/ui:4.13.0 visualregressiontracker/migration:4.13.0 postgres:12

are you having any specific error?

yes, I will get this error when I.vrtStart() is executed 2nd time in parallel thread

   {
    "path": "/builds",
    "name": "Error",
    "message": "\nInvalid `prisma.build.create()` invocation:\n\n\n  Unique constraint failed on the fields: (`projectId`,`ciBuildId`)",
    "exception": {
        "code": "P2002",
        "clientVersion": "2.12.1",
        "meta": {
            "target": [
                "projectId",
                "ciBuildId"
            ]
        }
    },
    "stack": "Error: \nInvalid `prisma.build.create()` invocation:\n\n\n  Unique constraint failed on the fields: (`projectId`,`ciBuildId`)\n    at PrismaClientFetcher.request (/node_modules/@prisma/client/runtime/index.js:79355:15)\n    at processTicksAndRejections (internal/process/task_queues.js:97:5)"
}

JustSittinHere commented 3 years ago

I had a similar issue with the wdio reporter in parallel. Didn't happen every time and looked like a race condition. Work around was to create the session in vrt before any workers started (so the create operation was only executed once)

https://github.com/JustSittinHere/wdio-visual-regression-tracker-service/blob/d24a9acce66a67efcef04da4a3e26f0b609860b0/src/launcher.ts#L14

ngraf commented 3 years ago

Thanks for the input @JustSittinHere . That approach of starting the VRT build before the workers spawn, is also what I tried, but the current implementation of agent-codeceptjs is not capable of this approach, because I.vrtTrack has a hard-coded dependency that I.vrtStart has been executed before in same worker thread. Thatswhy I think the agent needs a refactoring. One approach would be to somehow provide a flag to tell the helper that build has already been started by somebody somewhere else and the helper can trust you that I.vrtTrack will work just fine with the given configuration of the helper.

pashidlos commented 3 years ago

@ngraf @JustSittinHere thanks for clarification this one is fixed in upcoming release 4.14 in scope of https://github.com/Visual-Regression-Tracker/Visual-Regression-Tracker/issues/258

more details about release https://github.com/Visual-Regression-Tracker/Visual-Regression-Tracker/releases/tag/4.14.0

you could already upgrade image versions to: ui:4.14.0 api:4.14.0 migration:4.14.0

please, reopen if reproduces

Visual-Regression-Tracker / agent-codeceptjs

Feature Request: support for parallel testrun #28