New idea for remote CC

[SamMousa]

After reviewing #85 (good work keep it going!) I thought of a different approach to code coverage that might work cleanly as well.

The idea:

1. Before a test starts, connect to C3, this is a "long lived" connection. C3 creates a file temp file with a fixed name and watches it for data, this data is streamed back to the coverage module.
2. During the test each request to the server is profiled and the serialized result (php-codecoverage library supports serialization) is atomically written to the local temp file.
3. After the test ends we disconnect the stream from 1 and this prompts the remote coverage file to be removed

This could simplify things:

• We no longer need to mark each request with a cookie, which needs to be set in the browser module (Selenium, REST, or Php)
• We no longer have edge cases where fetch requests might not send cookies for example
• We cannot support parallel tests since we won't be able to know to which test the coverage belongs

Streaming coverage

We can stream coverage for each request like this using the PHP report class (copied below since its short)

final class PHP
{
    public function process(CodeCoverage $coverage, ?string $target = null): string
    {
        $coverage->clearCache();

        $buffer = "<?php
return \unserialize(<<<'END_OF_COVERAGE_SERIALIZATION'" . PHP_EOL . serialize($coverage) . PHP_EOL . 'END_OF_COVERAGE_SERIALIZATION' . PHP_EOL . ');';

        if ($target !== null) {
            if (!strpos($target, '://') !== false) {
                Filesystem::createDirectory(dirname($target));
            }

            if (@file_put_contents($target, $buffer) === false) {
                throw new WriteOperationFailedException($target);
            }
        }

        return $buffer;
    }
}

1. Capture it like normal
2. Use the SebastianBergmann\CodeCoverage\Report\PHP report to process it to a string
3. Stream the result to the calling code
4. The calling code splits it using the end marker.
5. Each part is deserialized to an of Coverage, which can be merged with the base coverage object via $baseCoverage->append($unserializedCoverage)`

Optionally we could do some automatic path detection which in most cases should actually be trivial.

Thoughts?

[SamMousa]

I actually couldn't let this rest, here's a working implementation that collects coverage and streams it as SSE. (I chose SSE since traefik recognizes text/event-stream mimetype as something it should not cache)

https://github.com/Codeception/c3/assets/547021/45083734-6248-4075-85b6-f283a8d02741

The full implementation for the server is this (Proof of Concept quality):

<?php

namespace collecthor\helpers;

use SebastianBergmann\CodeCoverage\CodeCoverage;
use SebastianBergmann\CodeCoverage\Driver\Selector;
use SebastianBergmann\CodeCoverage\Filter;
use SebastianBergmann\CodeCoverage\Report\PHP;

final class C3 {

    private string $coverageFile;
    public function __construct(
        string $secret,
        string|null $tempDir = null
    )
    {
        $this->coverageFile = ($tempDir ?? sys_get_temp_dir()) . '/' . md5($secret);

        // Check if this is a request to stream coverage, this is
        if (isset($_REQUEST['codeception_coverage_start'])) {
            // Check secret is valid
            if (!hash_equals($secret, $_REQUEST['codeception_coverage_start'])) {
                return;
            }

            $this->hijackRequest();
        }
        $this->collectCoverage();
    }

    /**
     * Hijack the request and stream coverage report from other requests
     */
    private function hijackRequest(): never
    {
        set_time_limit(0);
        ob_implicit_flush(1);
        header("Cache-Control: no-store");
        header('Content-Type: text/event-stream');
        header('X-Accel-Buffering: no');

        touch($this->coverageFile);
        register_shutdown_function(fn() => unlink($this->coverageFile));
        $handle = fopen($this->coverageFile, 'r');
        echo "event: welcome\n";
        echo 'data: Welcome to the coverage streamer!';
        echo "\n\n";
        while (connection_aborted() === 0) {
            echo "event: ping\n";
            echo 'data: {"time": "' . date(\DateTimeInterface::ATOM) . '"}';
            echo "\n\n";

            $buffer = stream_get_contents($handle);
            if ($buffer !== "") {
                echo "event: coverage\n";
                echo "data: start " . substr($buffer, 0, 100);
                echo "data: end " . substr($buffer, -100, 100);
                echo "\n\n";
            }
            sleep(3);
        }

        exit;
    }

    private function collectCoverage(): void
    {
        // TODO: Retrieve this filter from request
        $filter = new Filter();
        $coverage = new CodeCoverage((new Selector())->forLineCoverage($filter), $filter);
        // Use coverageFile as ID
        $coverage->start($this->coverageFile);

        register_shutdown_function(function() use ($coverage) {
            $coverage->stop();
            $serializedCoverage = (new PHP())->process($coverage);
            file_put_contents($this->coverageFile, $serializedCoverage, FILE_APPEND | LOCK_EX);
        });

    }

}

I used it in my entry scripts like this:

new C3('test');

The codeception side (client side) implementation is trivial, all we gotta do is do the request I did in the terminal, read from the socket parse SSE which probably has been implemented in some library and unserialize the coverage objects...

Sounds very cool to me!

[samdark]

Nice 👍 Looks simpler than current solution.

[SamMousa]

This library can be used to parse the SSE events from the server: https://github.com/clue/reactphp-eventsource

Edit: it is a big dependency to pull in, so we should probably not bundle it with the Codeception core (current implementations of the C3 client live in the core)

Edit 2: another simpler implementation that we could include or build ourselves https://github.com/obsh/sse-client

[DavertMik]

@SamMousa how good was this approach in the long run? Have you tried to use it for collecting code coverage?

If it worked well, we can think to create a new project that eventually might replace c3, if you think this implementation is better

[SamMousa]

I have not put it in production and am not focused on e2e tests in the browser at this time. Had to re-read this thread just to remember I did this...