Make console testing easier

[matt-allan]

The console tests are kind of hard to use.

If your expectsOutput assertion is wrong you get a simple error message that doesn't tell you what was printed instead.

There was 1 failure:

1) Tests\Feature\InspiringCommandTest::testInspiringCommand
Output "Simplicity is the ultimate sophistication!" was not printed.

/Users/matt/code/ex-cli/vendor/laravel-zero/foundation/src/Illuminate/Foundation/Testing/Concerns/InteractsWithConsole.php:52
/Users/matt/code/ex-cli/vendor/laravel-zero/foundation/src/Illuminate/Foundation/Testing/TestCase.php:140

FAILURES!
Tests: 1, Assertions: 2, Failures: 1.

   PHPUnit\Framework\AssertionFailedError  : Output "Simplicity is the ultimate sophistication!" was not printed.

I've been working around this by commenting out the artisan call, using Artisan::call to see what it actually prints, then fixing the assertion. PHPUnit's assertions are normally able to give you a diff which makes fixing these kinds of errors a lot simpler.

It's also not easy to assert multi line output like tables are printed. I've been working around that by doing this:

$this->artisan(...)
            ->expectsOutput('| Project | Date        | Start    | End      | Elapsed |')
            ->expectsOutput('| blog    | May 4, 2019 | 12:00 pm | 12:30 pm | 0:30    |')
            ->expectsOutput('| blog    | May 5, 2019 | 12:00 pm | 1:30 pm  | 1:30    |')
            ->expectsOutput('| blog    | May 5, 2019 | 2:00 pm  | 3:30 pm  | 1:30    |')
            ->expectsOutput('Total hours: 3:30');

Some other things I think would be nice:

• negative output assertions i.e. doesNotExpectOutput
• partial output matches i.e. phpunit's assertRegexp

If we could get to the actual output (the same way you can access the actual $response object for HTTP tests) it would be possible to fix this, but currently we can't.

There is also an existing issue about the expectsQuestion helper not working with multiple choice questions.

Another thing that is really confusing is, because the assertions aren't made until the beforeApplicationDestroyed callback, the collision stack trace shows the error coming from the tearDown method even though it really comes from the test method:

There was 1 failure:

1) Tests\Feature\InspiringCommandTest::testInspiringCommand
Output "Simplicity is the ultimate sophistication!" was not printed.

/Users/matt/code/ex-cli/vendor/laravel-zero/foundation/src/Illuminate/Foundation/Testing/Concerns/InteractsWithConsole.php:52
/Users/matt/code/ex-cli/vendor/laravel-zero/foundation/src/Illuminate/Foundation/Testing/TestCase.php:140
/Users/matt/code/ex-cli/tests/Feature/InspiringCommandTest.php:13

FAILURES!
Tests: 1, Assertions: 2, Failures: 1.

   PHPUnit\Framework\AssertionFailedError  : Output "Simplicity is the ultimate sophistication!" was not printed.

  at /Users/matt/code/ex-cli/tests/Feature/InspiringCommandTest.php:13
     9|     public function tearDown(): void
    10|     {
    11|         // clean up fixtures or something
    12| 
  > 13|         parent::tearDown();
    14|     }
    15| 
    16|     /**
    17|      * A basic test example.

A lot of these issues are difficult to fix at the moment because Illuminate\Foundation\Testing\PendingCommand is using mocks for input and output. You can't really get to the underlying input or output objects.

Symfony has some excellent test helpers that I think we can use instead. The tester does a lot of nice things for you like splitting STDERR and STDOUT, making the input and output accessible, setting the verbosity and is-interactive flags, prevents the input stream from blocking, etc.

I think we might be able to use these instead and solve a lot of these issues without changing the public interface?

[mfn]

I realized this a few years ago and used the CommandTester in my application; all the existing helpers weren't really useful to me.

I added a helper to the TestCase of my app and pass the already resolved command to it:

    /**
     * The `CommandTester` is directly returned, use methods like
     * `->getDisplay()` or `->getStatusCode()` on it.
     *
     * @param Command $command
     * @param array $arguments The command line arguments, array of key=>value
     *   Examples:
     *   - named  arguments: ['model' => 'Post']
     *   - boolean flags: ['--all' => true]
     *   - arguments with values: ['--arg' => 'value']
     * @param array $interactiveInput Interactive responses to the command
     *   I.e. anything the command `->ask()` or `->confirm()`, etc.
     * @return CommandTester
     */
    protected function runCommand(Command $command, array $arguments = [], array $interactiveInput = []): CommandTester
    {
        $command->setLaravel($this->app);

        $tester = new CommandTester($command);
        $tester->setInputs($interactiveInput);

        $tester->execute($arguments);

        return $tester;
    }

and then in a test:

        $command = $this->app->make(CommandToTest::class);
        $tester = $this->runCommand($command, [
            'param1' => $param1,
            'param2' => $param2,
        ], [
            'y', // response to interactive question
        ]);

Then, on the tester you can perform the assertion via regular assert methods ($this->assertSame(…, $tester->getStatusCode());, $this->assertRegExp(…, $tester->getDisplay());)

[sebdesign]

I created some helper methods a few years ago to test the actual output of commands instead of using mocks: https://github.com/sebdesign/laravel-state-machine/blob/v1/tests/ConsoleHelpers.php

[ohnotnow]

Just a little extra voice on this. Spent quite a lot of time debugging a Output "my expected bit of text" was not printed' which could have been resolved in a few seconds if the test error showed what _was_ printed. Or some kind of$this->artisan('my:command')->dump()` feature I guess.

[michaelklopf]

@ohnotnow What was the reason for your error? I have the same issue at the moment, but don't know where to look first.

[ohnotnow]

@michaelklopf I can't remember I'm afraid. But it's a generic error message so not sure my case would be of much help.

[peterjaap]

Still nothing on this? Testing console commands is extremely hard when we can't know whats actually being sent to the output...