Add Support for the Testing API

[bscan]

Add support for the vscode testing API. This would allow for auto-discovering and running t/*.t tests tests directly in vscode. Open to design decisions, but I suspect we should run them via https://perldoc.perl.org/TAP::Harness as it is a core module and we can leverage the perl interpreter setting (as opposed to requiring a user to specify a location for the prove binary).

This is what the API looks like for an example set of tests in Python.

https://code.visualstudio.com/api/extension-guides/testing

[m-dango]

Making a start on this. Using TAP::Harness is a good idea, but would ideally like to allow support for App::Yath/Test2::Harness as well.

[rabbiveesh]

Yeah, I second allowing some config for what to run tests with. This is cool!

[m-dango]

I'm going to investigate potentially using Test2::API for this, I believe its available as a core module.

[m-dango]

Poking around with the details of TestController and figuring out how to piece everything together (I don't have much experience with writing VSCode plugins).

Looking to make use of Test2::API::intercept:

my $events = intercept {
    do '/path/to/test.t';
};

https://metacpan.org/pod/Test2::API::InterceptResult#SYNOPSIS

The goal would be to obtain the relevant data to pass into TestItem.

There may be a more effective way of achieving this but still experimenting and investigating the docs/code.

[m-dango]

use Test2::API qw<intercept_deep>;
use JSON::PP qw<encode_json>;

my $events = intercept_deep {
    do $ARGV[0];
};

say encode_json($events->squash_info->flatten);

Example test file as input for above:

use strict;
use warnings;

use Test::More;

ok 1, 'yay';
ok 0, 'nay';

subtest 'nest' => sub {
    ok 1, 'subtest yay';
    ok 0, 'subtest nay';
    is_deeply {foo => 1}, {bar => 0}, 'subtest hash'
};

done_testing;

Resulting JSON:

[
  {
    "name": "yay",
    "pass": 1,
    "trace_file": "./t/hello.t",
    "causes_failure": 0,
    "trace_line": 8
  },
  {
    "trace_line": 9,
    "trace_file": "./t/hello.t",
    "causes_failure": 1,
    "pass": 0,
    "name": "nay",
    "diag": [
      "  Failed test 'nay'\n  at ./t/hello.t line 9.\n"
    ]
  },
  {
    "note": [
      "Subtest: nest"
    ],
    "causes_failure": 0,
    "trace_file": "./t/hello.t",
    "trace_line": 15
  },
  {
    "trace_line": 12,
    "trace_file": "./t/hello.t",
    "causes_failure": 0,
    "pass": 1,
    "name": "subtest yay"
  },
  {
    "trace_line": 13,
    "trace_file": "./t/hello.t",
    "causes_failure": 1,
    "pass": 0,
    "name": "subtest nay",
    "diag": [
      "  Failed test 'subtest nay'\n  at ./t/hello.t line 13.\n"
    ]
  },
  {
    "name": "subtest hash",
    "diag": [
      "  Failed test 'subtest hash'\n  at ./t/hello.t line 14.\n"
    ],
    "pass": 0,
    "trace_file": "./t/hello.t",
    "causes_failure": 1,
    "trace_line": 14
  },
  {
    "causes_failure": 0,
    "trace_file": "./t/hello.t",
    "trace_line": 14,
    "diag": [
      "    Structures begin differing at:\n         $got->{bar} = Does not exist\n    $expected->{bar} = '0'\n"
    ]
  },
  {
    "trace_file": "./t/hello.t",
    "causes_failure": 0,
    "trace_line": 15,
    "plan": "3"
  },
  {
    "diag": [
      "Looks like you failed 2 tests of 3.\n"
    ],
    "trace_line": 831,
    "trace_file": "/home/dango/.asdf/installs/perl/5.38.0/lib/5.38.0/Test/More.pm",
    "causes_failure": 0
  },
  {
    "trace_line": 15,
    "subtest": {
      "follows_plan": 1,
      "is_passing": 0,
      "failed": 2,
      "count": 3,
      "plan": 3
    },
    "causes_failure": 1,
    "trace_file": "./t/hello.t",
    "pass": 0,
    "name": "nest",
    "diag": [
      "  Failed test 'nest'\n  at ./t/hello.t line 15.\n"
    ]
  },
  {
    "trace_line": 17,
    "trace_file": "./t/hello.t",
    "causes_failure": 0,
    "plan": "3"
  }
]

[bscan]

Thanks @m-dango, this is great. Support for tests would be a great addition. I'll take a closer look later this week, but I also wanted to point out this extension that's worth looking at for design ideas. https://github.com/Nihilus118/vscode-perl-test-provider