dataProviders always run, even if no active tests need them

[fredemmott]

<?php

class ProviderTest extends PHPUnit_Framework_TestCase {
  /**
   * @dataProvider myProvider
   */
  public function testStuff($foo, $bar) {
  }

  public function myProvider() {
    echo "Provider is running (and sleeping)!";
    sleep(3);
    echo "Now throwing an exception to see if PHPUnit cares";
    throw new Exception('meh');
  }
}

If all tests are run (this output is expected):

fred@hhvm-oss-bench:~/testproject$ hhvm vendor/bin/phpunit ProviderTest.php
Provider is running (and sleeping)!Now throwing an exception to see if PHPUnit caresPHPUnit 5.1.3 by Sebastian Bergmann and contributors.

W                                                                   1 / 1 (100%)

Time: 4.42 seconds, Memory: 5.85Mb

There was 1 warning:

1) Warning
The data provider specified for ProviderTest::testStuff is invalid.
meh

WARNINGS!
Tests: 1, Assertions: 0, Warnings: 1.

If --group is used and no tests match:

fred@hhvm-oss-bench:~/testproject$ hhvm vendor/bin/phpunit --group noTestsInThisGroup ProviderTest.php
Provider is running (and sleeping)!Now throwing an exception to see if PHPUnit caresPHPUnit 5.1.3 by Sebastian Bergmann and contributors.

Time: 4.3 seconds, Memory: 5.42Mb

No tests executed!

I'm throwing the exception there to point out that PHPUnit seems aware that the dataprovider failing doesn't matter, as no tests use it.

I care about this behavior as I have a test suite I can run both locally and against a remote server, and the data provider for one of my tests does a few thousand page loads. I don't want this provider to run when I specify --group remote, as no tests that use the provider are in that group, and the provider is /really/ slow when ran against a remote server.

[sebastianbergmann]

This is a known limitation of PHPUnit's test runner that cannot (properly) be overcome without rewriting it.