Make regreph Composer friendly

[hkdobrev]

Right now Regreph expects the library which would be benchmarked to be installed via Composer as a dependency of Regreph.

Projects not publicly listed as a Composer package (e.g. on packagist.org) could not be benched with the default flow.

My ultimate goal would be Regreph to be installed as a dependency similar to how PHPUnit is installed as dependency when you run tests with it.

This way you could easily require Regrpeh in a CI tool and bench a project.

Generally this would be a more natural flow and more similar to other tools.

[lox]

I was thinking about this a fair bit last night. Being able to install regreph as a composer dependency and then run it against the current codebase would absolutely make it a more useful tool .

The issues I've been grappling with is where to locate the regression test cases. In the proof of concept I made them totally separate, so that you could compare arbitrary revisions of code without worrying about whether that revision had performance tests in them.

There are a few scenarios:

1) You have performance tests in the current revision and in the baseline revision and you just want to run them. This is easy, you just checkout the baseline revision into a .regreph/ dir and run the comparison on the two.

2) You have tests in the current revision but want to compare it to a baseline that didn't have tests. You could have a --backport flag that simply copied performance tests from the current revision to the baseline checkout dir.

3) Same as 2), except the current tests won't work due to incompatible code in a baseline revision. Perhaps you could have some form of annotation on test methods that show what git revision that work agains? Imagine /** @workswith ~v2.0.0*/ or /** @workswith >#as32afd59 */. This sounds complicated and is probably worth just not supporting.

4) No tests in the repo. For this you could support the current "fetch tests from a url" approach, or allow a regreph branch with tests in it.

All of those scenarios are pretty manageable. Having regreph as an automated tool in CI would be awesome, excited to see it happen.

[lox]

My thinking is we could either locate the tests in a canonical location that is friendly to the majority of php projects, something like tests/regreph or we could have a top level regreph.json file with information about where the tests are and other configuration.

Thoughts?

[hkdobrev]

@lox These are good ideas!

Here are my thoughts:

• First of all we should not call them tests, test cases or anything like that. These are profiles for the Regreph profiler. An example of such naming is the PHPPerf project: https://github.com/makeusabrew/phpperf/
• About the directory default name I suggest one of the following:
  • speed/
  • profiles/
  • regreph/

I like speed/ and regreph/ more, because you would often put just one profile class and a lot of profiles data files. For example if you are benchmarking a Markdown parser you'd have the following file structure:

root
|
|--src/
|--speed/
   |
   |--Profile.php
   |--profiles/
      |
      |--jquery.readme.md
      |--angularjs.readme.md

• About regreph.json I think we would have very little information to put in it. So for now we could just use CLI arguments. Using a configuration file is quite a future lock-in so it should be introduced only when needed and things are finalized.
• About the branches and the incompatible profiles. I think backporting would create more problems than it would solve. I will think more about that.