Support installation of extensions for tools

[sebastianbergmann]

PHPUnit 5.7 introduces a mechanism for loading all PHAR archives from a directory (for instance tools/phpunit.d). This would allow the easy usage of PHPUnit extensions when PHPUnit is used from a PHAR.

It would be nice if Phive could install such extensions and not just the tool they extend.

[mwgamble]

PHPStan also has the issue of it being unclear how plugins should be installed when it is installed using Phive.

[ondrejmirtes]

I cleared up the extension installation for PHPStan here: https://github.com/phpstan/phpstan/issues/891

TL; DR: It's not currently possible and I don't see the need for it. Use Composer with phpstan/phpstan-shim instead 😊

[theseer]

The reason we did not yet implement support for extensions is that we need a working concept for their version constraints and update handling first.

Let's assume we add support for installing extensions, then these extensions (obviously) depend on a certain version of the tool they extend. While that itself seems comparatively easy to implement as it's not recursive for a single extension, it gets more complex with multiple extensions or update requests. Now we have to find a working set of versions of the tool and the extensions that satisfy the given version constraints as well as php version: Welcome to dep-solving ;)

While that would of course be possible to implement - we're not the first ones to deal with this kind of issue ;) - in contrast to popular dependency resolvers, we do not (yet?) have a repository database that contains the available versions and dependencies of libraries, extensions or tools.

We could easily extend the phar-site-geneator for that, but how would that work with github releases? It won't, unless we want to download every release and extract the information from it.

Also, what's the expected behavior if I want to install an extension that requires a newer version of the tool? Automagically upgrade the tool? Choose an older version of the extension in case it's compatible with the currently installed version of the tool it's for? Fail?

I really want to support installing and managing extensions with phive. But we need to decide on where to get the information from without reinventing composer here ;)

[Update: Made the first sentence more clear]

[jaapio]

@theseer can we get this discussion back to live? For phpdocumentor we are investigating the possible ways to introduce extensions. I like the way phpunit does it now but there needs to be some way to distribute them when using a phar. And check the dependencies like composer does.

I fully argree with you that we shouldn't reinvent the wheel composer already provides. But I do agree with you that phive solves an issue which shouldn't be solved by composer. But without constraint checks the life of phar.io is quite limited.

Another issue we have in phpdocumentor is that we have plugins which require a php extension. But also phpdocumentor itself requires php extensions which can only be checked by phpdocumentor at runtime at this moment. The approach of @ondrejmirtes with phpstan/phpstan-shim is an interesting one. But doesn't solve all the conflicts we have. Escpecially in the reflection components of phpdocumentor which seem to be used by almost very tool :-)

If there is anything I could do to help you with some kind of prototype let me know.

[theseer]

Sure we can revive this :)

I didn't give up on this, just lacking a solid concept on how to approach the problems laid out above. Installing extensions with phive and for example using a custom directory (tools/$toolname.d for example?) is the easy part.

Regarding php-version and extensions: In case a manifest file is found, the requirements are verified by phive and all mismatches get reported.

Before we can start writing code for this (thanks for the offer!), we need to find answers to the above questions in my previous post.

If by any means possible I'd like to keep the distributed nature of things for phive. So far, everything works without any SPOF. Even the repositories.xml is technically not required as it only helps with alias resolving.

[jaapio]

Think the main issue is that you don't want to download all the revisions to see what is matching. I don't know the internals of composer but I think they are using packagist for this. The "easier" thing about the composer part is that they are reading a file from a repo. Which cannot be done in case of a phar. You will need to download the full package.

When you are doing the approach of composer and packagist, like every other package manager does I'm wondering if we can reuse packagist for this? Since most of the applications which are downloaded by phive are published on packagist. Only part that is missing is a link between the package and the phar? I understand that this will have an impact on the distributed repositories and way of installing.

The problem composer has to solve and phive doesn't have to is the dependencies between packages. phpunit and its extensions should be compatible. But conflicts between phpstan and phpunit are ok.

That makes the issue slightly easier to solve.

[theseer]

Hi again!

I'm afraid things are a bit more complicated than it seems. Even if we were ignoring for a second that relying on packagist for resolving would kill the idea of not having any single, central place that has to be available, the answers packagist could provide are not likely to help here.

We do not need a full (potentially not even resolvable) dependency graph that considers all the libraries used by all tools. We need this on a packaged phar base with versions already chosen.

The composer.json, which is the base of packagist's database, contains version constraints rather than actually selected versions. That's perfectly fine for composer of course but not helping in our situation.

We need a list of releases along with their requirements on the environment like php verison or extensions and, for extensions to applications, a version constraint for that application.

An extension to an application can of course come with its own set of libraries. As soon as you add a 2nd extension, you might run into a conflict unless they both prefix their bundled code with a private namespace - e.g. by using php-scoper. We simply cannot fix that situation as the phar's are already made.

So my point is: The very concept of composer and packagist is that it will try to come up with an installable set of libraries. That means, for example, my PHPUnit 7.2.1 installation can be very much different from yours if some other dependency forces different versions of libraries to be used. Technically speaking, there is no single 7.2.1 PHPUnit release with composer. Something at least I do consider a serious problem and one of the core reasons phive got invented.

With a phar, the selected versions are fix. That's not a problem for independently run tools as we support right now. But of course will be a problem for adding extensions to these tools as soon as you have a 2nd extension which may or may not come with its own set of 3rd party libraries.

[prudloff-insite]

We have a similar use case because we install PHPCS with phive on our CI and we need to install additional standards. We currently use wget but it is not ideal (no cache, no automatic checksum validation, etc.).

For some standards we could use phive with an URL, for example:

phive install https://ftp.drupal.org/files/projects/coder-8.3.9.zip --force-accept-unsigned

(And we then take care of unzipping it somewhere.)

But it will not work if the version can't be extract from the URL. For example:

phive install https://github.com/PHPCompatibility/PHPCompatibility/archive/9.3.5.zip --force-accept-unsigned