Alternative provider-concept

[mindplay-dk]

As discussed in this very long comment and further described and discussed in this longer thread with @mnapoli and @moufmouf, I now have some code-samples for review and further discussion.

The current version of the alternative interfaces are available (and documented) here:

https://github.com/mindplay-dk/provider-interop

They are not yet listed on packagist - the repository is listed in composer.json in the various forks I've created to provide more context for the discussion.

Also note that the the set() method (which was initially proposed in the discussions as a means of providing a micro-optimization) has been omitted from the current ServiceProviderInterface for simplicity. (and arguably may not be necessary at all - and could, for that matter, do harm in mutable containers such as Pimple/Simplex, where importing/exporting mutable entries could lead to unexpected side-effects...)

Container Support

I've implemented ServiceProviderInterface and ServiceRegistryInterface in an immutable container - I chose my own container mindplay/unbox because I know it well.

The fork is here:

https://github.com/mindplay-dk/unbox/compare/provider-interop

Support for ServiceRegistryInterface had to be implemented with a ServiceRegistryAdapter proxying ContainerFactory, because the method-name register() collided with Unbox's own register() method, which has an incompatible signature.

I also wanted to see how this works out in a mutable container, so I also forked mnapoli/simplex here:

https://github.com/mnapoli/simplex/compare/master...mindplay-dk:provider-interop

Implementation here was a bit more straightforward, as none of the method-names collide with Simplex' own methods - because the container is mutable, both ServiceProviderInterface and ServiceRegistryInterface were implemented directly on the Container class.

More notes regarding the changes and differences can be found in the commit-log here.

Provider Implementation

I chose the universal Twig-module as a test-subject for retro-porting this bootstrapping to a proprietary Simplex-provider - you can view the changes here:

https://github.com/thecodingmachine/twig-universal-module/compare/1.0...mindplay-dk:provider-interop

More detailed notes can be found in the commit-log here.

Essentially, this was pretty simple, and doesn't directly have anything to do with the alternative proposal, but I wanted to illstrate the difference - you can now come at this as a developer, having selected a DI container with features based on your requirements.

Providing interoperability is no longer the responsibility of every individual developer building a module - interoperability is now a concern you deal with at the container-level, not at the provider-level, so this requires quite a different mindset, but you no longer have to choose between writing a "standard" vs "proprietary" provider.

Modularity

Note that this simple fork of Simplex doesn't allow you to specify what gets imported/exported, so there is no evidence at this time of the potential benefits of isolation between modules, e.g. no such thing as internal dependencies; everything gets exported.

The long-term concept here is that a given vendor, say, the vendor of Twig, would select the container that will be used to bootstrap their domain - the entire Twig domain will be bootstrapped, in isolation, in a self-contained, dedicated container exclusive to the Twig domain, so this includes any additional bootstrapping of Twig extensions, which would need to be performed against the container-implementation selected by the vendor of that domain.

This could be seen as a limitation, but I view it as a strength - the Symfony and Zend guys, for example, are likely already bootstrapping their modules using their selected containers, and using this approach, they don't need to port or rewrite their providers to a provider-standard; they just need to bootstrap their containers way they've always done, and maybe, in addition, whitelist/blacklist dependencies for export.

This should reduce the chance of collisions, since, e.g. a PSR-16 implementation can be registered as CacheInterface::class for internal use in a given domain/module/framework/etc. without the risk of overwriting a CacheInterface::class registration in your project.

I think this provides better and safer encapsulation of systems.

It doesn't attempt to replace proprietary provider-concepts native to various container-implementations with a "standard" - instead, it leverages existing proprietary provider-concepts indirectly.

While, clearly, this comes at the "cost" of not having features such as aliasing/extending/replacing in modules distributed this way, I don't believe these features are necessary or beneficial in the first place, for two reasons:

1. Registering entire systems/modules/domains in a single container is risky and likely leads to collisions - expecting vendors to coordinate and agree on conventions and risks across the entire eco-system is asking for a lot. I don't believe the single-container approach scales to complexity.

2. The current proposal strives for non-isolation, which is something that provider-concepts in existing containers already provide - having this kind of extensibility within a domain/system/vendor-space may be safe (at a smaller scale, e.g. Twig extensions bootstrapped against the same Twig-module-designated container) but having isolation between domains/systems/vendors at a larger scale is, I think, a must.

We have seen early symptoms of these issues in our own architecture, where e.g. 25-30 modules are currently bootstrapping a single container - so we're starting to see the need to separate domains/systems into dedicated containers in a more modular fashion, providing better isolation of sub-systems, and enabling us to selectively export entries designated for public consumption (e.g. services) while keeping internal dependencies (implementation details, such as repositories, caches, etc.) closed-off to actors operating within a given domain/system-boundary, e.g. system/domain-specific container instances.

I realize the problem we're trying to solve is somewhat different from the problem you're trying to solve, but I don't believe that standardization of providers, in the sense that you're proposing, is necessary. Modular "unsafe" bootstrapping is already possible with most container-implementations and, yes, requires proprietary configuration, but proprietary features is why we select a specific container in the first place - while it may lead to interoperability at a very low level, I don't believe that erasing those differences leads to improved high-level interoperability.

And yes, I understand that some of you don't believe that standardizing on this will erase the differences between containers, but I have a different point of view. I don't see the work that "developers" do as being different from the work that "frameworks" do - ultimately, that work is also done by developers, and monolithic frameworks are increasingly not a thing; most packages can stand alone, operate in the context of different frameworks, so, ultimately, everyday package developers will be the ones who have to make the choice between writing "standard" and "proprietary" providers. I believe this choice will lead to some degree of shaming of those who choose a proprietary container - in practice, most of use will have to forego the joys of our favorite container, which ends up being an implementation detail, except perhaps for some additional project-specific bootstrapping, but the tendency will most likely be to make everything modular and non-proprietary, so, in practice, it's likely we'll end up writing most (if not all) bootstrapping in the "standard" format.

And if that's the situation, the convenience of your favorite DI container will most likely end up being an annoyance, because you'll get in the habit of writing "standard" bootstrapping, and then will have to mentally "switch modes", in the rare case where you're "allowed" and feel good about using the proprietary syntax/features of your chosen container, e.g. on proprietary closed-source projects, at night, when no one's watching ;-)

And at that point, you will most likely just resign yourself to always writing "standard" providers.

I believe we can achieve better modularity with isolation of sub-systems and high-level interoperability, than with low-level interoperability and a new provider-format which will almost invariably end up competing with proprietary formats.

[moufmouf]

Hey Rasmus,

First, thanks a lot for opening the issue in here, this is definitely the right place.

This issue is kind of fascinating to me and I've got a huge number of comments to make. I'll try to keep those as organized as possible.

About the scope

I realize the problem we're trying to solve is somewhat different from the problem you're trying to solve, but I don't believe that standardization of providers, in the sense that you're proposing, is necessary.

Indeed. There is a real question about the scope of service providers. You are trying to solve a very complex and generic problem. When @mnapoli started container-interop/service-provider, this was clearly out of scope. In Matthieu's mind, service providers were meant to solve 80% of the problems (leaving the last 20% to manual configuration of the DI container).

For instance:

• You have more than one database? Out of scope! You configure your second or third database connection inside the container directly, without using service providers.
• You have more than one cache service? Out of scope too!
• The same goes for multiple loggers, etc...

There is actually an ongoing discussion about widening the scope here: #27, so the issue of the scope is very much in discussion right now. Your proposition is quite opposite to the view of Matthieu I think, because you are aiming at 100% coverage of all use cases.

Also, to be honest, I don't think we are the ones who should decide what is the correct scope. We should probably ask framework authors on the PHP-FIG mailing list to gather feedback from Zend, Symfony et al.

Ease of use

This is a bit akin to the scope. In my ideal world, I'd like service providers to be automatically detectable by the framework (using Puli or thecodingmachine/discovery for instance). So basically, the developer does a composer require my/package and automatically, the services are injected in his container(s). Of course, he must have the option to opt out, but I'd really like to have the possibility to write a container that does not require any manual registration of service providers. This goes of course totally the opposite direction of what you are trying to achieve (fine grained access to dependencies that are created).

I don't see exactly how we could design a solution that is as easy to use yet with your proposal, since the developer will have to setup at least as many containers as the required service locators require (there might be a solution but I don't see it right now).

Ability to work with compiled containers

When it comes to "compiled" or "cached" containers, one thing is really important. The time to bootstrap the container must not vary depending on the number of instances in the container. Whether your containers contain 1000 or 10 instances, the container should bootstrap in the same amount of time.

This is somehow possible with container-interop/service-providers (see https://github.com/thecodingmachine/service-provider-bridge-bundle). With your proposal, it seems to me that the "register" method will have to be called at least as many times as there are instances to inject, on each startup of the container. This will be a real problem for folks with compiled containers (like Symfony). You might want to try to implement a bridge with Symfony to see how it goes and if you can find optimisation strategies for provider-interop.

(note: I'm proposing one such optimization below).

Feedback of existing frameworks

As I said, we should probably ask on the PHP-FIG mailing list for some feedback from major framework owners. I can however already guess the answer from Fabien Potencier to your proposal. Basically, Fabien accepted to vote +1 on PSR-11 in part because the delegate lookup feature was removed. He is strongly opposed to having several containers running side-by-side in an application. So passing a standard that pushes the idea that several containers will run side-by-side will be an uphill battle.

And to be clear, I absolutely love your idea of containers publishing entries into another container. This looks like a glorified version of the delegate dependency lookup feature we had in container-interop (and I was a fond of that one). I'm just saying that this will probably be a no-go for some PHP-FIG members.

Is it possible to solve your issues with the current container-interop/service-provider interface?

Solution 1:

One of your complaints about the current implementation is that some service providers would have colliding identifiers. For instance, an ORM could require an APC cache service while an HTTP cache middleware could require a memcached based cache service. If both cache services use the CacheInterface identifier, we are screwed.

$container->register(ApcCacheServiceProvider::class);            // exports a CacheInterface identifier
$container->register(MemcachedCacheServiceProvider::class);      // exports a CacheInterface identifier (boom!)
$container->register(OrmServiceProvider::class);             // The ORM service will use the last CacheInterface identifier registered
$container->register(HttpCacheMiddlewareServiceProvider::class); // The middleware service will use the last CacheInterface identifier registered too
// #failure

Interestingly enough, a container could be clever enough to allow registering "mappings" along the service provider.

Let's imagine you are writing a container with a register method that takes 2 additional parameters:

• an array of mappings for "outbound" services (services exposed by the service provider to the container)
• an array of mappings for "inbound" services (services fetched from the container by the service provider)

My sample would therefore look like this:

$container->register(ApcCacheServiceProvider::class);            // exports a CacheInterface identifier
$container->register(MemcachedCacheServiceProvider::class, [ CacheInterface::class => 'alternative_cache' ]);      // the CacheInterface provided by this service provider is actually renamed to 'alternative_cache'
$container->register(OrmServiceProvider::class);             // The ORM service will use the CacheInterface identifier
$container->register(HttpCacheMiddlewareServiceProvider::class, [], [ CacheInterface::class => 'alternative_cache' ]); // Calls to $container->get('CacheInterface::class') are actually redirected to the 'alternative_cache' entry.
// #success

So with only one container, I can actually deal with somewhat complex use cases (this is up to the container to implement the mapping). You will feel this is less "clean" than your approach but at least, this does not force developers into using several containers.

That being said, this is not dictating that you should use only one container. This leads me to solution 2.

Solution 2:

Nothing should prevent you from using container-interop/service-providers in multiple containers that share entries.

I would certainly take a more direct route that would be more compliant with compiled containers.

Something like this:

// Containers can expose a set of entries to other containers
interface ProviderInterface extends ContainerInterface {
    // Returns an array of all exposed identifiers
    public function keys() : array; 
}

$mainContainer->import($container); // each container can decide to "import" the identifiers of another container (the "import" method is container specific and does not need to be standardized)

Also, one could probably write containers (or container builders) that both implement the current ServiceProviderInterface and the ProviderInterface.

I'm sure there are a lot of things to be improved in this proposal, but first, what do you think about it?

[mindplay-dk]

Your proposition is quite opposite to the view of Matthieu I think, because you are aiming at 100% coverage of all use cases

I know my proposal is, in a sense, "opposite" to what is currently proposed - I have been pointing this out as well.

I wouldn't say I'm aiming for greater coverage of use-cases, I'd say the opposite - I'd say, what I'm proposing has a narrower scope, since what you're proposing has to support (and effectively replace) many features, including aliasing, overriding and extending, which existing containers do in many different ways.

What I've tried to do, is redefine the problem, from "how do we standardize currently proprietary provider-concepts", to "how do we make containers interact as providers for each other".

I don't think we are the ones who should decide what is the correct scope. We should probably ask framework authors on the PHP-FIG mailing list to gather feedback from Zend, Symfony et al.

Good point, I agree - I'm not asking you to make any decision, at this stage I was just trying to see if I could communicate this potential alternative to you; we should definitely open the discussion on the mailing-list and see how people respond to the idea.

So basically, the developer does a composer require my/package and automatically, the services are injected in his container(s)

Two thoughts on this matter.

First, I don't think that's realistic. Where would things like database connections or any other configuration-dependent values come from? Most modules have hard dependencies of some form or another.

Secondly, I think you're favoring easy over simple - which means we'll end up paying for this in terms of complexity.

I strongly prefer simple solutions to simple problems.

How much work is it to type $container->add(new SMTPMailServiceProvider("localhost", 25)) after installing the package?

How to provide dependencies like hostnames and portnumbers? Now you need some kind of configuration-file concept/facility as well.

What about dependencies that aren't simple values and can't be defined in a configuration file format in the first place? Now you need some kind of configuration-based factory for PDO and every other object dependency.

How do you locate the container instance to auto-bootstrap in the first place?

What happens under test, where I'd want to bootstrap a different container instance?

How would I be able to temporarily disabled a module? If there's a line of code, I can comment it out - as opposed to editing composer.json or other configuration-files and having to run composer update or some other tool.

All of these questions (and I'm sure many more) arise from the aversion to write a single line of code, and ultimately makes everything more complex and less transparent.

If some frameworks want to provide that kind of ease of use, and are willing to put up with all the extra work and complexity, that's their business, but I would be very sad if we and up shipping a community standard that imposes that kind of complexity on simple projects.

With your proposal, it seems to me that the "register" method will have to be called at least as many times as there are instances to inject, on each startup of the container. This will be a real problem for folks with compiled containers (like Symfony). You might want to try to implement a bridge with Symfony to see how it goes and if you can find optimisation strategies for provider-interop.

I can think of one strategy that would work, which is to add a method to ServiceProviderInterface that enumerates all entry identifiers, e.g.:

public function listIdentifiers(): string[]

This would enable a caching/compiling container to obtain a list of all available entry identifiers and cache or compile them, such that an external container isn't bootstrapped until one of it's identifiers is looked up.

I don't think it would be too bad to add this requirement for containers, since most of them will need to implement something similar for internal use anyway, in order to enumerate and register exported entries against a ServiceRegistryInterface.

Basically, Fabien accepted to vote +1 on PSR-11 in part because the delegate lookup feature was removed. He is strongly opposed to having several containers running side-by-side in an application. So passing a standard that pushes the idea that several containers will run side-by-side will be an uphill battle.

Probably not so much that they're running, more the fact that they need to be bootstrapped up-front? I think, if another container needs to load, so you can get a service from an external domain, that's probably fine.

To your "solution 2", that's interesting - it's kind of where I'm going with listIdentifiers() as discussed above, and makes me wonder about a "hybrid" approach somewhere between the current proposal and the alternative I've presented.

What if that was all a provider did? Listed it's identifiers, and expected one container to look up those dependencies in it, at a later time.

The only interface we would need is this:

interface ServerProviderInterface extends ContainerInterface
{
    /**
     * @return string[] list of available identifiers
     */
    public function listIdentifiers();
}

Note that this extends ContainerInterface - so a provider is now something that can enumerate it's identifiers and provide them via get().

The ServiceRegistryInterface either wouldn't be needed anymore, or could be just something akin to the service-provider interface we see in proprietary implementations today:

interface ServiceRegistryInterface
{
    /**
     * @param ServerProviderInterface $provider
     */
    public function import(ServerProviderInterface $provider)
}

An implementation of import() would internally ask the provider to list the available indentifiers, then register each of them to a simple callback that calls get() on the provider.

For caching/compiling containers, you could proxy ServerProviderInterface with something that caches the list of entries and defers the creation and bootstrapping of the actual container.

This alternative approach is probably worth exploring - I think another fork of two containers (mutable, immutable) and the Twig module would offer more practical insight. I'm out of time for now though...

[mindplay-dk]

I'm pondering the idea of a hybrid approach.

Embracing interface segregation and SRP, maybe we should avoid inheriting ContainerInterface and instead design with composition in mind:

interface ServiceProviderInterface
{
    /**
     * @return string[] list of available identifiers
     */
    public function listIdentifiers();

    /**
     * @return ContainerInterface
     */
    public function getContainer();
}

This may have some advantages over the inheritance approach.

For one, this will allow implementations with caching/compilation concerns to address these more naturally - an implementation can internally cache the identifiers and defer the creation of the actual container until getContainer() is invoked, so this should align much better with containers that use these patterns.

Secondly, this seems to align better with provider-concepts in existing container libraries today, in the sense that most containers have providers as a (of course coupled but) separate concept from containers - that is, providers and containers can exist independently of each other, whereas the extended container-interface would mandate that a provider also be a container, which (in some situations, as described above) might lead to work-arounds like having to proxy the real container to defer loading and bootstrapping.

Interestingly, the ServiceRegistryInterface then seems to become an implementation detail - containers don't need to support a specific method for registration, as in my proposal, nor do they need to support a very specific bootstrapping data-format as in your proposal, and it still satisfies my concerns and reservations about bootstrapping against a single container-interface.

The transaction itself (of adding a provider) is similar to your proposal, but the "data format" is just the list of identifiers, which should be even faster to import and cache, and you can defer the bootstrapping of entire sub-systems until first use, which in practice should work really well for modules that deal with a specific domain - for example images, mail, user-management, etc. which are rarely all involved in the same transaction, e.g. by a single controller or service.

If there's a second interface, it might consist of something (again) better resembling the relationship that containers and providers have in existing libraries, e.g.:

interface ServiceRegistryInterface
{
    public function import(ServerProviderInterface $provider);
}

Note that this is still not a replacement for proprietary providers, it doesn't compete (and isn't "at odds") with existing provider-concepts, which let you leverage proprietary features of each container for low-level integration, e.g. within a domain boundary.

I'm not even sure what I'm proposing should be termed "provider" - perhaps "module" is more accurate, since this creates modularity, high-level integration, and low-level isolation.

The "provider" concept you're proposing more closely resembles the provider-concept we see in existing containers, which does not create modularity per se, but rather creates low-level integration.

I think that low-level integration has value within a domain boundary, e.g. being able to add features/extensions to Twig, authentication-methods to a user-module, image transformations to an image-module, etc. - but I don't think we need a standard provider format to achieve that.

I think, if a specific container is used to bootstrap a specific domain, that's likely fine. We wouldn't have another provider format competing with proprietary formats, and the community wouldn't have to port existing providers to a new format.

@moufmouf @mnapoli I would love to hear your thoughts on this?

[moufmouf]

First, I don't think that's realistic. Where would things like database connections or any other configuration-dependent values come from? Most modules have hard dependencies of some form or another.

The current idea is to store the configuration directly in the container: https://github.com/container-interop/service-provider/#managing-configuration Because "Configuration values should be treated like dependencies" (see this article http://paul-m-jones.com/archives/6203)

Secondly, I think you're favoring easy over simple - which means we'll end up paying for this in terms of complexity. ... If some frameworks want to provide that kind of ease of use, and are willing to put up with all the extra work and complexity, that's their business, but I would be very sad if we and up shipping a community standard that imposes that kind of complexity on simple projects.

I'm just saying I want it to be possible to have an "easy" solution. By all means, the community standard should not impose auto-discovery of service providers, but I think it should make it one possible implementation. Frameworks like Laravel have shown (somehow to my dismay and probably to yours too) that there is a strong preference for "easy" solutions in the PHP community. If we can find a standard that allows both easy implementations and simple implementations, that's better IMHO.

Embracing interface segregation and SRP, maybe we should avoid inheriting ContainerInterface and instead design with composition in mind:

interface ServiceProviderInterface
{
    /**
     * @return string[] list of available identifiers
     */
    public function listIdentifiers();

    /**
     * @return ContainerInterface
     */
    public function getContainer();
}

Agreed. Although it might be weird for containers that are providers too (a getContainer method that returns $this might seem fishy). But we can solve this breaking down things even more:

interface ServiceListInterface // Naming is terrible (sorry)
{
    /**
     * @return string[] list of available identifiers
     */
    public function listIdentifiers();
}

interface ServiceProviderInterface extends ServiceListInterface
{
    /**
     * @return ContainerInterface
     */
    public function getContainer();
}

Now, a provider that is also a container could implement ServiceListInterface and ContainerInterface while a provider that is building a container could implement ServiceProviderInterface.

@moufmouf @mnapoli I would love to hear your thoughts on this?

I understand where you are going with this idea.

I clearly find some appeal to it. And I would very much have a standard that allows us to list identifiers in a container / service provider (that could be very useful in a number of situations, like writing a generic container debugger/analyzer).

Now, with your proposal, we are shifting the responsibility of the choice of the container from the end-user to the package writer. Honestly, I have a hard time figuring out if this is a good or a bad thing.

And as I previously said, I find the need to instantiate containers a bit of an overkill. Containers are here so we don't have to wire classes manually and we are trading this complexity for the complexity of wiring containers together. So I feel like we are trading a problem for a similar problem.

Anyway, we should probably ask for some feedback from framework authors on this issue. I was going to ask for the formation of a working group on service providers anyway. Can I bring the subject to the PHP-FIG mailing list now, or do you need some more time to think or write prototypes on your idea first?

[mindplay-dk]

Configuration values should be treated like dependencies

Absolutely, I agree, but the example you referenced perfectly illustrates the design problem I'm describing - I don't mean "where does the value come from in the first palce", I mean, once you have the value, how does it get into the container?

When I say "hidden dependencies", I mean precisely what you're doing in this example - there's a hard dependency on a logFilePath entry, by which I mean, it won't work without this registration, and it has to be a valid value of the correct type, etc.

In other words, this example does not treat that value like a dependency - under any normal circumstances, if something depends on a certain value to even exist, you would use constructor-injection - the logFilePath is a perfect example, you won't even have a working registration at all unless this is registered.

If we can find a standard that allows both easy implementations and simple implementations, that's better IMHO

Of course - I'm only saying we can't favorize the easy solution if it comes at the cost of imposed architecture or arbitrary complexity that prevents (or cripples) the simple implementations.

a getContainer method that returns $this might seem fishy

In the mutable container scenario, yes, that's going to seem a little odd, but it'll universally support mutable and immutable containers alike, as well as allowing natural implementation of the "delegate container" pattern - the approach is simpler and less opinionated in that sense.

And as I previously said, I find the need to instantiate containers a bit of an overkill.

In some cases it well might be - it depends on the scale at which you're operating; some complex modules with optional feature add-on modules may well benefit from container features, in the same way that you benefit from these features at the project-level with one of these containers now.

In other cases, like smaller modules that bootstrap just a couple of things, where you don't benefit from complicated vendor-specific features, in deed something much simpler might suffice - perhaps something much like what you're proposing, a simple format supporting only a few basic features.

In those cases, you can select a very simple container, or for that matter implement the interfaces by hand, which would definitely make sense in some cases.

The main thing I'm trying to accomplish, is I don't want to feel forced to use a competing container format because it has been deemed a "standard". Whether I'm currently acting as module/framework/application-developer, I'm a developer, and introducing another alternative container format will mean a case-by-case struggle between what I want and what the community thinks I should be doing - that's counter-productive and reduces proprietary container use to proprietary projects where no one is holding you accountable for having your own opinions.

My secondary concern is I think the proposed container format is very brittle and informal - it's basically a bunch of untyped data-structures, conventions and rules, with no means of checking or enforcing data-types or rules, e.g. a lot of hidden complexity behind a seemingly simple interface. Basically, the only thing you can enforce, is getServices() must return an array.

And I understand that this design accommodates caching/compiling containers, but there's a lot less that can go wrong with a simple list of entry-names.

I'm hoping we can find some middle ground - a safe, practical way to share entries, without erasing the differences between containers.

Can I bring the subject to the PHP-FIG mailing list now, or do you need some more time to think or write prototypes on your idea first?

I don't have time to come up with another example right now, so go ahead - hopefully it won't be all too theoretical and abstract to make sense of it...

[mindplay-dk]

@moufmouf Getting to work on this now, just reading through these last few comments.

Embracing interface segregation and SRP, maybe we should avoid inheriting ContainerInterface and instead design with composition in mind

Agreed. Although it might be weird for containers that are providers too (a getContainer method that returns $this might seem fishy).

On this subject, it just occurs to me - if implementing both interfaces is "weird" for containers that are also providers, that's not the only option: they don't have to implement both interfaces in a single class.

In fact, for almost any immutable container that comes with a mutable factory (as is common) it's likely going to make more sense for the factory to implement the service-provider interface - this way, it can defer the creation of the container itself until the first component is extracted.

For mutable containers, a single class implementing both container and provider is also not the only approach - for mutable containers, it's likely going to make just as much sense to defer the creation of the container until it's needed, so a separate class (a container "proxy" implementing the provider interface) is likely going to make just as much sense for mutable as for immutable containers.

Anyhow, I'm going to get cracking at this now and see where it goes :-)

[mindplay-dk]

@moufmouf I took a closer look at what you suggested about breaking down the provider into two interfaces, and I've decided not to go this route, for several reasons:

1. It isn't necessary: as noted in my last comment, you can already choose to break down things further, should you wish to, e.g. by implementing the service-provider interface as a completely separate class, e.g. an adapter of sorts.

2. It blurs the conceptual boundaries: the listed entries and the provided container are very closely related, in fact they're completely co-dependent - that is, the entry-list has to be the list of entries for the matching container, it can't be just any list of entries and any container; having two different facets for that makes it non-obvious how those are related.

3. The service-registry interface would look very strange then... registerProvider(ServiceListInterface $list, ServiceProviderInterface $provider) probably, but per (2) there is nothing about this contract that guarantees you're getting a matching list and provider.

I'll proceed with the new interfaces I proposed, and we'll see how this pans out both for a mutable and immutable container in practice.

While updating the interfaces and documentation, I've come across one new question: when you register a provider, if an entry in the list of identifiers collides with an existing entry in the receiving container/factory/builder, what should we do?

My first inclination was to throw an exception, because it shouldn't happen - providers should provide for a domain, which should mean they're providing for a given namespace, which should automatically mean collisions don't happen.

If a collision does happen, say, if someone attempted to register two different providers for, say, a Twig engine environment, I'd say that's an error on the client's part - maybe you had some custom registrations for something in the Twig-domain in your project and decided to use a provider instead, clearly you should update/remove any existing bootstrapping/configuration for that domain.

The other option is "last in wins", which is probably consistent with the way individual registrations work in most existing containers - I'm just not sure that's a good argument, because providers don't provide "bits and pieces" the way registrations do in individual containers, they provide related services and bootstrapping for an entire domain.

At least, that was my assumption. I suspect you may have other ambitions, or at least the provider-concept you've been working on until seems to have been aimed at normalizing/unifying the way "bits and pieces" get registered... What I keep thinking on this subject, is that the details of how an individual domain gets bootstrapped, is an already-solved problem - individual containers do that just fine, with many different proprietary features.

I view the problem a bit differently - like, okay, we solved that problem, lets move on to the next problem, of making these domains and environments, which already exist, combine and work together at scale.

I know that's not the original problem as you formulated it, but it's a potentially much simpler problem, the solution to which doesn't need to be "at odds" with existing containers, e.g. doesn't compete with proprietary container formats, because it addresses the integration issue at a much higher level, outside the scope of the problem domain in which existing containers operate: interoperability rather than standardization of different container implementations.

So, as you can tell, I already have a pretty clear opinion about this issue, but I'm pretty sure it would have come up in discussions when you see the direction in which this new proposal is going, so I figured I'd address it up front.

More soon... :-)

[mindplay-dk]

Alright, here we go.

First off, here's the updated spec and interfaces. Most of the changes have already been discussed above.

This iteration has been tagged 0.2.0 as these were breaking changes.

I ported mindplay/unbox to this revision - there are notes in the commit-log, and you can review the changes vs the first version or changes vs the original unbox, whatever makes more sense to you.

Next, I ported mnapoli/simplex, and there is not much to comment on here - again, you can review the changes vs the first version or full changes vs the original simplex if you prefer.

As for changes to twig-universal-module, this consists merely of updating dependencies but the structure and approach is otherwise the same as for the previous revision, so the notes and commit-logs from before apply if you need to review them again - but the structure of this provider really has little to do with this specification in the first place, because pimple/simplex doesn't really have a provider interface/concept of it's own, so it's really just my personal idea of how to do it anyway.

My experience so far doing this, is it looks appealing. Though, as discussed, it may be addressing a somewhat different problem from the design that you were proposing - it operates at a very different level. Dare I say it, but it's possible that your proposal and this one could actually co-exist, because they are in deed solving different problems - although I am increasingly of the opinion that the problem you're solving by standardizing low-level provider formats is not a problem that needs to be solved, if we have the high-level facets for interoperability between different container-implementations, and especially if with these changes we can address the issues around having multiple cooperative containers.

So, we have the two basic implementations for mutable and immutable containers, and the changes to the interfaces seemed to cause no problems for either of those - so perhaps an implementation of these interfaces for one of the caching or compiled containers would be the next relevant thing - to see if this revised pattern will indeed address those concerns.

I spent most of the day implementing this, and have no experience using any of the compiled or caching containers, so if somebody else wants to step up, that would be great.

If not, perhaps someone with experience with several of those container could point me in the right direction: which container-implementation would be the most relevant for such as experiment, or which one poses the greater challenge?

I may be able to find more time this weekend to mash on that.

Alright, well, I look forward to hear what you think so far.

[moufmouf]

Hey Rasmus,

Thanks for the feedback!

Here are a few leads if you want to look into compiled containers (I'm not a big specialist but this is what I would do). Try to look into Symfony's container. Symfony feature something they call a compiler pass. A compiler pass is essentially a callback that is called when building the container. It lets you access all services definitions (an object describing how the service will be built) and let's you register additional definitions. More here: http://symfony.com/doc/current/components/dependency_injection/compilation.html#components-di-compiler-pass

In your case, I would write a compiler pass that

• injects each service provider as a service (using http://symfony.com/doc/current/service_container/synthetic_services.html)
• scans your provider (calling ServiceProviderInterface::listIdentifiers) and that for each service, adds a "Factory" definition that calls the "get" method of the service provider (you can do this since the service provider is now a service).

A bit tricky but it would work. I did something similar for container-interop/service-provider: https://github.com/thecodingmachine/service-provider-bridge-bundle (code might be a bit rough on the edges: it's a prototype and I'm no Symfony expert).

In the end, I'm pretty sure you can make this work!

While updating the interfaces and documentation, I've come across one new question: when you register a provider, if an entry in the list of identifiers collides with an existing entry in the receiving container/factory/builder, what should we do?

The current idea here is that "last wins". This is needed to be coherent with the way a service can extend another one:

    public function getServices()
    {
        return [
            'my_service' => function(ContainerInterface $container, callable $getPrevious = null) {
                $dependency = $container->get('my_other_service');
                return new MyService($dependency);
            }
        ];
    }

The previously configured service is passed in parameter of the "getPrevious" parameter (actually, it's a callable that generates the service). If 2 service-providers declare the same service, it is assumed the second one will "extend" the service defined by the first service-provider. The second service-provider can decide to simply ignore the $getPrevious parameter. In this case, it simply overrides the service.

There are of course other ways to look at this. There is an open PR to split the interface in 2 methods: one that create services and one that extends services. In this case, we could start thinking whether overriding is allowed or not. Also, one of the issues I have right now is that when registering services (in the getServices method), we have no way to know if the service we are registering already exists or not. We might want to change the signature to getServices(ContainerInterface $container) but it comes with its own set of challenges (in particular, a call to get would fail for objects).

My experience so far doing this, is it looks appealing. Though, as discussed, it may be addressing a somewhat different problem from the design that you were proposing - it operates at a very different level.

I came to realize something weird a few days ago. Those two proposals are not as different as they seem. At least, I can easily write a container-interop/service-provider service provider that take one of your Provider in parameter! Look how easy it is:

class Bridge implements Interop\Container\ServiceProviderInterface {
    public function __construct(RasmusProviderInterface $provider) {
        $this->provider = $provider;
    }

    public function getServices()
    {
        $factories = [];
        foreach ($this->provider->listIdentifiers() as $id) {
            $factories[$id] = function() {
                return $this->provider->getContainer()->get($id);
            }
        }
        return $factories;
    }
}

I'm not sure this is of any help here, but kind of fun to see we can make bridges.

I have a number of other comments to make but I'm running out of time here. Will happily continue this discussion next week.

[mindplay-dk]

Hmm, I don't think I have the energy to start digging into the compiler feature of the Symfony DI container now - it sounds really complicated. Maybe later.

With regards to extending and overriding - in my opinion, when we start going into those features, we begin to supplant the proprietary features of DI containers that already solve these problems in my different ways.

I mean, I get it - it's an alluring pursuit, if you're working from the idea of cramming everything into a single container, but my perspective is that we don't need to do that, and that, even if we could do that, it's not necessarily a good thing.

When you have domains or subsystems in isolated containers, these will typically align with a namespace. Twig has a namespace, Aura.Router has a namespace, and so on. In your project, you're going to bootstrap each of these namespaces with a single provider, once. There's nothing to extend or override - that's taken care of internally in the provider, if needed, using proprietary facilities provided by the DI container used by that provider, which you can regard as an implementation detail. Or, in some cases, maybe the vendor finds it more suitable to ship a custom implementation of the provider-interface, e.g. a Twig module which only cares about Twig-proprietary components, such as Twig-extensions/functions/tags/filters, all of which are required to bootstrap a Twig_Environment, which may be the only (or one of a few) interesting components that are actually exposed to a project via the provider-interface.

Anyway, I'm going to leave this here for a bit and let it simmer, as I'm running a bit of a temperature from a pesky head cold, so probably not the best time to start digging into the details of Symfony DI ;-)

(I hope to hear from @mnapoli on these subjects as well at some point.)

[moufmouf]

To be perfectly clear, I don't really care whether we use one container or many. What I really want is to be sure that whatever solution we choose, I can use "autodiscovery" to automatically add all the service providers together in a way that makes sense "by default".

Ideally, you simply require your package in Composer and the framework you use will automatically detect the service provider and plug it to your container. Whether the service provider is actually a container or not does not make any difference to me. The only thing that matters is that I want to be able to write a framework that makes these steps automatic with good defaults. Of course, there must be a way to disable some service providers and customize things.

There is a strong trend in the PHP community towards autodiscovery currently. Puli started the trend. I've tried to push things forward with Discovery (https://thecodingmachine.github.io/discovery/) and now Symfony/flex (to be released with Symfony 4 this fall) will completely embrace this concept (https://medium.com/@fabpot/symfony-4-automate-your-workflow-fbbf609b5a1d).

A strongly believe the standard we will design must be "autodiscovery" friendly (ie it must be "easy"). And of course, it must also be possible for power users to tweak / customize the service providers (the "simple" part you are promoting)

In the end, if we can find a way to have a package "auto-wire" the container it is providing in the main application container, I would feel way more comfortable about your proposal.

Anyway, I'm going to leave this here for a bit and let it simmer, as I'm running a bit of a temperature from a pesky head cold, so probably not the best time to start digging into the details of Symfony DI ;-)

Have a good rest!

[shochdoerfer]

I might be missing a point but how would I be able to share dependencies in this approach? Like defining and configuring a logger instance which then gets used in one of the dependencies.

In an ideal world we need to merge (and manage aka change/modify) all defined dependencies from all the packages installed by composer in one location/container. Ideally without any interaction from the developer. But in case of conflicts e.g. identifier clashes "customizations" are needed.

[mindplay-dk]

There is a strong trend in the PHP community towards autodiscovery currently

There has always been a strong trend in the PHP community towards auto-discovering and auto-configuring things. "Install it and run", as opposed to writing a line of code and explicitly registering a provider.

Doesn't fit with my ideals about programming, but let's not get into that here.

Is there any reason you think these interfaces might not permit what you'd like to do?

[mindplay-dk]

Working through some use-cases (in a private project) last night, I was starting to reluctantly think that there may in fact be a use-case for what you're proposing (or something similar) as well as what I'm proposing.

I've said before that I was trying to reformulate the problem, but I am starting to have doubts - perhaps I haven't reformulated the problem, so much as identified a new problem.

I'm starting to think that there are two classes of use-cases.

For lack of another term, I will continue to refer to both as "providers", since they both provide stuff - but I'm going to qualify them as either "invasive" or "isolated" providers.

---

Invasive Providers

An invasive provider makes modifications to your container. Modifications such as registering new services and making modifications to existing services, e.g. by extending or overriding existing registrations.

One use-case is, you want somebody's provider to provide "core" bootstrapping on which you're going to build on top - or you may want somebody's provider to modify your bootstrapping, or bootstrapping provided by another provider. For example, one provider may create a working Twig environment, and another provider may configure that environment by adding additional plug-ins/features.

This is what you're proposing.

Isolated Providers

An isolated provider does not modify or override any existing services in your container - it may have a container of it's own, or it may manage it's own internal dependencies in some other way, but it will only ever register new services.

It will expose only services that are central to it's domain. If it has common internal services, such as PSR logger or cache, that aren't it's primary domain, it won't try to expose those services - those are regarded as implementation details, and what it exposes will usually be components in a namespace that belongs to it's domain.

This is what I've been proposing.

---

So, thinking about this, it occurs to me that isolated providers could in fact be implemented using the same mechanism as invasive providers - it's merely a matter of only making clean registrations, and so, arguably, if we must have a mechanism and standard for invasive providers, a separate mechanism for isolated providers is likely redundant.

With that said, creating isolated providers requires excellent understanding of the issues and differences - it requires discipline and restraint, and a good understanding of the consequences and differences of choosing to expose, or not expose, each individual dependency within a module.

This is why I'm reluctantly arriving at this conclusion.

I'm concerned that most people will barrel ahead and register, modify and override things in your container without any real understanding of the consequences. "I need a cache, so I'll just register one" - whoops, there goes the cache you had registered and configured for your own uses.

Even if the vendor has complete and total discipline, there is no way the vendor can know anything about what is or isn't already bootstrapped in your container.

That is, whether or not extending or overriding a given services is meaningful, or even possible, is completely dependent on (1) which other providers and project-custom registrations have been made, and (2) the order in which providers were added or other custom registrations were made.

In other words, the entire concept hinges on side-effects, assumptions or total knowledge of every relevant prior registration made.

It's a simple dependency problem, really.

We engineered DI containers as a means of solving a dependency problem - but with the introduction of providers, in any form, we now have a new dependency problem, because providers are dependencies too.

It seems we've recreated the original problem at a higher level - only this time, it's even harder to deal with, because there is no type information or static relationships (e.g. type-hints in constructor arguments, return-types, etc.) as evidence of these dependencies. You can't tell if a provider depends on another provider, except by reading through all of it's source-code and figuring out exactly what it does.

The situation is different with the high-level providers I've been proposing - these do not have dependencies on existing registrations, and they do not modify or replace your existing registrations.

Using @shochdoerfer's question to illustrate the problem:

I might be missing a point but how would I be able to share dependencies in this approach? Like defining and configuring a logger instance which then gets used in one of the dependencies.

How do you provide a database-connection, a cache, or a logger to a provider?

With an invasive provider, you simply make these registrations first, cross your fingers and hope you did it right - so that, by the time a component from that provider needs that cache or logger, hopefully it will find it. But there is no evidence of this dependency, anywhere, except maybe in documentation, or if you read through the provider's code. Nothing enforces or guarantees that these dependencies are satisfied.

With an isolated provider, which may be using a DI container internally, it would need to have those dependencies declared in it's constructor - so it can register those dependencies internally. We use dependency injection, and the requirements of the provider are clear and visible. There's a firm contract: if you want this provider to provide A, you have to provide B - the order and presence of any dependencies are guaranteed by the fact that you can't create or register such a provider without explicitly satisfying the dependencies of that provider.

So far so good, but we then have a new problem: if a provider depends on, say, a database-connection, a cache, and a logger, creating all of those dependencies up front is expensive, so now we have a new and unacceptable performance problem.

I don't have a ready answer to that issue, so I'm leaving this here for you to ponder.

Just one idea comes to mind. In mindplay/unbox, I use a "boxed reference" as a means of working around similar issues of creating components "too soon".

So imagine that there was a standardized interface for something similar. You would now be able to pass around references to dependencies without actually unboxing them. You could pass them between containers, which might create interesting new possibilities for interoperability, and you could use them as constructor-arguments to isolated providers that have external dependencies.

Ideally, this type would be generic - such that a provider could ask for BoxedReference<PDO>, but we don't have generics in PHP, so we couldn't make this type-safe. But it's the next-best thing: at least, we can formally state that this provider has dependencies, forcing someone to at least provide the boxed references to something - a developer will need to take position and deal with the dependencies of the provider in order to use it, which is surely safer than just fingers crossed ~ registration order and side-effects.

Alright, guys, that's all I got at this time. Hopefully more food for thought :-)

[mindplay-dk]

I'm dropping this pursuit - I'm no longer sure where I was going with this, and I've got some good traction on the original service provider PSR this weekend.

Closing.