dynamic "code" loading

[lsmith77]

This ticket will likely spawn other tickets. The idea is that this ticket will group them and serve as an entry point to people joining the discussion.

The goal is to make it possible for Symfony2 component using applications to dynamically add code to the application enabling people to for example download modules/Bundles from the web and enabling them. This will usually mean regenerating the DIC and routing caches.

/cc @crell @drak @Tobion @lolautruche @dmitriysoroka

[pjedrzejewski]

Two years ago I started a bundle for plugins (well, bundles) with enable/disable option though web interface. The implementation is at least clunky, but it worked and you could install / uninstall plugins. This involved regenerating the DIC and clearing some parts of cache.

In coming months I'd like to approach this feature again (probably from scratch), some random thoughts I had during brainstorming about it:

• Regenerating the whole DIC via click in the administration panel is problematic, because if the plugin contains even a simple typo in the services definition, the whole app is broken.
• I thought about having a separate container for plugins, which would have access to the main container, but not sure if this really solves any problems from the point 1.
• Another possibility is to have a fully event based plugin system, as listeners can be registered during the runtime. This is more error-proof than regenerating whole container, but much less powerful.
• For me, most powerful engine will work on top of the bundles architecture, I believe we should not create any new module system and work on Symfony bundles mainly.
• Very random idea = Some "rescue console" written in Cilex to remove broken plugins/bundles from the kernel and regenerating the DIC, worst solution in terms of user-friendliness.
• Another option would be to generate a "test" container during plugin install, if it causes any errors, do not enable the module and give some feedback to the user.
• Some code from Composer could be reused to download/unzip the plugins and/or their dependencies.
• Would be awesome if the available plugins/modules list could use multiple different sources, for example display modules unpacked to particular director or load them via some API. (e.g. from "app market")

[merk]

I always thought the best approach here would be to have a separate admin front controller that handles registration of bundles and rebuilding of the container- something that didn't actually depend on the container being able to compile. (or at the very least, was much more resistant to failure).

That meant if the container did fail to compile for whatever reason you still had an interface to deal with it.

[sstok]

About generating the container, I think (performance wise) it would be best to just build the container and then validate that. https://github.com/matthiasnoback/symfony-service-definition-validator is a great example on how you can validate the container before dumping.

[johnkary]

We built dynamic bundle discovery into the current Symfony SE project I've been working on most recently. We call each dynamically discovered bundle a "module" and each module is made to operate fully independent of other dynamically discovered modules. But we have a custom layer of code (i.e. our custom framework code) between Symfony SE and the module bundles. This layer ties the modules together, provides common functionality between them, and can perform cross-module functionality for things like search. So any number of modules could be enabled for any given install of our core application.

Some random thoughts/notes from our implementation:

1. We created a custom Kernel that discovers bundles/modules in registerBundles(). So AppKernel extends our custom Kernel. AppKernel loads all the explicitly enabled bundles like normal, then calls parent::registerBundles() to our custom Kernel to load the dynamic ones. We used conventions to discover ours: each module has a *Bundle directory name that is the root dir, and to designate it as a module we checked for the existence of a specifically-named file at a specific path within the bundle. All of the registerBundles() results and list of enabled bundles are cached as part of compiling the Container so scanning the filesystem with the Finder component wasn't a big deal for performance.
2. Each module has a unique "module id" string that prefixes things like route names, service ids, roles, etc. to ensure they're unique within the greater framework. Because each module doesn't depend on other modules we chose to ensure uniqueness that way. Our Bundle objects serve as the authoritative source for that kind of data by implementing a custom interface (e.g. ModuleInterface) that expose methods with their custom data.
3. In our custom cross-module code that lives on top of Symfony SE we use a service ModuleFinder that is knowledgeable of all registered modules. We use it in a few places in our framework code that needs to use module metadata or allow the module to expose other behavior as part of its ModuleInterface. We built ModuleFinder to have AppKernel injected into it to iterate over the list of bundles and check for if ($bundle implements ModuleInterface) ... I'm not thrilled depending on AppKernel but it works for now.
4. Each module makes heavy use of tagged services and event listeners that allow modules to register services with our custom framework code or with other Symfony/third-party bundles.
5. Each module also exposes its own role system and hierarchy too. We did some really cool work to register custom role hierarchies with the Security component and still allow our "admin" level users (e.g. cross-module administrators of the organization) to assume module-level roles, and still allow module admins to administer roles for users of their module.

[docteurklein]

Here is my implementation of "auto discovered" bundles: https://github.com/KnpLabs/symfony-light/commit/72187a26383bb1472ceda84899057d4299be80bb

It was basically a dumb Finder based kernel, that register every bundle it finds in given directories.

[redkite-labs]

Hi, I've taken this approach to automatically load bundles for my CMS https://github.com/redkite-labs/BootstrapBundle.

[hice3000]

@docteurklein Any chance to have your SmartKernel submitted as PR?

[michaelcullum]

:+1: I could see this could be quite helpful for projects like phpBB and Drupal.

[docteurklein]

@hice3000 I don't think so. Until it's considered as a good approach and people would like me to do it :) It was just an experimental, hacky solution.

[docteurklein]

I have in mind a problem that bugged me: bundle constructor requirements. Some bundles require to pass the kernel as argument, some other could have other requirements.

[docteurklein]

I just cross ref something I found about that: #6082

[docteurklein]

I also have another idea in mind, which is to take the approach made by so many nix tools out there: the `conf.d/` approach (if you see what I mean. If not, http://unix.stackexchange.com/a/4040).

Composer, or any other tool, would leave a file in a specific folder (f.e: app/bundles/), that contain (php?) code/config to instanciate the bundle:

cat app/bundles/framework.php
return new FrameworkBundle($this); // access to kernel via $this

Then, any bundle maintainer would have to create a post(Install|up|remove) composer script that manipulates this file.

Then, the kernel would require any file found in this folder and append the returned value (if instanceof BundleInterface) to the bundles array.

[lolautruche]

Hi

I think the right answer is somewhere between @docteurklein's SmartKernel solution and BootstrapBundle by @redkite-labs . Problem with BootstrapBundle is that it requires an autoloader.json file and there doesn't seem to be a cache layer, which is problematic since it uses Reflection (if there is some cache, I might just be unable to find it so I apologize by advance :blush:).

The great thing with SmartKernel is that is kind of universal and can hence work with any bundle out-there. It could be a fallback solution if no autoloader.json can be found.

My 2 cents

[merk]

The installation process will need to take into account required configuration as well. Consider FOSUserBundle - you need to create some entities somewhere, map them, add those entities to the app/config/config.yml file, which cant be done automatically.

[ghost]

We solved this for Zikula in the following way allowing bundles to become modules which can be installed, and deactivated and still allow the kernel to be recompiled. If a module is disabled it will be dynamically removed from the kernel. This is done because we have AbstractModule which extends Bundle. The module has state (installed, etc) and will disable the DependencyInjection extension for the module-bundle if the module is not in an installed state.

References:

https://github.com/zikula/core/blob/1.3/src/app/ZikulaKernel.php#L70 https://github.com/zikula/core/tree/1.3/src/lib/Zikula/Bundle/CoreBundle/Bundle https://github.com/zikula/core/blob/1.3/src/lib/Zikula/Bundle/CoreBundle/HttpKernel/ZikulaKernel.php#L381 (the final magic happens here).

Module have a composer.json which gives certain metadata, you can see it here: https://github.com/zikula/core/blob/1.3/UPGRADE-1.3.6.md#module-composerjson

This is still a work in progress and POC, but it is very much working. There is a lot to do, including adding a similar layer to override configurations and we've already begun integrating https://github.com/matthiasnoback/symfony-service-definition-validator which will validate a module's service configurations before allowing the module to be installed and prevent bricking the system if a module-bundle has invalid service configurations.

I think the problem however for Symfony core is that Bundles are first class citizens and people are wanting to use them like second class citizens (like modules etc). This is at fundamental discord with what Bundles are about.

Symfony was designed as a low level framework where it is expected you have access to the console, of course, web applications are generally designed as systems without access to the console. That is why second class citizens work well but of course they wont have access to modify the DIC. I'm therefor suspicious of any Symfony core implementation that changes this.

I hope you can glean some ideas from what we are doing in Zikula and I would be happy to answer any questions you have.

[hice3000]

So why don't take the concept from @docteurklein . Every bundle developer, who wants to use the loading, places a Loader class in his bundle's root, which'll take care of registering the bundle, routing, config ... .

Providing a php class would also be great for all third party applications, which can't be extended via the DIC (e.g. Assetic asset's). The loader may be accessed by the kernel, so everyone can check whether it implement's a custom method.

Would be great, without breaking the BC or forcing everybody to change their bundle's structure.

[stof]

@hice3000 at least for the routing, I see an issue here: the order of routes is meaningful. currently, the application controls it when importing files. If they are registered automatically, it is not under the app control anymore.

[ghost]

@stoff what about having some kind of weighting system like events then bundles can they can be ordered.

[lsmith77]

well it quickly gets quite complex. order matters indeed but in most cases one will likely just add a new subpath of routes. for the case where one adds routes that overlap, maybe this is something where we need an api to order the routes of such Bundles, but the actual behavior needs to be configured by a human via some UI.

[hice3000]

@drak this won't solve the problem @stof told, coz the bundles will still define the priority by themselves, so the app can't influence the routes order. But I'm sure we'll find a solution for that.

[Crell]

I've asked @chx to weigh in here for Drupal. There's some touchy trickiness needed around Apache writing code that it can then execute, for security reasons. He wrote the Drupal solution, so I'll let him provide the details.

[DimaSoroka]

This is very hot topic for Oro Platform team and we are working on solution with following requirements:

• Bundle that will allow install, unistall, enable and disable packages without runtime dependency on them
• Package is a composer package that can include one or multiple bundles
• Convention for bundles registration (considering configuration file and PHP interface options). Bundles registration includes dependency definition and helps to solve initialization order issue

This is very close to @drak solution, looks as it is reasonable to generalize it.

What would be the best way to do this?

[chx]

So, one of the security concerns of the Drupal project -- which might or might not be a concern here -- is: we are operating in a very heterogenous environment with an equally heterogenous codebase and we have little control over what the users run and on what hosting environment. One of the biggest actual security threats I perceive these years are scripts that allow for uploading, say, images but they read the resulting filename-directory from $_GET. Such scripts often ship with third party libraries, often unused, forgotten. This has lead to pwning some really, realy high profile websites. So, when writing our generated PHP files we needed to be sure that an attacker can't use such a script to write a PHP file that Drupal would include. We have written (in this case, lead by me) some really tricky code to achieve this in MTimeProtectedFileStorage -- but that code won't help Symfony because it's GPL.

[jameshalsall]

ping @markwilson

[cordoval]

for this Symfony Plugin Component I think so far i have seen discussed:

• downloading handling similar to composer (I wanted to add here a reference https://github.com/composer/installers) and multiple sources api etc
• routing and other settings, priority and ordering
• caching related
• enabling/disabling UI problems
• regenerating DIC and problems with wrong configurations
• security issues
• maybe some i am missing

my 2 cents is that I see this as a component that is not dependent on symfony2, that it can actually allow someone building an app that is not based on symfony2 framework, but based on this symfony2 component to enable/disable code modules. Notice this can be also applied to bundles, but that is just another use case. So talking about reusage of the bundle appkernel way of loading things is not necessarily compatible.

I agree with @drak in that reusing the bundle mentality for modules is not the best. However the temptation is great to reuse this bundle infrastructure to save on rewriting several issues already solved for the bundle system. :baby:

[DimaSoroka]

Great summary, @cordoval.

You are right, we have two items here:

• Packages/modules management UI based on composer - generic functionality without dependency on Symfony.
• Application maintenance and bundles management - requires a change to standard application AppKernel to support bundles activation (including routing) but can be handled with another base base kernel class.

This project can be handled independently of symfony framework as well as can be a very good addition to it. @orocrm we are planning to develop this as part of the platform and share as a standalone bundles later on.

[brpaz]

I will need something similar in the application i am working on. I am working on as SaaS app where each user can have specific code. (like a complete bundle or just some custom logic inside some controller). I need a way of loading that code only for that specific user so I dont have conflicts (like routes for example) and also to be able to overide funcionality of the core application bundles just for that user without affecting the others.

[docteurklein]

@brpaz You are doing multitenancy. It's not really related to this issue.

[liuggio]

I think this task could introduce risks, the same risks of a new deploy.

Is this task only for develop env? Thinking about it in production, is scaring.

download modules/Bundles from the web: What about multi instances?

• Recreate the app/cache is not easy even with the link to the new folder.
• Recreate the Autoloader, classes
• What if the new bundle/module has some packages to install as dep?
• Bytecode should be reloaded?
• What about external caches eg.databases
• What about CDN

Is possible to reduce the task? this seems to big to me, and very related to the user domain.

Why not an external service, seems more a DevOps+Dependency related problem than a framework problem.

But are just my opinions.

@liuggio

[brpaz]

@docteurklein. May be I have a wrong idea about multitenancy but from what I knowi I would have a separate database and code base for each client, which is not the case. The database and code is shared across all the clients. I just want to extend some base funcionality by adding a client specific bundle which needs to be loaded at runtime.(Well, the specific client tables will go to a different database) but the code base will be the same.

[jameshalsall]

@liuggio I think as long as the solution provided entry points to allow user space code to hook into the update/install process then those things should all be attainable.

I think a better solution would be to have a completely separate kernel that has no bundles/modules by default. All of your core bundles required for your app to function properly would be managed by the main kernel (app/AppKernel.php in standard distribution). Then you aren't rebuilding the entire kernel and the risk is a lot lower than if you were updating core application dependencies.

Possibly using an event system we could allow user space code to execute before certain key points of the process, allowing for things like enabling a maintenance mode in their application, gracefully logging users out of their application, database backups etc.

It's not something that could be created to encompass all scenarios, and providing events would be a much better way of guaranteeing wider compatibility.

[chx]

If there's still interest, you have my permission to MIT license the phpstorage component. I am definitely not happy with MIT licensing my code but let's do it.

Someone needs to do the git archeology to see who else contributed to this code but if I surrendered then probably so will everyone else.

[linaori]

I'm not sure if this is useful for this discussion, but I know that bolt cms 2 uses composer to install plugins/modules. I'm sure that that they have some sort of logic to load them dynamically and refresh the cache (I never used it).

[javiereguiluz]

Closing it in favor of #6082, which is older and it's asking the same feature.