Why not use the vendor/autoload.php

Processwire v3 have a composer.json , thank you for it.

Why not make use of the vendor/autoload.php ?

In that case people can import other 3rd party packagist library and don't need to worry about autoloading. And you also don't need to require

if(!class_exists("ProcessWire", false)) require_once("$rootPath/wire/core/ProcessWire.php");

in that case. Composer will do the work for you.

I am all for keeping a

$composer_autoloader_file = __DIR__ . '/vendor/autoload.php';
if (file_exists($composer_autoloader_file)) {
    require $composer_autoloader_file;
}

This will really help in the future.

Thanks @harikt I've been meaning to get in touch with you about this. I don't use composer everyday and thus don't consider myself an expert on it, but know that you are. Our composer support is only partially there at present. We could use your help! I was wondering if you might help us complete this part of the 3.x core?

Here's our context, which is probably a little different from most projects, so our approach to composer is I'm guessing also going to be a little different. Composer isn't a common need in the ProcessWire world, because most using ProcessWire are using it to develop websites just with PW, and rarely bring in code from other projects, other than existing PW modules. Most of our users are web developers and designers rather than full-time PHP developers. That being said, we'd like to open the opportunities for those that could benefit from composer in PW and attract more full-time PHP developers, especially on the framework side of PW.

What we do want:

We do want our system to work well with it when someone is already using composer or wanting to make use of it in PW.
We want people to be able to easily pull PW into another project using composer.
We want PW to be available on Packagist once 3.x is released (I already have an account).
We want to have clear documentation about how one uses composer in PW, when they should use it, when they shouldn't, and what the benefits to them will be when they do use it.
We want to have a couple tangible examples of using PW with composer.

What we don't want:

We don't want to make our system built around composer, or be dependent upon it.
We don't want PW's core or modules system to use composer's autoloader. PW's autoloader is already optimized to its context, which supports PSR-4, but doesn't require it.
We don't want to inherit any overhead from composer and its autoloader, except where necessary, unless the developer has decided they want it as part of their PW project. Basically, we don't want to introduce code that implies "everyone is using this."
We aren't looking to replace things like existing require_once() calls.

Regarding the last item above: I prefer to use autoloaders only for classes that "might" be loaded on a given request. For classes that are "always" loaded I prefer to bypass autoloaders entirely in order to reduce overhead. For instance, there's a reason why we have this, because it is just plain faster than letting it all that pass through the autoloader (there is a measurable difference in speed).

Another few points to mention:

PW3 supports multi-instance, so that one can have multiple instances of PW (connected to different databases/sites) available to a master instance. In this context, we have to be careful about what gets loaded and from where, something that PW's autoloader manages, and we have to be careful that Composers autoloader doesn't interfere with.
PW3's core doesn't use PSR-4 at present, and doesn't intend to, as its modules system already solves the same thing in a manner more appropriate for PW's context and structure. However, PW3's autoloader does implement support for PSR-4 already, for those that want to use it in 3rd party modules or in their site. But I've not hand the opportunity to test it outside of the core context.
You'd proposed composer installation support for modules in the past. I'm not sure we want to go there, since this is something that PW already handles and we don't want module authors to be required to maintain composer-specific data for their modules, unless they want to. But this is something I'd like to look closer at in the future. In the short term, I'd like to focus on the things mentioned in the "what we do want" section above.

Okay that's a wall of text, and probably a few redundant points, but I think is everything I was already planning to email to you next week. Since you've gotten to me first, I figured I'd write it all out now. :) Given the above and your recommendation, it sounds like the first thing we should do is add the code block you indicated to our /index.php. Can you confirm? What other recommendations do you have? Thanks for your help.

Hi @ryancramerdesign ,

Thank you for getting in touch with me so quickly.

First feeling first ..

There is lots of text over here, which I am unsure I could address all with my limited knowledge in composer. But I will try to help you with what I know.

Here's our context, which is probably a little different from most projects, so our approach to composer is I'm guessing also going to be a little different. Composer isn't a common need in the ProcessWire world, because most using ProcessWire are using it to develop websites just with PW, and rarely bring in code from other projects, other than existing PW modules. Most of our users are web developers and designers rather than full-time PHP developers. That being said, we'd like to open the opportunities for those that could benefit from composer in PW and attract more full-time PHP developers, especially on the framework side of PW.

I do understand, and I am trying to bring solution that can help both sides.

We do want our system to work well with it when someone is already using composer or wanting to make use of it in PW.

Great :-) . This is already possible, and the proposal of adding the vendor/autoload.php is in case processwire ask to replace index.php .

We want people to be able to easily pull PW into another project using composer.

I am not sure about this, we will see when using the same.

We want PW to be available on Packagist once 3.x is released (I already have an account).

Nice :-) .

We want to have clear documentation about how one uses composer in PW, when they should use it, when they shouldn't, and what the benefits to them will be when they do use it.

It is all about your dependencies getting updated when ever a security / bug fix happens. But I do understand your concern. Probably this page may address to solve regarding it ?

We want to have a couple tangible examples of using PW with composer.

I am not sure about this part, what to do to make the examples.

What we don't want:

We don't want to make our system built around composer, or be dependent upon it.

sure. I am not sure whether you know a command that can help to create quickly a project .

composer create-project -s dev processwire/processwire next-project

This command actually downloads the dev version of processwire/processwire ( assuming you have registered this name on packagist ). If there is stable you don't need the -s flag.

But I do agree your concern and support the case in case if they are not proficient in PHP or composer.

ie why I told regarding if (file_exists()) ;-) .

We don't want PW's core or modules system to use composer's autoloader. PW's autoloader is already optimized to its context, which supports PSR-4, but doesn't require it.

/me nods

We don't want to inherit any overhead from composer and its autoloader, except where necessary, unless the developer has decided they want it as part of their PW project. Basically, we don't want to introduce code that implies "everyone is using this." We aren't looking to replace things like existing require_once() calls.

/me nods

Regarding the last item above: I prefer to use autoloaders only for classes that "might" be loaded on a given request. For classes that are "always" loaded I prefer to bypass autoloaders entirely in order to reduce overhead. For instance, there's a reason why we have this, because it is just plain faster than letting it all that pass through the autoloader (there is a measurable difference in speed).

There is a command called composer dumpautoload -o , which creates an optimized autoloader, which is basically an array of class names which is in memory. But if I am correct it don't require the class until the class is called. Anyway that doesn't have any issues with your own loader.

PW3 supports multi-instance, so that one can have multiple instances of PW (connected to different databases/sites) available to a master instance. In this context, we have to be careful about what gets loaded and from where, something that PW's autoloader manages, and we have to be careful that Composers autoloader doesn't interfere with.

As far as the entry point is https://github.com/ryancramerdesign/ProcessWire/blob/515c6e5f601bc4265aa8f43b11310084a2d11cd0/composer.json#L20 composers autoloader has nothing to do with loading the module. it is all loading via processwire itself. Regarding module if you are using something like http://harikt.com/blog/2013/11/16/composer-support-for-processwire-modules/ then also the module is just downloaded via composer on the appropriate place, but autoloading is not done by composer. But if there is a 3'rd party dependency eg : Twig, I want it to get loaded via composer than processwire handling it. Why ? I need to update the bug fix / security updates when ever it happen and I will any how rely on composer. So I wish people don't keep all the Twig in a module, but make use of composer. But that will not break anything on processwire and its loading of modules.

PW3's core doesn't use PSR-4 at present, and doesn't intend to, as its modules system already solves the same thing in a manner more appropriate for PW's context and structure. However, PW3's autoloader does implement support for PSR-4 already, for those that want to use it in 3rd party modules or in their site. But I've not hand the opportunity to test it outside of the core context.

You'd proposed composer installation support for modules in the past.

Oh yes, part-1 and part-2 , but I was disappointed at the time for you didn't supported it that much :( . But I am more happy now that you are considering some sort of support and you are caring the wider audience.

I'm not sure we want to go there, since this is something that PW already handles and we don't want module authors to be required to maintain composer-specific data for their modules, unless they want to. But this is something I'd like to look closer at in the future. In the short term, I'd like to focus on the things mentioned in the "what we do want" section above.

I hope I addressed some of your concerns. If not we can further discuss this, I can ask more in case I don't know about it and get some support to this topic.

Okay that's a wall of text, and probably a few redundant points, but I think is everything I was already planning to email to you next week. Since you've gotten to me first, I figured I'd write it all out now. :)

I am a flattered to hear you have already thought about me. Thank you for considering.

Given the above and your recommendation, it sounds like the first thing we should do is add the code block you indicated to our /index.php. Can you confirm?

Exactly :-) .

What other recommendations do you have?

Currently I don't have anything more. If vendor/autoload.php is there ie all we need I believe so I /others can promote the usage of composer to build modules for processwire with composer support .

Given your comments I do think you may not like to bring the usage of composer too much for modules. https://github.com/harikt/pwmoduleinstaller

But isn't that awesome when someone can just do

composer require ryancramerdesign/textformattervideoembed

or similar?

Thanks for your help.

Always happy to help when I can.

I hope I tried to be neutral and have not moved you / project to make processwire dependent on composer. But in case if 3rd party modules need yes there can be a dependency if you can use wonderful packages outside.

After writing everything I understood I missed something. In the .htaccess we probably need to make sure that the composer.json , composer.lock , vendor/ files are not called directly as in the site , wire folders.

Thank you

Thanks @harikt this is very helpful. I have gone ahead and added support for the composer vendor autoload per your suggestion.

Rather than putting it in the /index.php file, I put it in the ProcessWire::buildConfig() method (in /wire/core/ProcessWire.php). That's because someone might boot PW without using the index.php file. Further, it ensures that in a multi-instance environment, all of the composer autoload files are loaded for each instance, rather than just the master/first instance. Though I'm not 100% certain that's the right way to go, but seems like a good place to start.

I also changed the require() to a require_once() just in case someone has already modified their index.php to include the composer autoload file already.

Since I haven't implemented it exactly like you indicated, do you think that this should all still work according to what you were suggesting? Thanks again for your help here.

Regarding modules and updating via composer, I agree that the examples you mentioned are very cool, and that's definitely a good point about being able to more easily keep a module's dependencies up-to-date. Though identifying when a module has updates, or downloading and installing an update is something that the core can already do. The core also keeps track of dependencies and versions between modules. The ProcessWireUpgrade module also enhances these capabilities. Does composer provide a better and more complete solution for developers? I'm certain it does. But the issue comes down to our audience, many of whom might only have FTP access to their PW installation (no command-line). So PW has to provide these capabilities at the admin/UI side, and some (depending on environment) might be limited to installing such updates via FTP. Though when the server and/or developer has enabled PW to write to the file system, it enables updates to be installed by non-developers, superusers that might not even have FTP access.

When it comes to Composer, the ability to update things is pretty much limited to the developer, as I understand it. In many cases, a PW site may be handed over to a client who has no ability to develop or knowledge of how to access anything other than the PW admin. The original developer might be long gone. As a standardized process for PW, I feel it's important that PW is able to keep track of versions, dependencies and installation of updates or new modules (when enabled) with admin-specific tools, rather than just developer-specific tools.

If we were purely a framework, I think we could completely rely on Composer for handling this. But since we're equal parts framework and CMS, I think it requires that we exist in a world that may not have Composer, while at the same time supporting it when we do have it.

Now some (a smaller part) of our audience actually is purely developer-centric and experienced PHP devs. If there are things we can do to better support them and their needs, I fully support it. If supporting composer for modules is something we can take advantage of and make available to this audience (without requiring it for everyone else) then it's definitely something we should pursue. Perhaps that's one of the next things we should look at in regard to our Composer support.

Hi @ryancramerdesign ,

Can't it be kept in boot.php than in buildConfig ? https://github.com/ryancramerdesign/ProcessWire/blob/19fd0ded383ab2c5b05383a39fae7b1cf0c75053/wire/core/ProcessWire.php#L594 . I am really not sure when will this function get called.

I also changed the require() to a require_once() just in case someone has already modified their index.php to include the composer autoload file already.

Probably you don't need to do it for composers autoloader uses certain hash to create the classes. But all is good :) .

Since I haven't implemented it exactly like you indicated, do you think that this should all still work according to what you were suggesting?

I hope so. I haven't got time to look at 3.x for life is busy with kid and other personal stuffs :-/ . But I like it and love it to be improved :-) .

Thanks again for your help here.

sure :) . Thank you.

Does composer provide a better and more complete solution for developers?

I am not sure what you mean by provide better and more complete solution for developers. Could you please expand? Are you saying some sort of gui ? If so, I am not aware of.

But the issue comes down to our audience, many of whom might only have FTP access to their PW installation (no command-line).

Those people can run composer install / update from the local system and upload all the folders as they are doing. Only thing is it will also have the vendor folder.

So PW has to provide these capabilities at the admin/UI side, and some (depending on environment) might be limited to installing such updates via FTP. Though when the server and/or developer has enabled PW to write to the file system, it enables updates to be installed by non-developers, superusers that might not even have FTP access.

Hm, this is something that probably break. Eg : I have a module say tweet and assume we are relying on twitter php api package. Twitter upgraded the library to 2.0 , bc break, and I upgraded my module to 2.x . Then processwire admin can download the pw module, but the dependency used by my module is not upgraded. It may be still on 1.x . So that will break the upgrade. I don't know what drupal is doing here for they are also relying on composer. Probably I can ping someone to here, who may be able to help with. And we are php community after all :) .

When it comes to Composer, the ability to update things is pretty much limited to the developer, as I understand it. In many cases, a PW site may be handed over to a client who has no ability to develop or knowledge of how to access anything other than the PW admin. The original developer might be long gone. As a standardized process for PW, I feel it's important that PW is able to keep track of versions, dependencies and installation of updates or new modules (when enabled) with admin-specific tools, rather than just developer-specific tools.

I understand your concern.

If we were purely a framework, I think we could completely rely on Composer for handling this. But since we're equal parts framework and CMS, I think it requires that we exist in a world that may not have Composer, while at the same time supporting it when we do have it.

Now some (a smaller part) of our audience actually is purely developer-centric and experienced PHP devs. If there are things we can do to better support them and their needs, I fully support it. If supporting composer for modules is something we can take advantage of and make available to this audience (without requiring it for everyone else) then it's definitely something we should pursue. Perhaps that's one of the next things we should look at in regard to our Composer support.

sure, the only thing they need to do is

1 ) add a composer.json file 2 ) Register on http://packagist.org/ or host something on your own via http://github.com/composer/satis 3 ) Tag your releases

Done :+1: .

Thank you

Can't it be kept in boot.php than in buildConfig ?

It could be, but boot.php is only called for the master instance, not for any other PW instances that might be loaded. Whereas buildConfig() is called for every single instance of PW instantiated. Meaning, if it's in buildConfig() then it would load the vendor autoload for each of the installations, rather than just the first one. When one is using composer and multi-instance PW, there's a good chance they have different things in the /vendor/ dirs for each of the instances. Btw, buildConfig() is called by /index.php, and before the ProcessWire instance is instantiated. Given this, do you still think better to move it to boot.php?

I am not sure what you mean by provide better and more complete solution for developers. Could you please expand? Are you saying some sort of gui ? If so, I am not aware of.

That was my lead-in to the next sentence which stated: "I'm certain it does."

To word it differently, I'm explaining that I'm aware Composer is the definitely best solution when it comes to managing these things from the developer viewpoint. And that we would certainly want to standardize purely on Composer for module installation/management, if our primary audience were PHP devs.

I'm trying to say I understand your interest in Composer when it comes to module install/updates, and I like your ideas and proposals you've put forward here in the past. The reason I've not pursued them yet is because our audience (who are not primarily PHP devs) must be able to upgrade and install modules without developer-specific tools. However, I do think we should look into opening the option (as an alternative) for those like you and me that would like be able to install/upgrade modules with Composer.

Those people can run composer install / update from the local system and upload all the folders as they are doing. Only thing is it will also have the vendor folder.

This is certainly a viable option for some no doubt. Still a developer-only process, but a good way to support that part of the audience.

Then processwire admin can download the pw module, but the dependency used by my module is not upgraded. It may be still on 1.x . So that will break the upgrade.

PW won't let you install/update modules that don't have their version dependencies met. The user would be required to install/update the dependency first.

1 ) add a composer.json file 2 ) Register on http://packagist.org/ or host something on your own via http://github.com/composer/satis 3 ) Tag your releases

Sounds good to me!

Regarding the .htaccess updates you mentioned, do you know of another project that is already blocking those things, that we can pull from? I wanted to see if there was something known working/complete, before I attempt to write them myself.

Regarding the .htaccess updates you mentioned, do you know of another project that is already blocking those things, that we can pull from? I wanted to see if there was something known working/complete, before I attempt to write them myself.

Drupal uses https://github.com/drupal/drupal/blob/8.1.x/.htaccess

@harikt poked me on twitter, so hopefully here's a vaguely useful explanation of how Drupal handles modules:

Historically (D7 and earlier) most Drupal modules were "drop in directory, push button in UI, done". A rare few had some external library (JS or PHP) that needed to be added separately in another directory, and a module would then bridge it into Drupal via the libraries module.

For D8, we're now using Composer for much of Core's functionality. In 8.0.x we're using composer to download those libs and build an autoloader, then checking the whole thing into Git. We know that's bad (there are various historical reasons we did that), so just this week we removed the vendor directory from Git for the upcoming 8.1 release. The generated tarball will include the vendor directory, but git checkouts will not. There is also an unofficial Packagist create-project script, available here: https://www.drupal.org/node/2471553

For contribs, well, that's messy. For those that do not have any Composer dependencies of their own, nothing changes. Drop into the right directory, click a button, done. We register a PSR-4 root for enabled contrib modules with the Composer autoloader at boot time, so that we don't need to generate code nor do we get disabled modules loaded.

For those with composer dependencies, there are two options, both of which require CLI access. One, use a module called composer_manager that aggregates all of the composer.json files from each module together into a top-level composer.json file, then composer install that. The other approach is to just buckle down and go composer-everything, using the tool linked above or similar. We have a drupal-installer composer package that lets us put Drupal modules in the right place, rather than in /vendor, and there is a Packagist-clone site that auto-creates composer.json files for all Drupal modules on Drupal.org (with some fudging around version numbers). That is, using that approach you composer install everything, including Drupal modules, and then composer does the rest. Then you don't even check Drupal modules, or the core directory, into Git. This is the closest to "true Composer workflow" we offer currently.

If taking approach 2, that means that you cannot manage the site without the CLI. This is what people have been pushing back against for a long time, and is a valid critique: Composer is a tool for people who have access to the CLI, and ideally are generating build artifacts for production. That excludes a certain class of "FTP is too advanced for me" users. I don't know of any project that has a good solution to that. I asked Joomla a while back, and their approach was even less elegant than Drupal's. :smile:

Hopefully that answered the question...

@Crell Thank you very much for taking the time to reply here, this is all very helpful, excellent info. There's a lot to consider and think about here for now and in the future. It's a relatively small group of us that produce most of ProcessWire's 3rd party modules, so perhaps supporting composer options for our modules as an alternative is feasible to introduce in the near future.

We do still have to support that group of people that doesn't want to manually move files to and from the server (via FTP, rsync, etc.), as a sizable portion of our audience comes to ProcessWire after outgrowing WordPress (or similar), and some developers (and clients) seem to latch on to that 1-click upgrade/install of modules and core. But when it comes to enterprise/large, mission-critical and/or high traffic sites, I think it's safe to assume there's a developer and staging server involved, and a composer workflow is going to be certainly preferable and likely safer. This conversation here with @harikt and you is definitely making me enthusiastic about supporting this. Thanks again for your insights on this.

@ryancramerdesign Yeah, that same split-audience problem is one Drupal is facing. It's a tricky problem, and I don't know of any good solution at present. The approach that Drupal is taking means I predict over time Drupal will become a CLI-only tool, and people for whom FTP is as much as they have (either due to skill level or the access their host offers) will simply not be able to use it. Whether that is good or bad is a debatable point, but I predict that's what will happen.

Wow, now thats some discussion getting going in a few hours.

So the big elephant in the room is that we're trying to merge two systems (composer and pw modules) which both don't know about each other (yet?) while possibly depending on each other.

Say I want to use two modules, which both depend on a library, which is available via composer. Now at best this library would only be pulled in once. The fully processwire way would be bundling the library in a separate wrapper module and requiring it from both the modules (updates need to be done via the pw modules directory, a.k.a not the original library maintainer). The composer way would be having a composer.json for both files, which does make sure the dependency is installed when the modules are "installed" (e.g. composer required, not pw installed).

The only way I see for them to work together without resulting in things getting out of sync would really be a bridge, which does allow pw modules to require libraries installed by composer, e.g. as part of the modules info. Composer packages could just be handled as some kind of "minor" module as the composer.json should hold a quite comparable amount of data as the pw module json files. If composer itself is installed locally (e.g. the processwire folder) it might be possible to even use composer's functionality from the admin UI, but I'm not sure about that part.

Also would it be nice to have composer be able to install processwire modules, which seems like it's already been tackled by @harikt.

Everything else installed via composer (e.g. Carbon for comfortable date handling) shouldn't be a problem, because it's installed by the dev creating the project and is therefore not in our responsibility.

@Crell We've got to continue supporting the admin install/upgrade process of modules and core, just because we've already gone down that path… and can't really take it away. One of the ways PW grows is by adopting some of these turnkey processes. Even if I really prefer to rsync or Git these things to and from the server myself, and recommend the same for our more enterprise side users. But if we've got Composer as another option to accomplish the same, more options is better in my book. So I think we will do this.

If I was in Drupal's shoes, I might consider a good middle ground to be version monitoring and reporting on the admin side (and maybe that's something already present). Like if some module were out of date, or had a known issue, there would be a central repo monitoring this stuff that could be pinged, so that the admins/editors of the site would at least know it's time to get the developer involved.

Heading towards CLI-only for Drupal is not a bad thing I don't think. Several years ago CLI stuff seemed to be more the territory of coders. Now anyone doing anything on the web is using CLI tools (front-end, back-end, even designers), thanks to the likes of sass, various build and automation tools, git, and so on. It seems to be growing and good to see.

Good points @LostKobrakai

Say I want to use two modules, which both depend on a library, which is available via composer. Now at best this library would only be pulled in once.

In some ways we're lucky that we don't have a massive module directory (yet at least). And among those, very few have separate dependences outside of a core version. We don't have a lot of problems for Composer to solve really. It seems like @harikt method potentially works right in very nicely with our existing system. Though I still have questions, but all that can be answered after a little more time playing with it.

...it might be possible to even use composer's functionality from the admin UI, but I'm not sure about that part.

Chances are the only way to do that is to exec() to composer and that would only work on systems that support it (many hosts disallow it). Not to mention that exec() just seems like a last-ditch option for anything. It would probably require that Apache is running as the user account as well. Not something for the core to do, but maybe okay for a 3rd party module to tackle. Still, at this point I think one is just better off using what's already in the core and/or ProcessWireUpgrade module.

I actually think the core module install/upgrade works pretty well at present, there never seem to be any real issues that come up related to it (other than host firewalling or GitHub being inaccessible for one reason or another). But the problem is that if things do go wrong, then it's harder to fix.

The only safe way for the web process to run composer from the UI is to open an ssh connection back to itself and execute it that way. That requires the non-default ssh pecl extension. Drupal had a tool that let you download a module through the GUI that used that approach; basically no one used it because it was clunky and worked on about 2% of all hosts. :smile:

Thank you @Crell for taking your time to reply. I really appreciate.

@ryancramerdesign actually I got something to add here.

How does wire core being upgraded ? Is it only via the copy / paste of wire folder . Or does it upgrade automatically from the ui now .

Else this could also be handled via composer. Probably doing a split of the wire folder as a read-only folder to github. Something automated . There are lots of projects doing the same. Eg : Symfony, Laravel etc.

Thank you all for your participation. I am happy to help with what I can when ever I can.

Have a nice day.

This seems to point out exactly what I though of. Requiring composer as package locally should allow the usage of those php classes: http://stackoverflow.com/a/25208897

ryancramerdesign / ProcessWire

Why not use the vendor/autoload.php #1662