`composer update` takes too much time

[kenjis]

Because when we run composer update, Composer reads all composer.json in all branches/tags for all Fuel repositories.

[Art4]

That's another reason why we defined the package repositories in our composer.json to specific commits:

    "repositories" : [
        {
            "type": "package",
            "package": {
                "name": "fuel/core",
                "type": "fuel-package",
                "version": "1.8.0",
                "dist": {
                    "url": "https://github.com/fuel/core/archive/26f6432a9f41659280c70ba47c16170adf1e7611.zip",
                    "type": "zip"
                },
                "source": {
                    "url": "https://github.com/fuel/core.git",
                    "type": "git",
                    "reference": "1.8/master"
                }
            }
        },
        {
            //...
        }
    ],

It would be great, if fuel would use tags.

[kenjis]

@Art4 Do you know why Composer reads all composer.json in all branches/tags now? I don't understand why Composer needs to do so.

[Art4]

I'm not sure, but I think because the repositories are defined as vcs types: { "type": "vcs", "url": "https://github.com/fuel/core" },

Composer looks first for the repository fuel/core at packagist.org, but it isn't defined there. Second it read the defined repositories and must read all branches to determine, which branch is required. Thats the reason why it reads all branches and tags. But that is only my guess.

[Art4]

I've done some more research and I can confirm my guess. If you define a vcs repository than composer has to scan all repositories for the composer.json, tags and branches. So there are two solutions:

1) Define a package repository

If you define a package repository you can define specific versions to a specific distribution. See my example above:

    "repositories" : [
        {
            "type": "package",
            "package": {
                "name": "fuel/core",
                "type": "fuel-package",
                "version": "1.8.0",
                "dist": {
                    "url": "https://github.com/fuel/core/archive/26f6432a9f41659280c70ba47c16170adf1e7611.zip",
                    "type": "zip"
                },
                "source": {
                    "url": "https://github.com/fuel/core.git",
                    "type": "git",
                    "reference": "1.8/master"
                }
            }
        },
        {
            //...
        }
    ],

2) Use a tool/service for caching the data

There are tools and services out there which can scan the repositories and caches the results. Such tools are Satis or packagist (as a service). :smile:

So it would be nice if @WanWizard could publish the packages core, oil and so on on packagist. This way we have no need to define the repositories, because composer will search by default on packagist for a package. And if the packages had tags, we don't have do define the versions as dev-1.8/master, but could just define 1.8.*.

[kenjis]

@Art4 Thank you for your investigation!

I hope 2). It is common usage of Composer, and it seems it is the best practice. If we choose 1), we can't update Fuel using composer update. We (both Fuel users and a release manager) will have to change composer.json manually on every release.

[Art4]

@kenjis I would suggest to keep the composer.json and composer.lock with git. This way only one developer has to do composer update and commits the json and lock file. All other developer only has to execute composer install. This works very nice in our team.

This is also the recommended way of composer:

Commit your application's composer.lock (along with composer.json) into version control.

This is important because the install command checks if a lock file is present, and if it is, it downloads the versions specified there (regardless of what composer.json says).

[kenjis]

@Art4 Yeah, it is the best practice for team development. I always run git add -f composer.lock.

[Art4]

Then I don't unterstand your problem. If you're commiting the composer.lock file, the other developers only have to run composer install and the update will run fast and without delay for searching the right branches.

[WanWizard]

The main reason for not being on packagist was that in the past our "non-standard" method of versioning (using branch names instead of tags) was not supported. Not sure it is now.

Also, in the "early days" a lot of backporting happened, which makes tagging more complex. So Dan and Jelmer couldn't be bothered, and nothing has changed since.

I'm not a composer expect (as you may have noticed), I'm open for any good suggestion. That doesn't involve tons of work at our end.

[Art4]

@WanWizard I'm pretty sure it should be possible to just add one point (creating a tag) to your release activities and you are ready for packagist, but don't have to change your branching workflow. :smile: I'll working this out a little:

This has to be done only one time:

1) Remove the repositories section

In https://github.com/fuel/fuel/blob/1.8/master/composer.json you can remove the repository section and require the packages with tags instead of branches.

-    "repositories": [
-        { "type": "vcs", "url": "https://github.com/fuel/docs" },
-        { "type": "vcs", "url": "https://github.com/fuel/core" },
-        { "type": "vcs", "url": "https://github.com/fuel/auth" },
-        { "type": "vcs", "url": "https://github.com/fuel/email" },
-        { "type": "vcs", "url": "https://github.com/fuel/oil" },
-        { "type": "vcs", "url": "https://github.com/fuel/orm" },
-        { "type": "vcs", "url": "https://github.com/fuel/parser" }
-    ],
    "require": {
        "php": ">=5.3.3",
        "composer/installers": "~1.0",
-        "fuel/core": "dev-1.8/master",
-        "fuel/auth": "dev-1.8/master",
-        "fuel/email": "dev-1.8/master",
-        "fuel/oil": "dev-1.8/master",
-        "fuel/orm": "dev-1.8/master",
-        "fuel/parser": "dev-1.8/master",
+        "fuel/core": "1.8.0",
+        "fuel/auth": "1.8.0",
+        "fuel/email": "1.8.0",
+        "fuel/oil": "1.8.0",
+        "fuel/orm": "1.8.0",
+        "fuel/parser": "1.8.0",
        "fuelphp/upload": "2.0.6",
        "monolog/monolog": "1.5.*",
        "phpseclib/phpseclib": "2.0.0",
        "michelf/php-markdown": "1.4.0"
    },

2) Creating tags

Create a tag 1.8.0 (or 1.8 or v1.8.0, it don't really care, but be consistent) for every 1.8/master branch. You can do this starting for release 1.8 and don't have to wait for 1.8.1 or 1.9.

$ git tag -a 1.8.0 -m '1.8.0' 1.8/master

and push the tag to Github. This has to be done in every package (fuel, core, oil, orm, parser, email and docs).

In theory you could do this also backwards for 1.7, 1.7.1, 1,7,2 and 1.7.3, but I wouldn't recommend that, as it brings no benefit and can confuse users.

3) Submit to packagist

Submit all packages to packagist.com, except fuel/fuel because it is already submitted.

That's all you have to do for now. Maybe updating the zip file?

New Release activities

For new releases add a new point to your release activities. (I'm not sure if it is out of date...)

4. Branch the current develop to master for all framework repo's (fuel, core, docs, auth, oil, orm, email, parser).
5. Add the changelog to fuel/fuel CONTRIBUTING.md in the new master repo.
6. Add the new version to the docs download page.
7. Remove the '-dev' suffix from the Fuel::$version and the Docs version number in master.
+ 8. Create a new tag in all framework repo's (fuel, core, docs, auth, oil, orm, email, parser).
9. Create a fuelphp-version zip file from master, and upload it to the website.
10. Branch the current develop to the new develop, make the new develop repo's current.
11. Bump the Fuel::$version and the Docs version number.

It is also possible to release different versions of the packages (fuel/fuel: 1.8.4, fuel/core: 1.8.2, fuel/orm: 1.8.1, ...), but I think this would be to complicated to maintain the zip file and release notes. Keeping all packages on the same release seems to be the best practice at the moment.

I don't know if I've forgotten something but I'm open for questions and support. :smile:

[kenjis]

@Art4

If you're commiting the composer.lock file, the other developers only have to run composer install and the update will run fast and without delay for searching the right branches.

Yes, but when I run composer update, it takes more than 5 minutes. And the installation of Fuel via oil command also takes time.

I think it is problem that installation/update takes more than 5 minutes.

[kenjis]

I agree with @Art4 I've been wanting tagging. If Fuel has tags, we could install a specific version if we want. We don't need --prefer-dist option when installing.

[Art4]

@kenjis Yes, composer update takes 5 minutes because of scanning the vcs packages as I mentioned above. But after that the composer.lock is ready and can be commited. The other developers just run "composer install" (not "update"!). This shouldn't take much time, because this will install the exact packages from the composer.lock. Otherwise I would assuming network issues.

The fuel installation via php oil r install just "chmod"ed 4 folders, see https://github.com/fuel/core/blob/1.9/develop/tasks/install.php. This also shouldn't take so much time if you havn't modifed the install tasks.

[kenjis]

@Art4 I mean the quick installation http://fuelphp.com/docs/#/quick_install, or oil install. Not oil r install.

[Art4]

@kenjis Oh, I see. This problem should also be resolved with the use of tags and packagist.

[WanWizard]

I'll put it on the todo list

[WanWizard]

Changes are made, composer create-project fuel/fuel works fine without checking repo's. Also modified the oil install code to use this.

What would now be the correct way of installing a dev version (like 1.9/develop)? I haven't touched that composer.json yet.

[Art4]

@WanWizard I fuel/fuel on branch 1.9/develop you can require the specific branches prefixed with dev- like in your old workflow.

    "require": {
        "php": ">=5.3.3",
        "composer/installers": "~1.0",
        "fuel/core": "dev-1.9/develop",
        "fuel/auth": "dev-1.9/develop",
        "fuel/email": "dev-1.9/develop",
        "fuel/oil": "dev-1.9/develop",
        "fuel/orm": "dev-1.9/develop",
        "fuel/parser": "dev-1.9/develop",

For a new release, e.g. 1.8.1: In fuel/fuel you should merge the 1.9/develop branch into 1.8/master, make sure, the requirements points to 1.8.* again and create the tag 1.8.1.

In all other packages you only have to merge 1.9/develop branch into 1.8/master branch and create the tag 1.8.1.

[WanWizard]

Learning all the time. ;-)

I've updated the "update procedure" with this info for minor version updates. And removed the vcs section in the json, now that the development packages are also on packagist.

Guess this can be closed now?

[Art4]

I recommend you to define the specific tags 1.8.0 in fuel/fuel, not with the wildcard * because of the high dependencies of the packages to fuel/core and to each other. In example you release version 1.8.1 for each package it wouldn't possible to install 1.8.0 again, because fuel/fuel:1.8.0 requires the latest packages from the 1.8.* branch. So you would have fuel/fuel:1.8.0 but with fuel/core:1.8.1 and so on.

[WanWizard]

That was the idea behind the star. Minor releases are bugfixes (in general), so you always would want to have the latest available. It also means that someone that has 1.8.0 running locally will automatically update to 1.8.1 on a composer update without having to change the json file.

There's a 1.8.0.1 in the making to fix a bug in the release code, and I would like everyone that installs fuel to get that one automatically.

[Art4]

Great. This is even better. :+1:

But now I'm confusing about 1.8.1 and 1.8.0.1. What is a bugfix and what a minor release? :sweat_smile:

[WanWizard]

lol.

For me, 1.8.0.1 is a bugfix on 1.8.0, and 1.8.1 is a new minor release.

For example, fixing https://github.com/fuel/core/issues/1990 doesn't warant a new minor release for me.

[Art4]

Ok. I was thinking about semantic versioning. :smile:

[WanWizard]

That's a bit too modern, that's for v2. This can be closed I think.

[WanWizard]

Need to come back to this: installing 1.9/develop now fails:

$ composer install --prefer-source
You are running composer with xdebug enabled. This has a major impact on runtime performance. See https://getcomposer.org/xdebug
Loading composer repositories with package information
Installing dependencies (including require-dev)
Your requirements could not be resolved to an installable set of packages.

  Problem 1
    - The requested package fuel/auth could not be found in any version, there may be a typo in the package name.
  Problem 2
    - The requested package fuel/email could not be found in any version, there may be a typo in the package name.
  Problem 3
    - The requested package fuel/oil could not be found in any version, there may be a typo in the package name.
  Problem 4
    - The requested package fuel/orm could not be found in any version, there may be a typo in the package name.
  Problem 5
    - The requested package fuel/parser could not be found in any version, there may be a typo in the package name.

Potential causes:
 - A typo in the package name
 - The package is not available in a stable-enough version according to your minimum-stability setting
   see <https://getcomposer.org/doc/04-schema.md#minimum-stability> for more details.

Read <https://getcomposer.org/doc/articles/troubleshooting.md> for further common problems.

I don't immediately see what is wrong, "dev-1.9/develop" exists on packagist.

[Art4]

For some reason the branches from auth, email, oil, orm and parser aren't listet at packagist. In fuel and core there are only the 1.8 and 1.9 branches.

[WanWizard]

It doesn't even get that far. Packagist doesn't like "dev-1.9/develop", it doesn't recognize it as a valid package, see https://packagist.org/packages/fuel/fuel#dev-1.9/develop

Even though it exists for core: https://packagist.org/packages/fuel/core#dev-1.9/develop

Also, you'd expect it would go to github and clone the repo if you use "--prefer-source", but it doesn't do that either.

[Art4]

I don't know if this has something to do with this issue but all packages except fuel/fuel has in composer.json defined "type": "fuel-package". This isn't a valid type, see https://getcomposer.org/doc/04-schema.md#type

[WanWizard]

"fuel-package" is a valid custom defined type, see https://github.com/composer/installers#current-supported-types. The install location for "fuel/core" has an override in the "extra" section.

[Art4]

Good to know, thanks. So, I can't determine why not all branches are listet at packagist.

I've made some tests on my own package with the same branch names. I've created a branch 1.9/develop, it shows up immediately on packagist and I can require and install the branch with composer.

[WanWizard]

I've fixed that, they should all be there now.

And that seems to have solved it, although I don't know why I got an error about fuel/core, which did have that branch available. Weird, but hey...

[Art4]

:+1: Works on my site too. May I ask what the problem was?

[WanWizard]

Don't know, just did a manual update.

[kenjis]

Thanks, guys. Now fuel packages are cached. So I can reinstall Fuel very fast.

$ time oil create blog
composer create-project fuel/fuel blog
Installing fuel/fuel (1.8.0)
  - Installing fuel/fuel (1.8.0)
    Loading from cache

Created project in blog
Loading composer repositories with package information
Updating dependencies (including require-dev)
    Prefetch start: total: 13
    Finished: success:0, skipped:13, failure:0, total: 13
  - Installing composer/installers (v1.0.25)
    Loading from cache

  - Installing fuel/core (1.8.0)
    Loading from cache

  - Installing fuel/auth (1.8.0)
    Loading from cache

  - Installing fuel/email (1.8.0)
    Loading from cache

  - Installing fuel/oil (1.8.0)
    Loading from cache

  - Installing fuel/orm (1.8.0)
    Loading from cache

  - Installing fuel/parser (1.8.0)
    Loading from cache

  - Installing fuelphp/upload (2.0.6)
    Loading from cache

  - Installing psr/log (1.0.0)
    Loading from cache

  - Installing monolog/monolog (1.5.0)
    Loading from cache

  - Installing phpseclib/phpseclib (2.0.0)
    Loading from cache

  - Installing michelf/php-markdown (1.4.0)
    Loading from cache

  - Installing fuel/docs (1.8.0)
    Loading from cache

fuel/fuel suggests installing dwoo/dwoo (Allow Dwoo templating with the Parser package)
fuel/fuel suggests installing mustache/mustache (Allow Mustache templating with the Parser package)
fuel/fuel suggests installing smarty/smarty (Allow Smarty templating with the Parser package)
fuel/fuel suggests installing twig/twig (Allow Twig templating with the Parser package)
fuel/fuel suggests installing pyrocms/lex (Allow Lex templating with the Parser package)
fuel/fuel suggests installing mthaml/mthaml (Allow Haml templating with Twig supports with the Parser package)
monolog/monolog suggests installing mlehner/gelf-php (Allow sending log messages to a GrayLog2 server)
monolog/monolog suggests installing raven/raven (Allow sending log messages to a Sentry server)
monolog/monolog suggests installing doctrine/couchdb (Allow sending log messages to a CouchDB server)
monolog/monolog suggests installing ext-amqp (Allow sending log messages to an AMQP server (1.0+ required))
monolog/monolog suggests installing ext-mongo (Allow sending log messages to a MongoDB server)
phpseclib/phpseclib suggests installing ext-libsodium (SSH2/SFTP can make use of some algorithms provided by the libsodium-php extension.)
phpseclib/phpseclib suggests installing ext-gmp (Install the GMP (GNU Multiple Precision) extension in order to speed up arbitrary precision integer arithmetic operations.)
phpseclib/phpseclib suggests installing pear-pear/PHP_Compat (Install PHP_Compat to get phpseclib working on PHP < 5.0.0.)
Writing lock file
Generating autoload files

real    0m16.303s
user    0m5.650s
sys 0m0.930s

[Art4]

:+1: