Maintaining core extensions separately

[qiangxue]

In order to enable faster version releases and make core extensions more scalable, we are thinking to maintain each core extension and application template separately in individual github projects. Below is a preliminary plan. Before we execute it, we would like to have it reviewed by everyone to make sure we do not omit anything important. Thank you!

Tasks

• [x] stop automatic subsplit
• [x] delete /extensions and /apps from yii2 repo
• [x] move/adjust unit tests and documentation specific for each extension to the corresponding extension repo
• [x] adjust yii2-dev to keep BC (solved by dev command)
• [x] adjust build scripts to support major/minor releases
• [ ] adjust apidoc tool for generating api/guide docs for extensions
• [x] enhance build scripts to support automatic patch releases
• [x] adjust travis configs. Each extension or app will need its own travis setup
• adjust internal docs
  • [x] update https://github.com/yiisoft/yii2/blob/master/docs/internals/versions.md
  • [x] adjust git workflow for extensions https://github.com/yiisoft/yii2/blob/master/docs/internals/git-workflow.md
  • [x] adjust getting started https://github.com/yiisoft/yii2/blob/master/docs/internals/getting-started.md

    Project Organization

    1. Maintain individual core extensions and application templates in separate independent GitHub projects. Each extension/app project will
  • keep using its current project name to ensure BC. For example, the yii2-gii extension will be maintained in yiisoft/yii2-gii project.
  • maintain its tutorial doc in its "docs" folder. The API doc will be generated on-the-fly when the extension/app is being released.
  • maintain its own test code in its "tests" folder.
  • maintain its own message translations and all other relevant meta code.
  • track issues via the corresponding GitHub project.
  • be released independently as needed. See versioning policy below.
    2. Continue using "yii2" as the main project to maintain the core framework. "yii2" should contain the following stuff
  • the core framework code
  • the definitive guide. Extension-specific guide should be put in the corresponding extension projects. API doc should be generated on-the-fly when releasing the core framework.
  • the test code for the core framework only
  • build tools and internal docs
    3. Continue doing automatic subsplit from "yii2" to "yii2-framework" to ensure BC.
    4. Track the core framework issues in the "yii2" project. Extension issues should be reported to the corresponding extension projects.

       Release and Versioning Policy

    5. Version numbers are in the format of 2.x.y.z.
  • 2.x: major releases, containing major features. May be BC-breaking. Release cycle is around 6 months. With news announcements. Project site will be updated.
  • 2.x.y: minor releases, containing minor features and bug fixes. Must be BC with prior 2.x.* releases. Release cycle is around 1 to 2 months. With news announcements. Project site will be updated.
  • 2.x.y.z: patch releases, containing bug fixes only. Must be BC with prior 2.x.*.* releases. Release cycle is around 1 to 2 weeks. No news announcement or project site update (unless it contains major/security issue fixes). The release process will be made mostly automatic.
    2. Branching policy
  • master branch is the dev branch for the current major release. Once a new minor release is ready, create a maintenance branch named 2.x.y off master.
  • Create tags 2.x.y.z on 2.x.y branch to mark patch releases (including 2.x.y.0 release).
  • Changes on 2.x.y will be merged back to master constantly.
  • For the next major release, create a dev branch named 2.t-dev. Once the next major release is ready, it will be merged to master, and then create a 2.t.0 branch off master (the old 2.t-dev branch will be deleted to avoid confusion).
    3. yii2 and core extension/application projects are released independently of each other.
  • Both yii2 and core extension projects follow the above versioning and branching policies.
  • The release cycles described above only apply to the core framework.
  • Extensions and application templates are released on demand. It is likely that an extension has no new releases for a very long time because it does not receive any bug fixes or enhancements.
  • Extensions and application templates may have version numbers that are different from that of the core framework. For example, yii2-gii may have the latest version number as 2.0.5 while yii2 is already at 2.1.3, even though currently they are all synchronized at 2.0.3.

[Ragazzo]

should be done already long time ago :)

[mtangoo]

"API doc should be generated on-the-fly when releasing the core framework" How does this affect us who were generating Documentations locally? Will it need extra gym or it will just work? I can't see any problem to the proposed plan so far!

[nicklasos]

+1

[i-sevostyanov]

+ good idea

[nineinchnick]

So it's only about versioning? I mostly care about being able to clone everything at once to hack it.

Looks good to me.

[lynicidn]

:+1:

[Ryadnov]

:+1:

[farmani]

it should be done earlier but its nice to hear you and I believe that some new developer can join to developers team to focus on special extension and so new idea brings up by focusing on some specific part of a big project

[SDKiller]

Remove bower-asset/yii2-pjax from require section of framework composer.json and move framework/widgets/PjaxAsset.php and framework/widgets/Pjax.php to separate extension like bootstrap and jui.

Reasons:

• it is not mandatory for framework functionality
• someone may not even need it
• someone may want to use another push state implementation
• it depends much on upstream and may need instant fixes like https://github.com/yiisoft/yii2/issues/7629
• each bower-asset adds overhead on composer update

But now it is even not possible to avoid bower-asset/yii2-pjax installation as it is required by framework's composer.json

[harry-flipkod]

+1

[ghost]

+1

[zmilan]

Good one :+1:

[creocoder]

@SDKiller

Remove bower-asset/yii2-pjax from require section of framework composer.json and move framework/widgets/PjaxAsset.php and framework/widgets/Pjax.php to separate extension like bootstrap and jui.

:+1: But... its not enough, all JS stuff should be removed to extensions include client side from validators, etc. But yes, currently main pain is fxp/composer-asset-plugin. And there is nothing in first post about that. So in general everything looks like simple cosmetic changes which will solve only small portion of troubles.

[voskobovich]

+1

[lichunqiang]

+1 for @creocoder

[samdark]

@SDKiller, @creocoder while I understand why you want to avoid using fxp/composer-asset-plugin, it can't be easily done in 2.0.x since it breaks backwards compatibility significantly.

[tecnologiaterabyte]

Excellent allow deploy fixes and enhancements fastest way. 👍

[adamaltman]

This proposal sounds good at first, from a code/test organization perspective. But if the goal is to really speed up releases, this only works if this structure empowers more people to contribute in more meaningful ways than currently possible, including reviewing and accepting pull requests.

Also, I wonder is it possible that core changes break some extension compatibility? (As I understand the core may not depend on an extension, but an extension may depend on the core. Unless I'm wrong there.) And who is responsible in that scenario for regaining compatibility? And how would they be aware? What would be the behavior of the CI for a pull request? Does it mean that when someone changes an extension that relies on a particular core api, that they must also submit tests to the core to ensure that api persists in future versions of the core?

[dynasource]

Good decision. Eventually it had to be done, so starting now is wize.

With respect to external parts like Pjax. I advocate to put these in separate extensions too because of either limitations / or not using it / or the use of alternative packages like i.e Intercooler.

About the fxp/composer-asset-plugin. This significant limitation on proper behavior of composer is also something that deserves attention on short term. If I could choose one thing to do first, it would be this.

[samdark]

@adamaltman the current issues are

1. git subsplit from main repo to read only copies. It is extremely slow.
2. we can't release extensions independently.

Proposed change solves it.

Compatibility between extension / framework versions could be achieved if extension will specify framework version in its requirements. We're going to do it as well.

CI will work for extensions i.e. extension will be tested with framework version specified in its composer.json. It's nowadays rarely required to change core code in order to fix something in extensions afterwards. If it's the case, extension release has to wait for framework release.

[samdark]

@dynasource I'm afraid it could not be done in 2.0.x... Or probably I don't see how to do it w/o breaking changes. Anyway, it's a separate issue and it's better to discuss it separately.

[adamaltman]

Yes, @samdark for issues 1 and 2, it's good. :+1:

[qiangxue]

@SDKiller Thanks for pointing out the issue with pjax. I think we handle this in a separate issue because it involves breaking BC. In this issue, we will focus on resolving this subsplit issue and make sure no BC is broken.

The following changes will need to be made for this issue:

• stop automatic subsplit
• move/adjust unit tests and documentation specific for each extension to the corresponding extension repo
• adjust yii2-dev to keep BC
• adjust build scripts to support major/minor releases
• enhance build scripts to support automatic patch releases
• what else?

[samdark]

1. Decide on how to build/display docs for individual extensions.
2. Decide on how to build/display API docs for extensions.

[lynicidn]

@qiangxue minimizing dependencies

[samdark]

@lynicidn that's separate issue.

[lynicidn]

how u want check requirments? in all extensions put requirments.php ? https://github.com/yiisoft/yii2/blob/master/apps/advanced/requirements.php#L124

[qiangxue]

I think requirement checker should be made in application templates only. Or perhaps we should simply rely on composer dependency to check them.

[lynicidn]

may be need add interface for checker and check before put in vendor/autoload.php ? If i don't use mailer or imagine - why i should have it in template ? Logicaly check it before configure

[qiangxue]

Let's discuss about requirement checking in a separate issue. I hope we can make this subsplit change during the weekend. We will adjust other issues found later.

[lynicidn]

@qiangxue ok, thx for answer

[johonunu]

:+1:

Also, I am on board with @creocoder about "fxp/composer-asset-plugin". This needs a better alternative/solution.

[grikdotnet]

Authors of 3rd party libs will just not update compatibility list with each major release of each core extension they depend on. Users will stick with old versions or get into package conflicts, as it happens with unofficial OS repositories. You may want to bind core extensions major releases to releases of the framework and discourage 3rd party libraries authors to bind to minor releases.

[samdark]

A guide would solve it. One can specify ~2.0.3 and it should work properly till 2.1 release.

[samdark]

@johonunu it's another issue that deserves its own ticket and discussion.

[robregonm]

Agreed!

[qiangxue]

I listed the tasks to be done in the first post.

Need help to finish these tasks...

[cebe]

adjust yii2-dev to keep BC

@qiangxue what do you mean with this?

[samdark]

I can handle moving docs.

[samdark]

@qiangxue we need write permissions to former read only repos.

[qiangxue]

@samdark Done. I also think we probably can get rid of the auto subsplit of yii2-framework. Will do this after we finish this issue first.

[qiangxue]

adjust yii2-dev to keep BC

Sorry, no need BC for yii2-dev. I mainly want to still have a way to develop both the core and extensions. Right now we have a problem because when you install an extension, it will bring in yii2-framework while we actually should work on yii2 repo.

[janisto]

👍

[cebe]

I also think we probably can get rid of the auto subsplit of yii2-framework. Will do this after we finish this issue first.

why and how? If it is only about speed, I have thought a lot about how to improve the subsplit speed. We can fix that speed problem.

[cebe]

status of moving the tests:

• [x] basic app
• [x] advanced app
• [x] authclient
• [x] bootstrap
• [x] elasticsearch
• [x] faker
• [x] gii
• [x] imagine
• [x] mongodb
• [x] redis
• [x] smarty
• [x] sphinx
• [x] swiftmailer
• [x] twig
• [x] jui

[samdark]

Status of moving docs:

• [x] basic app
• [x] advanced app
• [x] authclient
• [x] bootstrap
• [x] elasticsearch
• [x] faker
• [x] gii
• [x] imagine
• [x] mongodb
• [x] redis
• [x] smarty
• [x] sphinx
• [x] swiftmailer
• [x] twig
• [x] jui
• [x] codeception
• [x] debug

[schmunk42]

:+1: asked for it in 2013 ;) https://github.com/yiisoft/yii2/issues/379#issuecomment-29099296

[samdark]

@schmunk42 good ideas have to cool down a bit before settiling into enough number of minds to be applied :)

[shaan360]

Cool its going to be cleaner :+1:

[SDKiller]

Probably it is also nesessary to specify major versions for 3rd-party dependencies

Related https://github.com/yiisoft/yii2/issues/7848