How does Yii address externally built and optimized asset files?

[tom--]

I have tried and failed to understand the new asset management automation in Yii 2. I am not yet a regular Yii 2 user so it's a bit hard. Conversations in #yii on Freenode with CeBe, samdark and others and haven't resolved my understanding.

So instead of trying to critique Yii 2's asset management, I decided that I would describe the asset management problem, as I see it, and derive a set of requirements.

Source assets

I need to deal with four different kinds of source asset files:

• Project asset files the are mine and specific to this project
• Asset files that are part of my own reusable widgets and modules (my extensions)
• Asset files that are part of third-party extensions
• Library asset files, such as jQuery

  Dependencies

There are two kinds of dependency that must be met:

• Web pages depend on project assets and on extensions (both mine and 3rd party)
• Project assets and extensions can depend library assets

Each dependency may involve version constraints.

Target assets

My web pages need a set of target asset files that are built from the sources.

Each web page delivered to a client's browser will generally include a small number of (often just one) JavaScript target asset files and a small number of (often just one) CSS target asset files.

Each target asset file is specified in terms of a recipe that the build system (which in my case is currently based on Grunt) uses to construct the target from some subset of the source assets.

Target optimization

Design of the set of target asset files is constrained by the dependencies but it also needs to consider use of the client's limited bandwidth, page load time, client cache use, and a statistical model of client behavior when visiting my site.

If I (stupidly) design the set of targets to eliminate wasted bandwidth on individual page loads without regard to a sequence of page loads and client asset caching (i.e. each web page receives only the assets that it depends on) then I will have more targets than I want. In the worst possible case, each page has its own target, many targets contain the same sources, and the client cache is not be used.

On the other hand, if I (stupidly) specify only one JS target and one CSS target for the entire web site then the client will enjoy a cache hit on every page except the first. But this strategy might fail on two counts. First, the delay and bandwidth use for a client's first page view of our site might be unacceptable. Second, I might expect many visitors to my site to view only one page before moving on.

Both of these stupid strategies are easy to automate but neither is, in general, acceptable. I need to use some intelligence. There is no alternative but to manually design the set of target assets given the dependencies and my understanding of client behavior. Target optimization takes intelligent design.

Assigning target assets to pages

This is another problem that requires intelligent design. There is no mapping from source assets to targets because this is in general a one-to-many relation. When more than one target can meet a page's dependencies, selection among them needs to account for all the same factors: file size, bandwidth, page load time, client cache use, and client behavior.

Example

Say my site has these four pages with the shown page dependencies. Forget about lib deps for now.

Page-1
   |---MyScript-1
   |---MyWidget-1
   |---3pWidget

Page-2
   |---MyScript-1
   |---MyWidget-1

Page-3
   |---MyScript-2
   |---MyWidget-2
   |---3pWidget

Page-4
   |---3pWidget

Now, given the sizes of these assets, I decide that I want the following two target assets.

TargetAsset-1
   |---MyScript-1
   |---MyWidget-1
   |---3pWidget

TargetAsset-2
   |---MyScript-2
   |---MyWidget-2
   |---3pWidget

Registering TargetAsset-1 on Page-1 and TargetAsset-2 on Page-3 is easily automated. I imagine Yii's Asset Bundles protocol handles it.

I guess that if there is no Asset Bundle other than TargetAsset-1 that contains MyScript-1 and MyWidget-1 then Yii can figure out that it has to register TargetAsset-1 on Page-3, which was intended in the design of the two target assets.

What to register on Page-4 is ambiguous without some statistics. In addition to the sizes of the targets, I need to know the respective chances that targets are in a client's cache when the client requests Page-4. I can get the information together and make the decision but I don't think Yii can.

Now what if 3pWidget was delivered with an Asset Bundle? Will Yii automatically register that on Page-4? What if I think that my TargetAsset-1 should be used instead? In general, 3pWidget's dependencies can be met by one of many target assets and the selection among them will differ on different pages.

Asset management requirements

So, given the above, I need from asset management in Yii:

1. a way to formally specify the dependencies
2. an approach that all 3rd party extensions use to deliver their source assets to my project
3. a means to obtain all depended-on library source assets
4. a way for me to find the path to each source asset so I can use it in my Gruntfile
5. a way to specify back into Yii's asset management the dependencies that each target meets
6. a hinting system that I can use, on a per view, action or controller basis, to resolve potential ambiguity in resolving targets from dependencies

With this, I and 3rd part extension developers can declare and register sources and use extensions without regard to target assets. When Yii assembles a page it can refer to the hints to find the appropriate target assets or fall back on some default source->target resolution algorithm.

In this way, as the overall designer of my web site, I have control over the build system and which pages use which optimized target asset files.

Questions

The basic question is: how does Yii meet these requirements?

My understanding is incomplete and muddled.

Iiuc, a reusable extension uses an Asset Bundle to declare the assets it needs. And I can write asset bundles for my project assets and register those. Using these bundles and some combination of Composer and Bower, these source assets are obtained. That covers the first three requirements.

Now I need to write my Gruntfile. In it I describe how each target is built from the sources by modules that do the processing needed (e.g. minify, concatenate). The description is a JavaScript object literal that references the sources by their filesystem path. So, I need the file path (let's say, relative to the project root) of each source asset. How do I get that within the context of my Gruntfile?

Now, thinking about the example above, how do I optionally override the 3rd-party widget's registration of its assets with one of my target assets built by Grunt? This cannot be a simple target(source) lookup because source-to-target is a one-to-many relation and the selection has to be, in general, made by an intelligent designer.

[qiangxue]

Related to https://github.com/yiisoft/yii2/issues/1411 I'm writing the structure-assets.md guide ATM which should answer some of your questions.

[qiangxue]

@tom-- Please read http://www.yiiframework.com/doc-2.0/guide-structure-assets.html to see if it answers all your questions. If not, please ask the unanswered ones again. Thanks.

[philippfrenzel]

how can you apply the position of an asset locally not globaly / so just one js needs to be loaded / on top

[samdark]

Via bundle dependencies.

[philippfrenzel]

ok, thats what I did, thought their might be a leander way...;) Thanks!

[tvdavid]

I read the new document but it's not entirely clear to me yet.

The asset publishing method isn't very reliable in a distributed setup: the server that ends up creating the symlink in /assets might not be the same as the one that ends up handling the GET request for that resource. Nginx is perfectly capable of serving the entire vendor/ directory directly; is there any way I can tell Yii globally not to do any asset publishing, but in such cases to rewrite the $baseUrl to a certain "root path"?

Or do I have to configure the assetManager for every asset bundle individually, like this?

'yii\imperavi\ImperaviRedactorAsset' => [
    'sourcePath' => null,
    'baseUrl'    => '@static/vendor/asofter/yii2-imperavi-redactor/yii/imperavi/assets'
]

[qiangxue]

Yes, right now you have configure every asset bundle if you want to avoid publishing their asset because these asset bundles by default have specified their assets are located behind Web.

I think we could enhance the asset command to do offline asset publishing as well as generating an asset bundle list that fits your need described here.

[tom--]

@tvdavid I think it has to be disabled per-bundle. And I think that's appropriate. For the production environment you described, I would think the goal would be to build a set of assets that will be, to AM, external.

[tvdavid]

Got it, thanks. The current method is workable, but it is quite tedious.

Couldn't we introduce some sort of "sourcePathMap" where you can tell Yii that any file under the $sourcePath of "x/y" will be accessible via a certain URL without asset publishing being necessary?

E.g.

sourcePathMap => [
    '@bower' => '@static/vendor/bower',
]

[qiangxue]

@tvdavid Yes, this is possible. We may consider enhancing asset manager further as we collect more feedback from real world. Currently to implement what you want, you may override AssetManager::loadBundle(), which shouldn't be too difficult.

[tom--]

@qiangxue I like the new asset manager and the new document. Great work! It seems all those discussions on this topic paid off. I hope that writing this issue was constructive.

I have one major concern. I don't see a good way to obtain the file paths of a 3rd party extension's asset bundle and its sources in the context of my Gruntfile. I haven't given this a great deal of thought but it seems that, in order to avoid unsustainable copy-paste workflow, I need a way to refer to assets and bundles in the Gruntfile context using the same names that are used in the Yii Asset Manager context. My first guess is that a JS function that returns asset bundle properties (as overriden by the AM config) might solve the problem.

I will write my other questions and comments later.

[qiangxue]

You can get the path to asset files in 3rd party asset bundles via their sourcePath property and the relative paths of the asset files.

[tom--]

@qiangxue Yes. But I'm unsure how to do that from outside the Yii/PHP environment. Write in JavaScript an AssetBundle class file parser?

[qiangxue]

Again, we can enhance the asset command to support the use with grunt. I have mentioned about this within the core team before. We just don't have time to do this now.

[tom--]

Some smaller comments on the new structure-assets.md

• There is no discussion of assets that are not JS or CSS files or files that compile into these. I would like to see how a 3rd-party extension bundles things like images and fonts.

• The doc talks about "overwriting" asset bundle properties, e.g.

  "When you specify the sourcePath property, the asset manager will publish the assets in this bundle to a Web-accessible directory and overwrite this property accordingly."

  I am not sure what this really means. It sounds like AM will rewrite my asset bundle class file, but that's implausible.

• The doc uses both "folder" and "directory". I'm not sure what is meant by the difference. If there is none then it would be better to use only one of the two.
• There are a few instances of "asset bundle class" and many instances of "asset bundle". I think most of the former would be better without the "class".
• YUI Compressor is obsolete. Please don't mention it.

[samdark]

There is no discussion of assets that are not JS or CSS files or files that compile into these. I would like to see how a 3rd-party extension bundles things like images and fonts.

Good idea. Generally you should publish whole directory in case of images and other resource groups.

It sounds like AM will rewrite my asset bundle class file, but that's implausible.

It means setting sourcePath in runtime after publishing is done. Should be phrased cleaner.

The doc uses both "folder" and "directory".

We should stick to directory since it's used all over the guide. Folder is UI concept, not FS.

YUI Compressor is obsolete. Please don't mention it.

Why? It's supported, regular releases and it compresses CSS pretty well. Do you know better alternative?

[tom--]

Regarding YUI Compressor, the last time I looked, a bit more than a year ago, it had been abandoned as a JS minifier. I didn't know it was a CSS minifier.

As for alternatives, clean-css (npm) and Rainpress (gem) are good. clean-css is the most popular among Grunt users and I use rainpress in my Nanoc projects

[dynasource]

YUI Compressor and jsCompressor are still powerful with many features. Please look at the well documented steps to integrate this in your application http://www.yiiframework.com/doc-2.0/guide-structure-assets.html#combining-compressing-assets