Open jonschlinkert opened 9 years ago
In the referring issue, are you asking me to port my plugin to assemble v0.6.0 and if so which changes do have to be made?
are you asking me to port my plugin to assemble v0.6.0
no, this would be a different project, we've been talking about publishing this as a combination of helpers, middleware and plugin(s), since pagination and indexing is such a big part of our plans for assemble.
which changes do have to be made?
@doowb and I started working on some pagination/index stuff a couple of weeks ago, the changes would actually be pretty extensive, aside from the pagination logic itself. e.g. the core concepts would be the same: get a template collection, loop over it, build up a pagination context, etc.
From a technical standpoint, if everything were documented, IMHO, the v0.6.0 API is much easier to use than what you had to do to get this working with assemble as a grunt plugin. But... we don't have docs yet so it's understandable if this is too much for someone to take on before the docs are done.
That said, if you or someone else wants to give this a shot, we'd love to see it as a core project, and we'd be happy to provide guidance as needed.
So, from my understanding:
Then, how do you configure individual tasks? Are the .pipe()s only available with gulp or how does the assemblerc.yml play together with the javascript stuff?
I'd really love to give this a try. Is there a repo with an example of the new version being used?
no grunt whatsoever
yes, that's correct
(for cli -> need be in assemble-cli to not clutter up system) and assemble locally (/bin shouldn't be in there).
that's the plan, we'll be moving the cli code to assemble-cli before release
I can create an assemblerc.yml which contains the configuration, plugins, middleware, etc.
Yes. We talked about auto-loading this, but for now you can load it using assemble.data('.assemble.yml')
. Or, if you want to use that file for configuration you would need to read it in (and parse it as yaml) and pass it to assemble.option()
how do you configure individual tasks? Are the .pipe()s only available with gulp or how does the assemblerc.yml play together with the javascript stuff?
let me put an example together, that would be more useful for you I think
I'd appreciate this. Thanks.
BTW: I did a showt require(assemble) and started inspecting the objects. Why are there spaces inside object property names such as 'default engines'
?
Why are there spaces inside object property names such as 'default engines'?
that comes from express, which was part of the inspiration for v0.6.0. It's only used on options that are defined like this:
assemble.enable('default engines');
assemble.disable('default engines');
Honestly, we went back and forth about that, but ultimately we decided it's fine (especially for built-in options) since it's far less likely to collide with any user-define values.
Here is the gist I started https://gist.github.com/jonschlinkert/e2da295ec7ca5d159914. Hope this helps. feel free to ask questions and I'll try to fill in blanks.
Probably the most exciting part of v0.6.0 (to me) is loaders, which aren't on the gist yet lol. I'll try to add them in a bit b/c I think they might come in handy with this.
Hi! I've successfully been able to run assemble
with your provided examples but I have trouble specifying default layouts. In particular, how to use layoutdir
, layoutext
? Is it even necessary? I'd like to be able to specify my default layout in the assemble task and then I'd like to also be able to put it in YFM of individual pages
.
Here's what I got so far: https://github.com/vwochnik/assemble-v060-test
Layouts are not being used (since they are not specified) and the file extension is not changed (.md).
that's great!
how to use layoutdir, layoutext
These aren't needed.
default layout in the assemble task
Do:
// project-level default layout
assemble.option('layout', 'default');
// task-level default layout
assemble.task('html', function() {
assemble.src('templates/*.hbs', {layout: 'default'})
.pipe(assemble.dest('_gh_pages'));
});
file extension is not changed (.md).
I might need to whip up a markdown engine :) currently we only use helpers for markdown, but if we're going to do this index page thing, I think it might call for a proper markdown engine.
Alright, I got that step done. How am I now including the body within a given layout?
{{#markdown}}
{{> body}}
{{/markdown}}
isn't cutting it. I get the error Partial body not found.
.
Further, why not just use marked
? Basically, for markdown all you need is
Or am I missing something?
You'll need to use {% body %}
, sorry about that. One sec I'll link to another gist
actually by now you already have all of this done or figured out, but here it is in case it helps
You should still be able to do {{> foo }}
, but also try using {{partial "foo"}}
. either should work, but the second is a helper and will ultimately be more flexible
I got everything working that I had in the old version (ofc not the indices :) ) but how do I change file extension from .md
to .html
? I have tried setting options.ext
to .html
but it doesn't seem to work. Is there a .pipe()
to change file extensions? This would be really awesome because then it would be possible to change the entire target directory structure as per description of a javascript function.
just drop in the extname
plugin, like this:
var extname = require('gulp-extname');
assemble.task('html', function () {
assemble.src('templates/*.hbs')
.pipe(extname())
.pipe(assemble.dest('dist/'));
});
it figures out the right extension to use based on the typeof source file type. it can be overridden but it covers a lot of common formats. we did it this way since assemble can work with any file type, not just html
So, I can say assemble.use(myModule)
and myModule
can then hook to various middleware stages? Also myModule
can then add helpers? That would be really great, 'cause then I could do in a task
assemble.use(indexPagesModule).src("**/*.hbs", {/* assemble + middleware opts */})
.pipe(...)
.assemble.dest("dist/");
is this correct? This would make it really straight forward, even for beginners who barely know to use assemble because the module itself configures itself as a middleware and adds necessary helpers.
Could you then please give me an example using .use()
where all middleware hooks are shown? You can also point me to the location of the source file defining these functions, I haven't found it in the assemble
repository.
EDIT: I saw that .use()
modules are called for EVERY template. IMO, that's not good because when I do index pages I need a hook .postRender()
every template and then one .postRun()
or .postTask()
but I want .use()
only to be executed once for the whole operation. Besides, when I hear the word use
then I intuitively think include
this.
Let's assume the code-snippet above would work, the module could look like this:
// this function is called by the .assembleuse() function with assemble being passed down as an argument.
module.exports = function(assemble) {
// here, we can make hooks, create a collection of contexts, etc.
var indexedItems = [];
assemble.postRender(/.*/, function(context, next) {
indexedItems.push(context);
});
assemble.postRun(function() {
// build index pages by creating a new assemble command chain
});
};
Then,
require()
dMaybe we can rename the current use()
into something else and use .use()
for this purpose.
That's close, but the .use
method is for running at all middleware methods or passing in a router to run on the specified middleware methods on the router.
the middleware methods we have right now are:
onLoad
=> runs after templates are added to the cache.preRender
=> just before the template is renderedpostRender
=> just after the template is rendered.We have another method called .transform()
which does what you're thinking of for .use
: assemble.transform('myModule', myModule);
This was created to transform or add data to the assemble
object, and it can also be used for adding middleware or creating custom templates types or whatever.... it's passed the instance of assemble
as the first argument so you can use it like in your example.
Also, the middleware methods are only run on a "per-template" basis (this might be in a batch when going through assemble
, but each one is checked for each template).
We would recommend making smaller middleware functions that collect information for the collections and export those functions so they can be assigned to a middleware based on an extension:
// index-items-middleware
module.exports = function (assemble) {
var indexedItems = assemble.get('indexItems');
return function (file, next) {
indextedItems.push(file);
};
};
// add the middleware
assemble.postRender(/\.md/, require('index-items-middleware')(assemble));
Then you can create a plugin that runs after the assemble.dest
in the .pipe
methods to use the indexed items:
var through = require('through2');
module.exports = function (assemble) {
return through.obj(function (file, enc, cb) {
// don't push the previous files into the stream
cb();
}, function (cb) {
var stream = this;
// create the index files and push them into the stream
var indexes = createIndexFiles(assemble.get('indexItems'));
indexes.forEach(function (index) {
stream.push(index);
});
cb():
});
};
Now you can use that plugin in your pipeline:
assemble.postRender(/\.md/, require('index-items-middleware')(assemble));
assemble.task('site', function () {
return assemble.src('templates/posts/**/*.md')
.pipe(extname())
.pipe(assemble.dest('dist'))
.pipe(require('index-plugin')(assemble))
.pipe(assemble.dest('blog'));
});
I hope this helps or at least gets you closer. If there's something that needs explained in more detail, let me know.
Following questions:
.transform()
method be used as per-task basis or will the loaded module persist in the assemble
variable during the execution of another task?require()
once and use various parts of that package.assemble.postRender(...).src(...)...;
because I only want the hook in that particular task. Does this make sense?assemble.dest()
multiple times, the cache will be emptied each time?How's this?
var assemble = require('assemble');
var Index = require('asemble-index-builder');
assemble.task('site', function () {
var index = new Index({/*...*/});
return assemble.postRender(/\.md/, index.middleware).src('templates/posts/**/*.md')
.pipe(extname())
.pipe(assemble.dest('dist'))
.pipe(index())
.pipe(assemble.dest('blog'));
});
This usage I find very intuitive.
Can the .transform() method be used as per-task basis
no, transforms are not task-specific.
This might help, Assemble inherits a number of these methods from Template. The tests in template are extensive, they might be useful to look at too.
assemble.postRender(...).src(...)...;
The middleware methods can't be chained with task methods. ... Actually, I just remembered that I wrote an overview that might help more with getting to know the API. Here is the intro to v0.6.0.
If possible I'll try to write more up today and post it when I have something to read.
I think I figured the best way to do modular plugins:
function doStuff() {
}
module.exports = function(app) {
app.transform('template-index', function(app) {
_.extend(app, {doStuff: doStuff});
});
};
That way, I can extend the Template
object (in this case Assemble) and I can create a method for instance app.extractDataFromTemplates(/*data property name*/, {/*options*/})
and another module with app.generateIndex('src/index.hbs', /*data property*/, {/*options*/})
.
In your opinion is this a good way to do this? After all it interferes with the Template
s and Assemble
s namespaces.
Furthermore, I understand that you do have access to the template contexts in the middleware but do you somehow also have access to those contexts by using the stream pipeline? In my through2.obj
flush
function, I do have access to the assemble
object instance (and session
) variables.
assemble.src(...)
.pipe(assemble.storeTemplateContexts(...))
.pipe(assemble.dest(...));
I wouldn't recommend that approach. all the methods needed to do this are all already available, so we should be able to accomplish this without creating new methods.
For example, you could build up a collection of tags
from front-matter in a plugin or a .preRender()
middleware, then in the flush function of a plugin, aggregate the tags and generate a list from a template.
here is a basic example: https://gist.github.com/jonschlinkert/debab424de26a0225cea (edit: I moved the code example to a gist)
I just published an example project too: https://github.com/assemble/assemble-tags-collection-example
Wow! THIS is exactly what I needed. I didn't know file.data
was in the plugin as well.
awesome, hope it helps!
I saw the plugin in lib/plugins
and naturally I tried the following which does not work. The gist you gave had the plugin in the assemblefile.js
which already has an assemble
global variable but external plugins do not.
// plugin function called by `.pipe()`
module.exports = function() {
var assemble = this; // this does not work
return through2.obj(...);
}
It seems within the .task()
function I do have access to this
-> assemble
but it is not passed down the stack. For now, I'm going with require('assemble-plugin-index')(assemble)
try doing it like this:
// my-plugin.js
module.exports = function(assemble) {
return function (options) {
return through.obj(function(file, enc, cb) {
console.log(assemble.views)
// do stuff to file
this.push(file);
cb();
});
};
}
// in the assemblefile.js
var plugin = require('./my-plugin.js')(assemble);
The paths plugin in lib/plugins
uses .call()
where the thisArg
is the assemble
object, which changes the invocation context to the assemble
object. In other words, .call(assemble, foo, bar)
makes this
the assemble
object.
As said, I'm interested in providing a plugin, so here it is: https://github.com/vwochnik/assemble-plugin-index
I've moved the old code to v0.4
branch and created something new there. Please take a look at lib/index.js
and tell me something about the code. I've put some TODO
s in there where I don't know further.
To come
index%%
where %% is pageit looks great! I'll do a pr in a minute
Thanks! I got your pull request! I hope you don't mind me asking a few questions
Thanks for the heads-up!
return through2.obj(function(file, enc, cb) {
// push the whole file through, so we can use any of its properties
files.push(file);
// through2 convention is to `push` the file (above), instead of passing
// it to the callback
cb();
So no this.push(file)
is required? Because files
is ours {}
and not the stream. Also, cb(null,file);
s straight from the docs! I'd keep it that way since it's the most performant and easy to read.
// actually load the template(s) defined by the user. this can be
// a file path, or a glob pattern. the `indices()` method was created
// above for this purpose.
assemble.indices(glob);
But I've specified in the user docs that the user has to do it himself just like the layouts:
assemble.indices('templates/indices/*.hbs');
Or does assemble do it differently? You also say {layout: 'page'}
in the options.
Also, I thought
// -- see above -- assemble.indices('templates/indices/*.hbs');
//...
.pipe(index('posts', {itemsPerPage: 10}))
Furthermore, is it possible if the user put this .pipe()
before .assemble.dest
that the templates passed through are rendered by the assemble core instead of manually by doing assemble.render()
?
Just like you described here
/**
* Now, we will take the context object that we just created in the `tags` loop
* and add it to the `file.data` object of the template. We _could_ instead pass
* the object to the render method, but passing it on `file.data` ensures that it
* will be used as context on this template only.
*/
Let's not focus on the building of the tags
object since I want to make the plugin more versatile but moreso focus on the process.
Within the through2
flush function
// get template
var tmpl = assemble.views.indices[opts.template];
// does this make a copy of the indices template? or is this why the previous `.indices(glob)` call?
tmpl.data = {items: ..., index: ...};
// now you said not to use `.render()`
this.push(tmpl); ?
One more thing: Shouldn't we also pull in session.get('src-opts')
in the options variable?
Looking forward to your comment. BTW, If this skeleton is 'set in stone' once, I think afterwards is going to be relatively easy.
Because files is ours {} and not the stream. Also,
oops, I forgot to re-add this.push(file)
below files.push(file)
. sorry about that. still reading the rest
Further, I would 'forget' about relative links since the template (index template) gets access to the data with the file destination and there the user can either make a relative link or an absolute /link/to/post.html
.
Another thing: I saw that isPartial: true
is default for whatever reason. So in the collection we create, state isPartial: false
?
Also, what do you have with your tags? :D
Index for me
Do you want:
? I think that this is unimportant now but can be accomplished by a grouping function somewhere. Once all data
objects are collected, they can be arranged and grouped how ever desired.
A little more detail:
Plugin does this:
// introduce renderable `index` template collection
assemble.create('index', 'indices', {isRenderable: true});
User does this:
assemble.indices('templates/indices/*.hbs');
User does this:
assemble.task('posts', function() {
assemble.src('templates/posts/*.hbs') // 'layout' can be specified in src.options
.pipe(index('posts', {limit: 10})) /// but 'layout' can be overridden here in options as well
.pipe(assemble.dest('dist/'));
});
Plugin collects all file.data
objects in first function.
In second function, plugin does this: options = {} < defaults < assemble.defaults < session.src-opts < plugin-opts data is list of data (can organize whatever unimportant now)
This is the first parameter. It's not a glob since we're using only one source template but rather a previously loaded template
var tpl = assemble.views.indices[template];
Plugin attaches data tpl.data = ...
But should plugin clone view first?
How to render afterwards? .render()
not necessary?
But I've specified in the user docs that the user has to do it himself just like the layouts:
We just need to make sure that .indices()
or .index()
actually exist when the user define them. Using it in the plugin ensures that the templates will be loaded, but... the user can still do what you defined in the docs as long as it's defined inside that task's function (closure).
cb(null,file);
s straight from the docs!
It is? where? ah wait I just looked, it's here https://github.com/rvagg/through2#options. But, note that the vast majority of gulp plugins I've seen use this.push(file)
. we try to follow gulp convention.
You also say
{layout: 'page'}
in the options.
layout
is a different concept. Layouts are applied at a different point in the build and this option allows the user to define a default layout to use for all templates.
is it possible if the user put this
.pipe()
before .assemble.dest that the templates passed through are rendered
yes, you can do it that way. which reminds me why I didn't push the files through in the main function, since after the dest we only want the index page to render. I just wanted to isolate the functionality to what we're doing.
Let's not focus on the building of the tags object since I want to make the plugin more versatile
These are the guidelines we follow for plugins. My recommendation is to just choose a specific thing for the plugin to do, obviously it's not tags, that was just a convenient example that I think makes sense for this. This way you could easily abstract out the different features into plugins, and if you wanted to you could generalize the logic into utils that can be shared across those plugins. Since the transformations are done in-stream, any impact on build-speed from using separate plugins is minimal. the model works pretty well.
Shouldn't we also pull in session.get('src-opts') in the options variable?
This is already taken care of in stack.js.
and there the user can either make a relative link or an absolute
yes, true. this is what I'm doing in the example. I like that approach better too.
I saw that isPartial: true is default for whatever reason
I can't think of a reason not to have it true
by default, since it allows any template to be included injected into other templates. There are lots of use cases for adding relative links/lists to other pages, rather than generating an index page.
I think afterwards is going to be relatively easy.
:+1: yeah, I think this is why I like focusing on one specific thing first, then I create a couple/few projects based on the same pattern it becomes more clear where logic should be generalized.
let me know if I missed anything!
A couple of comments:
But should plugin clone view first?
If you're going to do the pagination and push each page into the stream, then you need to clone the view and update the cloned data
with information you want for each one. Otherwise, the updates will change the data on the original view.
This is already taken care of in stack.js.
This is only going to happen if you actually use the plugin before assemble.dest
and push the new templates onto the stream. If you decide to use .render
after assemble.dest
and write out the files in the plugin, then you should merge in the src-opts
before rendering.
I would 'forget' about relative links since the template (index template) gets access to the data with the file destination and there the user can either make a relative link or an absolute
This is true when getting relative links in the index template, but if you want to build up a data collection containing links to the index pages so other pages can use them (like showing a list of tags in the right column), then you'll need to queue up or buffer the files (like doing files.push(file)
) and push those back into the stream in the flush function after you can guarantee all the data has been collected.
I think that's all for now.
@doowb do you want to do a new example repo that covers the things you mentioned, like pagination, since mine doesn't cover any of that?
Okay, thank you. Just a couple more:
js var tpl = assemble.views.indices['whatever'];
I forked @jonschlinkert's example repo and added pagination to it before realizing that @jonschlinkert is just showing how to list a collection of items with associated pages. I viewed index pages as a way to show paginated items with their list of pages and also showing paginated pages per item. I can see how they can go together but I'm having a hard time completely decoupling the ideas (when talking about index pages).
The changes I made are here: https://github.com/doowb/assemble-tags-collection-example but it's still limited in how the destination paths are calculated and used as relative paths.
In the example I didn't have to clone the view because I realized that the other way to do it is to split the data out and pass it in as locals
to assemble.render
. This allows rendering the same template with different data. Each page is given a unique file path and pushed back into the stream so it's written out normally.
As @jonschlinkert mentioned, it's best practise to keep plugins simple so a lot of those functions should be split out into modules that can be used in other places and more easily maintained.
I didn't have to clone the view because I realized that the other way to do it is to split the data out and pass it in as locals to assemble.render
did you test that? is that advantageous over putting in on the file.data
object of the file
that is created in the flush function?
I viewed index pages as a way to show paginated items with their list of pages and also showing paginated pages per item. I can see how they can go together but I'm having a hard time completely decoupling the ideas
Imagine 5 plugins in a row only build up collections, the last one generates the actual pages and pagination.
is that advantageous over putting in on the file.data object
That file object is being created after the content has been rendered using the data.
Imagine 5 plugins in a row only build up collections, the last one generates the actual pages and pagination.
I was just thinking that when thinking about how things could be split out. This is where you wouldn't want to render the files in the collection plugin and just add them to the views as a cloned object:
var clone = require('clone-deep');
var tmpl = assemble.views.indices[template];
var file = clone(tmpl);
file.data.tags = buildLinks(tags, files);
stream.push(file);
This is assuming that you're reusing the index template. I think @jonschlinkert was intending that the index template would only be used once for the collection you're building (e.g. tags
). If I'm wrong, please correct me @jonschlinkert.
When I do this.push(tmpl)
no matter whether with the original or cloned template, I'm getting an exception.
/Users/Vincent/Workbench/assemble-v060-test/node_modules/assemble/node_modules/a
ssemble-utils/node_modules/session-cache/node_modules/continuation-local-storage
/context.js:78
throw exception;
^
TypeError: Arguments to path.join must be strings
at path.js:360:15
at Array.filter (native)
at Object.exports.join (path.js:358:36)
at DestroyableTransform._transform (/Users/Vincent/Workbench/assemble-v060-test/node_modules/assemble/lib/plugins/paths.js:30:29)
at DestroyableTransform.Transform._read (/Users/Vincent/Workbench/assemble-v060-test/node_modules/assemble/node_modules/through2/node_modules/readable-stream/lib/_stream_transform.js:184:10)
That's probably because assemble.view.*.*
objects are not a file is it? When I push a new File(), it is working properly but I wanted to try this without the .render()
method.
I've replaced the .render()
block with this:
var file = clone(tmpl);
_.extend(file.data, locals);
this.push(file);
I forgot we haven't made some of the updates in assemble that we made in verb yet. Currently, the objects on assemble.views.*.*
are not vinyl files.
I read through some of the previous comments and I think we got off track from the original idea.
The main idea is to generate multiple pages based on a list of data, right? The list could be pages
, posts
, or something else (tags
).
Should the index/indices
views just be the source of the template to use for all generated pages or
should the assemble.views.indices
contain the template objects that are the generated pages?
If the indices
are just the source templates, where should we add the generated pages?
If the indices
are the generated pages, where should the source template come from?
@jonschlinkert I think these are the types of things template-utils
methods are good for.
I think I got confused somewhere in these comments and I'm just trying to understand what we're solving here (collections or paginated list pages).
I've probably already asked too many questions, but please spare your time once more and look at this code:
https://github.com/vwochnik/assemble-plugin-index/blob/master/lib/index.js
I know the pagination handling and stuff isn't great, but please look at:
template-utils
worker
functionI said I want to create a plugin and i will! I'm looking forward to the v0.6 release!
Great! I'll take a look when I get online
Sent from my iPhone
On Apr 1, 2015, at 11:23 AM, vwochnik notifications@github.com wrote:
I've probably already asked too many questions, but please spare your time once more and look at this code:
https://github.com/vwochnik/assemble-plugin-index/blob/master/lib/index.js
I know the pagination handling and stuff isn't great, but please look at:
Use of template-utils Error handling (so far) Rendering within the recursive worker function I said I want to create a plugin and i will! I'm looking forward to the v0.6 release!
— Reply to this email directly or view it on GitHub.
wow, that looks great! sorry been super busy.
I'll pull the code down and run through it as soon as I have a chance, then I might have more feedback. At first glance though, the only thing that might be good to change is the name of the .page()
method on Paginator. might be better to use something like .item()
, since pagination isn't limited to pages
(and page
and pages
are already assemble methods
Honestly for being new to these libs and assemble v0.6.0, I'm impressed at how many of the nuances you picked up on!
I've seen other people moan about classical v4.2 collections
. Would you consider it a good idea to split asemble-plugin-index
into a plugin to collect and put data into assemble
and the index plugin to take the collected data? Instead of .pipe(index())
you'd have .pipe(data()).pipe(index())
?
Btw, with assemble.data()
I can specify files to contain data but how do I actually access the data during the pipeline? Say I want to insert data and later extract it by another plugin how is that done?
related to https://github.com/assemble/assemble/issues/676