Allow `blog.sources` to accept arbitrary keys and multiple values

[lolmaus]

Hi!

I would like to have the following blog.sources structure:

blog.sources = "content/:category/:title.html"

This results in an empty blog.articles array.

I also tried blog.sources = "content/:category-:title.html" but it didn't work either.

It seems that whenever blog.sources contains a token other than :year, :month, :day or :title, no blog files are found.

If this is a bug, please fix it. If this is an expected behavior, please consider this to be a feature request. blog.sources should behave identically to blog.permalink which does allow arbitrary tokens.

Updated description (08:02 November 25, 2013):

• Please allow blog.sources to accept arbitrary tokens for directory names, allowing to distribute blog posts among multiple directories. The directory token should be available in article metadata.
• Please allow blog.sources to accept an array of multiple paths, allowing to distribute blog post files among different directories with different structure.
• Please make the :title token available as article metadata.

[tdreyno]

Permalink is able to have arbitrary sources because the data is known. Sources needs to know how to find files on disk. How would something like :category expand? Match any string? Would this match somehow become available to the template when rendering?

[lolmaus]

As i see it, :category in the above example should match full folder names. E. g. this configuration:

blog.sources = "content/:category/:title.html"

...should match:

• source/content/news/foo.html.md (category: "articles");
• source/content/product-decriptions.html.rb (category: "product-decriptions");

...should not match:

• source/content/bar.html.md;
• source/content/news/2012/baz.html.md.

The matched tokens should be available within templates just like other metadata, e. g. :title, :date.

Just imagine the power of this:

<% articles.select{ |a| a.category == 'news' }.each do |a| %>

The "maximum programme" for this feature request is ability to provide more than one source pattern, e. g.:

blog.sources = {
  news: "content/news/:year-:month-:day-:title.html",
  articles: "content/articles/:topic/:author/:title.html"
}

Sources' keys (:news and :articles in this example) should also be available as, say, articles[n].data.type.

[tdreyno]

Sounds reasonable.

[lolmaus]

Yay! Any chance for middleman-blog maintainers to take up?

[tdreyno]

@bhollis is the primary maintainer, but he's a busy man. Pull requests (and pull requests of test cases) would help a lot

[lolmaus]

I heard something of multiple blogs implementation, though i failed to find any documentation and release status for that feature.

Does it overlap with subj?

[bhollis]

I've been away for a bit - what is :category?

[lolmaus]

It's an example of an arbitrary key. I suggest that user should be allowed to put any key names he desires into blog.sources. Those keys should automatically be filled with values from file and folder names.

[bhollis]

Hm. Well I could definitely be OK with an implementation that could handle arbitrary tokens, but I'm not 100% sure how to do it nicely. I'd be happy to look at a PR though.

[bhollis]

This has been added in master. Simply refer to your custom data key in the :permalink URL template and it'll get used.

[tdreyno]

:beers:

[lolmaus]

So blog.sources = "content/:category/:title.html" no does not fail.

But :category is neither available in current_page/current_article, nor in the permalink setting.

I try this:

activate :blog do |blog|
  # set options on blog
  blog.sources = "content/:category/:title.html"
  blog.permalink = ":category/:title.html"
end

…and the blog posts are avaliable as :title instead of :catergory/:title. Here's a quotation from localhost:4567/__middleman/sitemap:

contentblogsfoo.html

Path content/blogs/foo.html Build Path foo/index.html URL /foo/ Source File d:/Dropbox/Code/Sandbox/middleman-blog-test/source/content/blogs/foo.html.md.erb Data ["date", Sat, 02 Nov 2013] Options [:lang, :en]

newsbar.html

Path content/news/bar.html Build Path bar/index.html URL /bar/ Source File d:/Dropbox/Code/Sandbox/middleman-blog-test/source/content/news/bar.html.md Data ["date", Fri, 05 Apr 2013] Options [:lang, :en]

[bhollis]

Let me see if I'm understanding you right - are you asking for the custom token (category in this case) to be written back into the article data?

[bhollis]

OK, 0811e2e76b82b9ae38221df70b47fac668a527a8 implements that. Your example should work fine now.

The only weird part is that you have to use current_article.metadata[:page]['category'] to access the information in your templates (or filter on the value in a select). That's because data is only frontmatter and we can't add on to it.

@tdreyno maybe we should add a new method to Resource like #page_data that provides a more direct route to the page-specific metadata? It's kind of confusing, though, because there's already data, and because the frontmatter data gets written into metadata[:page] as well. The difference is subtle (extensions can add on / override metadata[:page] but not data).

[lolmaus]

Affirmative. I would like directory tokens be available both for blog.permalink and in article data. The :year, :month and :date tokens already do that, appearing as article.data.date. (Why can't i use a :date token in the blog.sources instead of those three?)

I'm imagining this:

%ul
  - blog.articles.select{ |article| article.category == 'news' }.each do |article|
    %li<
      = link_to article.title, article.url
      = ", " + article.date.to_s

Ultimately, blog.sources should be capable of accepting an array of multiple paths.

Also, there's a confusion with the :title token. I thought that it is the same thing as its frontmatter namesake, but they appear to be different thing. I'd suggest the title token to be renamed to :id and also made available in article data and blog.permalink. But it should still be possible to use it as :title so that it doesn't break existing projects... scratches head

[lolmaus]

@bhollis, wow, you're coding faster than i'm expressing my request! Let me check it out.

[lolmaus]

@bhollis, it works and it is pure awesomeness.

Can you please comment on the following:

1. Will this new functionality make it into official documentation? Hiding it from the public would be a crime.
2. Do you feel like implementing blog.sources to accept an array of multiple paths?
3. Do you feel like allowing to use an exposable :id token instead of the non-exposable and confusingly named :title token (i i'm not suggesting to remove the latter)?
4. I've seen people discussing multiple blogs functionality for middleman-blog. I failed to find any documentation so i don't know how multiple blogs work. Ain't what you've just implemented (together with blog.sources accepting an array) equal to multiple blogs in functionality but drastically easier to use?

[lolmaus]

Hey @bhollis, to show my appreciation of your kind support, i've set a beer-worth tip on FreedomSponsors:

http://freedomsponsors.org/core/issue/410/allow-blogsources-to-accept-arbitrary-keys-and-multiple-values

I've edited the initial post accordingly.

Also, i've figure out that question number three from my previous comment does not pose a problem: article.title will not conflict with article.metadata[:page]['title']. The problem of confusion between two titles still exists.

[bhollis]

Will this new functionality make it into official documentation? Hiding it from the public would be a crime.

Sure. I have a pending update to the docs already.

Do you feel like implementing blog.sources to accept an array of multiple paths?

Perhaps - you'd have to explain to me what that would get you, preferably in a separate issue. My suspicion is that multi-blog support will already do this for you.

Do you feel like allowing to use an exposable :id token instead of the non-exposable and confusingly named :title token (i i'm not suggesting to remove the latter)?

Not really. The title, un-parameterized, is not suitable for URLs. All that's happening is the title is getting transformed into a "slug" (this is accessible via current_article.slug vs current_article.title. I do wish we'd named the path variable "slug" instead of "title" to avoid confusion, but at this point it's too much trouble to change.

I've seen people discussing multiple blogs functionality for middleman-blog. I failed to find any documentation so i don't know how multiple blogs work. Ain't what you've just implemented (together with blog.sources accepting an array) equal to multiple blogs in functionality but drastically easier to use?

No, multi-blog allows for multiple blogs that have totally different functionality. What I've done allows for a lot of the flexibility of multiple blogs, but for example you can't use totally different permalinks structures or different templates, or what-have-you. I just posted a PR for updated blog documentation that includes multi-blog stuff, so hopefully that'll help.

[bhollis]

Thanks! I've never used FreedomSponsors - I just signed up.

[lolmaus]

The title, un-parameterized, is not suitable for URLs. All that's happening is the title is getting transformed into a "slug" (this is accessible via current_article.slug vs current_article.title.

Slug is fine! One item less on the list.

Perhaps - you'd have to explain to me what that would get you, preferably in a separate issue. My suspicion is that multi-blog support will already do this for you.

I've already linked the tip here so please excuse me for not filing a separate issue.

My idea was to be able to create different content types (e. g. products, services, news, team members, etc) and be able to have relations between them. You have already made it possible:

%h1 List of trainers and their pokemon

- trainer_articles = blog.articles.select { |article| article.metadata[:page]['category'] == 'trainers' }
- trainer_articles.each do |trainer|

  %h2= trainer.title
  = link_to 'View profile', trainer.url

  %h3 Pokemons in posession:

  %ul
    - trainer_pokemon_names = trainer.data.pokemons.split(',').map(&:strip)
    - trainer_pokemon_articles = blog.articles.select { |article| article.metadata[:page]['category'] == 'pokemons' && trainer_pokemon_names.include?(article.slug) }

    - trainer_pokemon_articles.each do |pokemon|
      %li= link_to pokemon.title, pokemon.url

I knew nothing about how multiple blogs would work. I decided that it was to create multiple websites from a single project codebase and that it is too complicated.

Now when i've read your documentation update i see that the multiple blogs functionality seems to be very narrow and simple...

...and that the feature that you've just kindly implemented duplicates multiple blogs functionality. This

blog.articles.select { |article| article.metadata[:page]['category'] == 'news' }.each do |article|

seems substitutable with:

blog('news').articles.each do |article|

I asked for blog.sources to accept an array because i wanted to have different file/folder structure for different content types. E. g.:

blog.sources = {
  news: "content/news/:year-:month-:day-:title.html",
  articles: "content/articles/:topic/:author/:title.html"
}

I'm still not sure whether multiple blogs are a true replacement for arbitrary tokens. Please clarify this.

There's one more thing that i don't understand about multiple blogs: can you iterate blog names? E. g.:

%h1 Alphabetical list of content types

- article_types = []

- blog.articles.each do |article|
  - article_types << article.metadata[:page]['category']

- article_types.sort_by! { |name| name.downcase }

= article_types.join(', ')

[bhollis]

I think your news/articles example would be better done as two blogs, each with its own source format. You can always map their output to the same style file.

As far as iterating blog names, it's no problem:

blog_instances.each do |blog_name, blog|
  puts "Blog #{blog_name} has #{blog.data.articles.size} articles!"
end

I'm not sure I understand your example at the end, but assuming article_types is just the different categories, you could write:

article_types = Set.new(blog.articles.map {|a| a.metadata[:page]['category'] }

Does that help?

[bhollis]

There's another feature (that I've just enhanced a bit) that might be useful - "custom collections". So you could do:

activate :blog do |blog|
  blog.sources = '{category}/{year}-{month}-{day}-{title}.html'
  blog.custom_collections[:category] = {
    link: 'categories/{category}.html',
    template: 'category.html'
  }
end

And that'd automatically make a category page for each category. Inside the category template, you will have a local variable called category and a local variable called articles, so you can easily make a summary page.

[lolmaus]

This is awesome, @bhollis! Please mark the issue as finished on FreedomSponsors.

Lack of distinct documentation for such features as current_article.slug, blog_instances and blog.custom_collections are a huge issue. Functionality exists but nobody knows how to use it. The documentation on middlemanapp.com looks more like a tutorial than a complete documentation. One has to spelunk into Ruby autodocumentation (which is cryptic) or Padrino documentation which seems to fill some gaps.

I believe there's a demand for a reference documentation in addition to the tutorialish middlemanapp.com documentation. Do you think it is worth opening an issue ticket to start a discussion?

[tdreyno]

http://rubydoc.info/github/middleman/middleman-blog/frames

[lolmaus]

@tdreyno, shouldn't there be explicit reference documentation formatted for people, not robots? Auto-generated documentation contains all the stuff mixed together (and even mixed with internal Ruby stuff like inspect).

It is impossible to figure out what functionality Middleman offers by looking at the list of methods and attributes. To obtain that understanding, you have to actually study every item on the list of methods and attributes and sort it out in your head (or even take notes on paper).

I believe that there should be reference documentation written by humans for humans. It should contain desctiption of every Middleman method, attribute, helper, etc — sorted into reasonable categories, so that it is easy to grasp what features exist for every category, find documentation on desired features, etc.

A good example is Padrino documentation. We need the same quantity and quality of documentation for Middleman and all its subsystems, including middleman-blog.

Tutorialish middlemanapp.com docs are not enough. Currently Middleman users have to perform detective investigations of auto-generated documentation, Github issue queues, Stackoverflow and Google just to understand how Middleman and its subsystems can actually be used.

[lolmaus]

Just an example that autogenerated documentation is cryptic:

- (String) slug The “slug” of the article that shows up in its URL. http://rubydoc.info/github/middleman/middleman-blog/Middleman/Blog/BlogArticle:slug

How one supposed to know that this actually means data from the :title token of blog.sources?

[bhollis]

Unfortunately, writing good documentation is difficult and time consuming. We do what we can in between writing the actual features, but we could always use help. We also tend to forget what people do and don't know because we wrote the code in the first place.

Both the YARD documentation and the middlemanapp.com site are easy to update through the GitHub UI without even having to check anything out. I'd encourage anyone who's having trouble to contribute extra detail to the areas they had trouble with.

[lolmaus]

I've got a taste to writing stuff and i believe that i can do good documentation. Unfortunately, i understand very little in Middleman so i can basically update documentation only for those things that you've clarified for me, e. g. slug.

I'm also eager to enhance middleman-guides with some usage examples that i've found, for example creating a relation between different blog types (see my pokemon/trainers example above, but implemented with multiple blogs).

Here's another use case i would like to document. I discovered that it is possible to pass blocks into partials:

Partial _test.haml:

- if defined? block
  = block
- else
  Welcome, friend

Page index.html.haml:

// Calling the partial without a block
= partial 'test'

// Preparing a block
- captured_html = capture_html do
  %h2 Wassup!

// Calling the partial with a block
= partial 'test', locals: { block: captured_html }

It even allows passing multiple blocks to a single partial call, awesome! I wish i could have just read about that rather then playing Sherlock Holmes to investigate how it could be done (though it was undoubtedly fun!).

So i could start filing pull requests and you just reject or edit them if you don't like my writing. Are you fine with that?

Also, don't expect much from me. I only can promise that i eventually might do an edit or two, okay?

[bhollis]

Yeah, pull requests sound great, thanks!