Modifying RSS Feed Path

[Turnerj]

Currently I'm using Statiq.Web as-is with no crazy configurations. I've got my RSS feed generating (feed.rss) however I want to change the name to feed.xml but I'm struggling.

Looking at the source code, I see the GenerateFeeds module has the exact method I want but I'm having trouble getting access to the module. Following the code that calls it, I see it is in a nested ExcuteConfig inside a ForEachDocument module...

https://github.com/statiqdev/Statiq.Web/blob/d9da990fd75b14575d7679273373d7ab5da54b78/src/Statiq.Web/Pipelines/Feeds.cs#L24-L130

I see that the generate feeds module is returned in ExecuteConfig but while debugging my application, I can't find where the GenerateFeeds module ends up.

My thought was to add a post-process module (like the following) to do what I'm wanting:

ModifyPipeline(
    nameof(Statiq.Web.Pipelines.Feeds),
    x.PostProcessModules.Add(new ForAllDocuments
    {
        new ExecuteConfig(Config.FromDocument(feedDoc =>
        {
            // somehow get the `GenerateFeeds` instance
            generateFeeds.WithRssPath(feedDoc.Destination.ChangeExtension("xml"));
        }))
    })
);

I'm open to completely different suggestions for doing the same type of thing though! Really I just want the RSS file named the same so I don't need to update it in a bunch of places, not because I have a preference over the file extension itself.

[Turnerj]

Just thinking about this more - I guess maybe the idea here is to actually remove the Feeds pipeline and add my own?

[daveaglick]

You were right-on if the pipeline had been more straight forward. Unfortunately, the Statiq Web pipelines have gotten a little complicated due to the need to pass and react to lots of configuration at runtime (sometimes even producing entirely different modules). The ExecuteConfig module is a great way to do this, but it means that the modules are coming from a delegate and only instantiated during generation and so they can't really be modified with something like ModifyPipeline() (because there's no modules yet to modify since the ExecuteConfig hasn't run yet).

The balance I've been trying to strike is exposing as much as I can as settings. Those are cheap and easy to introduce. In this case, would a configurable RSS file name/path be helpful? If so I can add it to the next release.

Otherwise you're also on the right track with removing and replacing the pipeline. And since you don't have the same need to support arbitrary feed files, you can add a GenerateFeeds module directly that does exactly what you need.

And for completeness sake, there are probably a few other ways to handle this too. One that I can think of would be to add a module to the end of the existing feeds pipeline in the process phase that changes the document destinations to your liking.

[Turnerj]

You were right-on if the pipeline had been more straight forward. Unfortunately, the Statiq Web pipelines have gotten a little complicated due to the need to pass and react to lots of configuration at runtime (sometimes even producing entirely different modules). The ExecuteConfig module is a great way to do this, but it means that the modules are coming from a delegate and only instantiated during generation and so they can't really be modified with something like ModifyPipeline() (because there's no modules yet to modify since the ExecuteConfig hasn't run yet).

I know its too late to do for v1 but I wonder if a design similar to MSBuild might work - having a Before and After targets and have the system work out an order for modules within a pipeline?

The balance I've been trying to strike is exposing as much as I can as settings. Those are cheap and easy to introduce. In this case, would a configurable RSS file name/path be helpful? If so I can add it to the next release.

It definitely would be in my case BUT I think I've got a bit of an edge case. I think with what settings I can control currently, that would be fine for any new site or blog I build.

Like maybe having control of the extension itself for each type could work but I don't know - I think it would just add extra code that you'd need to maintain.

Otherwise you're also on the right track with removing and replacing the pipeline. And since you don't have the same need to support arbitrary feed files, you can add a GenerateFeeds module directly that does exactly what you need.

And for completeness sake, there are probably a few other ways to handle this too. One that I can think of would be to add a module to the end of the existing feeds pipeline in the process phase that changes the document destinations to your liking.

I'll have an experiment with those suggestions and see what works most easily!

[Turnerj]

I've got it working with the following:

bootstrapper
    .AddPipeline(
        "BlogFeed", 
        new Pipeline()
            .WithDependencies(
                nameof(Statiq.Web.Pipelines.Inputs),
                nameof(Statiq.Web.Pipelines.Content)
            )
            .WithProcessModules(new ModuleList
            {
                new GetPipelineDocuments(ContentType.Content),
                new FilterSources("blog/posts/*/*"),
                new FilterDocuments(Config.FromDocument(doc => doc.Get<DateTime>(WebKeys.Published) <= DateTime.UtcNow)),
                new RenderMarkdown()
                    .UseExtensions(),
                new GenerateFeeds()
                    .MaximumItems(20)
                    .WithRssPath(new NormalizedPath("blog/feed.xml"))
                    .WithFeedTitle("Turnerj")
                    .WithFeedDescription("aka. James Turner - A programmer and entrepreneur with a love of cars, music and technology.")
                    .WithFeedAuthor("James Turner")
            })
            .WithOutputModules(new ModuleList
            {
                new WriteFiles()
            })
    );

Using RenderMarkdown directly was also a lot easier than working out how to get this to run after markdown processing would normally run. Plus I might need to do some weird hacks for markdown on my blog posts themselves - I've committed some crimes against coding by making my markdown generate with tachyon CSS classes. I don't need these classes in my feed.

As an aside, I thought I'd be able to replace a pipeline but there doesn't seem to be an API to actually do that (I got an error adding a pipeline with the same name) so I've just added a new pipeline for my custom feed and removed the config file I had.

[Turnerj]

Actually I needed to include this after RenderMarkdown to fix the URLs in the feed:

new ForEachDocument()
{
    new ExecuteConfig(Config.FromDocument(doc =>
    {
        return doc.Clone(new NormalizedPath("blog") / doc.Destination.FileNameWithoutExtension);
    }))
},

This is what I get for needing a custom markdown renderer for my blog posts!

[Turnerj]

While it might not be widely applicable, one bit of feedback I'd say is that it seems like a lot of ceremony for something that I would imagine could be:

new ForEachDocument(doc =>
{
    // do work
})

There are probably a bunch of good reasons to do it the way you have - it just seems like all of that amounts to what I've written here.

[daveaglick]

I thought I'd be able to replace a pipeline but there doesn't seem to be an API to actually do that

There totally should be - the lack of an easy way to do this is a complete oversight on my part. There should totally be both .RemovePipeline() and .ReplacePipeline() methods in the bootstrapper fluent interface. I'll add those to the next version: https://github.com/statiqdev/Statiq.Framework/issues/198

Actually I needed to include this after RenderMarkdown to fix the URLs in the feed

You might be able to get away with a simple SetDestination module instead of the ForEachDocument/ExecuteConfig above:

new SetDestination(Config.FromDocument(doc =>
    new NormalizedPath("blog") / doc.Destination.FileNameWithoutExtension))

it seems like a lot of ceremony

Good feedback! I'm inclined to think this is probably more on the documentation front because ExecuteConfig with a Config.FromDocument() delegate is essentially ForEachDocument (I.e. you don't really need the outer for-each module in this scenario). It's probably worthwhile to create an entire dedicated docs page to the different control modules and how they work and interoperate for various patterns. It's obvious to me because I've been using it for so long, but that also makes me blind to how confusing the different modules can be to others.

[Turnerj]

There totally should be - the lack of an easy way to do this is a complete oversight on my part.

I'm sure that for most use cases it wouldn't matter but yeah, would be great to have that ability! I tried digging into the code to see how I could and I saw the PipelineCollection did seem to support it but (from memory) the concrete type was internal and the interface was read only.

You might be able to get away with a simple SetDestination module instead of the ForEachDocument/ExecuteConfig

That's good to know!

Good feedback! I'm inclined to think this is probably more on the documentation front because ExecuteConfig with a Config.FromDocument() delegate is essentially ForEachDocument

Yeah - the documentation on one hand has been great for certain things (like finding out about sidecar configuration was awesome amongst some other gems) though there might be some parts where it could be better.

One thing I found is that I jumped a bit between the Framework and the Web sections a bit looking to find some of the information I was looking for. That might be me learning Statiq or it might be a content structure thing - I'm not entirely sure.

When I have some time, I might try and note down a few things and run them by you for potentially adjusting the docs.

I think it might be useful for kinda getting an introduction into modifying pipelines a bit better. Admittedly I may have missed a part in the docs where you cover it really well but some of the combinations of modules etc seem a bit like a blackbox. Like the markdown thing you pointed out in your other comment, it didn't feel intuitive but might be something that clicks once you're more familiar with it - it is just getting from here to there.

Overall though, I've really enjoyed using Statiq and hope to potentially contribute bits here and there to make it the best it can be!

[daveaglick]

I jumped a bit between the Framework and the Web sections a bit…might be a content structure thing

It’s definitely a structure thing. This has been bugging me a lot too, both from a production standpoint and a user of the docs myself. I’ve got some ideas to make it better which essentially all come down to consolidating the docs into a single set with some sort of toggle or duplication for the different projects. I.e. when browsing Statiq Web docs, the Framework docs are there too without having to jump back to them. The entirely new (as yet undocumented) client-side search capabilities should help as well once I get that integrated with the docs site too.

[Turnerj]

Quick update on this, as I've been able to clean up other things in my code, I can now simplify changing the RSS Feed Path.

bootstrapper.ModifyPipeline(nameof(Statiq.Web.Pipelines.Feeds), pipeline =>
{
    pipeline.PostProcessModules.Add(new ForEachDocument
    {
        new ExecuteConfig(Config.FromDocument(feedDoc =>
        {
            if (feedDoc.Destination.Extension == ".rss")
            {
                return feedDoc.Clone(feedDoc.Destination.ChangeExtension("xml"));
            }
            return null;
        }))
    });
})

This change allows me to use the documented way of adding feeds (in my case, a feed.yml file) but also to allow me to change the extension I want. While discovering this, I think I've got a better understanding of where/how the GenerateFeeds module gets "injected" into the process.

While it isn't as elegant as specifying the extension in the feed config, this is quite a bit better than my original solution.

I might submit a PR in the future to support custom destinations for feeds (though knowing that this isn't exactly a common scenario, it might not be worth adding). If you specify FeedRss: my-feed.xml, that will treat it as both true and use the destination. If you specify just FeedRss: true, that will work as current.

[daveaglick]

I really like the idea of toggling output path based on whether FeedRss (and FeedAtom and/or FeedRdf) are either "true" or a non-"true" path. I can see it being helpful even if you don't intend to change the extension but want the different formats to go to different paths:

FeedRss: rss/feed.rss
FeedAtom: atom/feed.atom

And looking at the pipeline, should be able to plug that in fairly easily here in the GenerateFeeds module:

.WithRssPath(feedDoc.GetBool(WebKeys.FeedRss, false) ? feedDoc.Destination.ChangeExtension("rss") : null)
.WithAtomPath(feedDoc.GetBool(WebKeys.FeedAtom, false) ? feedDoc.Destination.ChangeExtension("atom") : null)
.WithRdfPath(feedDoc.GetBool(WebKeys.FeedRdf, false) ? feedDoc.Destination.ChangeExtension("rdf") : null)

I think it'll be straightforward enough that I'd like to get it in while it's fresh. Let me know if you're up for a PR, otherwise I'll go ahead and make the change. Either way is fine - just want you have the opportunity to grab it if you'd like :).

[Turnerj]

I've added a PR for this that seems to be a good fit.