Support folders based on URL

[cfjedimaster]

Given a post with metadata like so:

"link": "http://www.raymondcamden.com/2006/05/18/Round-two-goes-to-the-nerd",

The script should create a folder called 2006 (if it doesn't exist), 05 (ditto), 18 (ditto), and put the file name, Round-two-goes-to-the-nerd.html within that folder.

WP lets you customize your links of course, but I think you could support any /X/Y/TAIL system.

[cfjedimaster]

Note that you may end up needing to make a _data.json for posts, so we have programattic access to the complete list of posts, and one for each folder too so local variables work. I think thats fine imo.

[fenomas]

Hmm.. snazzy idea but I'm not sure how it would work Harp-wise. Can this be done without having multiple copies of the post layout? If posts don't all go in the same directory, and they're to have a different page layout than the root index, then each post folder (2006, 2007, ..) will need its own _layout.jade file, right?

[cfjedimaster]

See the Explicit section - http://harpjs.com/docs/development/layout. We can define it in _data.json.

[fenomas]

Defining layouts in the JSON would mean everything breaks if the user adds a new post and forgets to specify the layout, right?

The only other obvious option I can see would be to have one global layout that inspects its own URL, as in the "Nested Layouts" section of that doc page. But that would be hard to implement automatically, since one doesn't know what the folders will look like (they could be category names, etc).

What if the script exported example.com/2006/05/18/foo to /build_dir/posts/2006/05/18/foo? Then the harp.js setup would be simple and obvious, and if you want to deploy without breaking old links you'd just need a simple rewrite rule /posts/* => /*.

[cfjedimaster]

To be honest, do you really think we multiple layout files? Personally I prefer one file - maybe with a bit of logic (remember Harp passes you a "current" scope that makes it a bit easier to do nested layouts). I don't think your tool needs to handle this automatically. In fact, I think your tool could just one one layout. Let the user decide if s/he needs to do something different (layout wise) on the home page versus a post page.

Another option - use includes. When you write out your files, include a header and footer. Nm - you can't use partials in markdown files.

To your last question - I don't get it. Why is adding /post/ /going to help? Or are you thinking you put _layout in there so they all pick it up?

Another option. Use /_layout.jade for posts and use /_home_layout.jade for index.jade. In _data.json, tell /index.html to use that layout. Ie, you customize the layout for the home page and let the posts uses the default.

On Thu, Dec 3, 2015 at 7:31 PM, Andy Hall notifications@github.com wrote:

Defining layouts in the JSON would mean everything breaks if the user adds a new post and forgets to specify the layout, right?

The only other obvious option I can see would be to have one global layout that inspects its own URL, as in the "Nested Layouts" section of that doc page. But that would be hard to implement automatically, since one doesn't know what the folders will look like (they could be category names, etc).

What if the script exported example.com/2006/05/18/foo to /build_dir/posts/2006/05/18/foo? Then the harp.js setup would be simple and obvious, and if you want to deploy without breaking old links you'd just need a simple rewrite rule /posts/* => /*.

— Reply to this email directly or view it on GitHub https://github.com/andyhall/wp2harp/issues/1#issuecomment-161845846.

Raymond Camden, Developer Advocate for MobileFirst at IBM

Email : raymondcamden@gmail.com Blog : www.raymondcamden.com Twitter: raymondcamden

[fenomas]

I don't have any reason for wanting a posts folder, it's just that all the harp examples I'm working from do that. Ideally I want to export something similar to a Canonical Harp Blog, so users familiar with harp will know exactly what they're looking at and find it easy to start customizing and hacking out what they don't need. If I can do that and preserve wordpress's directories, that will be ideal, I'm just not sure how to do it in a straightforward way. If there's a harp blog somewhere that does what I need to do then please point me at it! :smile:

Now for specifics, yes I figured if the posts all share a folder then they can inherit a layout, as shown in the "multiple layouts" section of the docs you linked. But I'm now thinking that the "nested" section looks better. E.g. the whole site could use one root _layout file, which looks at itself and if it's not the root index it includes a partial meant for posts instead of yielding to the index file. Then the post partial would print out post metadata and yield to the actual post file. Make sense?

[fenomas]

Hmm. I got the above working, which nicely avoids putting any layout information in post folders. But another thought is, wordpress blogs tend not to have a single canonical folder structure. E.g. on mine, example.com/2014/06 lists posts from that month, while example.com/category/foo lists posts in the foo category. So I'm thinking there should be one and only one global post JSON, and index files that know how to index it and pass the correct single object to the post partial.

[cfjedimaster]

I think I may have said this already, but I'm too lazy to ready my earlier comments. ;) Remember you can have N JSON files. One big one that has all the data, one in /2006/10 to represent the month, etc. The files don't get pushed to the static version so there's no reason not to have them if they could possibly be useful.

Another option though is to just write UDFs that help parse. So for example, a function getPostsForMonth. See http://www.raymondcamden.com/2015/11/19/cflib-converted-to-harp-and-on-surge for an example.

On Thu, Dec 3, 2015 at 10:47 PM, Andy Hall notifications@github.com wrote:

Hmm. I got the above working, which nicely avoids putting any layout information in post folders. But another thought is, wordpress blogs tend not to have a single canonical folder structure. E.g. on mine, example.com/2014/06 lists posts from that month, while example.com/category/foo lists posts in the foo category. So I'm thinking there should be one and only one global post JSON, and index files that know how to index it and pass the correct single object to the post partial.

— Reply to this email directly or view it on GitHub https://github.com/andyhall/wp2harp/issues/1#issuecomment-161875936.

Raymond Camden, Developer Advocate for MobileFirst at IBM

Email : raymondcamden@gmail.com Blog : www.raymondcamden.com Twitter: raymondcamden

[fenomas]

Well, you originally suggested having one big global JSON and secondary ones in the subfolders that repeated the same data. I politely ignored that idea because I think you were drunk. ;)

With that said, I got an implementation working that parses all the needed subfolders and serves pages from a single global metadata file. I also played with using a 200.jade file that examines its own path and routes to the right page - this works nicely, and avoids mountains of subfolders. But I assume there's no way to get harp compile to do anything similar. (I guess anyway - the docs don't say a hell of a lot about harp compile except that it exists.)

However, I then realized something rather important - wordpress generally serves links as url/year/month/slug, NOT as url/year/month/slug.html. I don't see how harp can mimic this, unless you were to create a subfolder for every single post, and name all your post files index.md, which would be even hairier than what I have already. This makes me think that converting a wordpress blog to harp (or indeed any SSG) may simply not be possible to do seamlessly without setting up mod_rewrite-style redirects to support the old URLs, and if that's the case I'm thinking this project should keep to a single posts folder and instead output a JSON file of what URL redirects will need to be made.

Thoughts?

(Also: UDF?)

[cfjedimaster]

Surge supports x/y/slug mapping to x/y/slug.html, so if you are ok saying "Use Surge or .htaccess", you should be good.

On Fri, Dec 4, 2015 at 6:13 AM, Andy Hall notifications@github.com wrote:

Well, you originally suggested having one big global JSON and secondary ones in the subfolders. I politely ignored that idea because I think you were drunk. ;)

With that said, I got an implementation working that parses all the needed subfolders and serves pages from a single global metadata file. I also played with using a 200.jade file that examines its own path and routes to the right page - this works nicely, and avoids mountains of subfolders. But I assume there's no way to get harp compile to do anything similar. (I guess anyway - the docs don't say a hell of a lot about harp compile except that it exists.)

However, I then realized something rather important - wordpress generally serves links as url/year/month/slug, NOT as url/year/month/slug.html. I don't see how harp can mimic this, unless you were to create a subfolder for every single post, and name all your post files index.md, which would be even hairier than what I have already. This makes me think that converting a wordpress blog to harp may simply not be possible to do seamlessly without setting up mod_rewrite-style redirects to support the old URLs, and if that's the case I'm thinking this project should keep to a single posts folder and instead output a JSON file of what URL redirects will need to be made.

Thoughts?

— Reply to this email directly or view it on GitHub https://github.com/andyhall/wp2harp/issues/1#issuecomment-161953065.

Raymond Camden, Developer Advocate for MobileFirst at IBM

Email : raymondcamden@gmail.com Blog : www.raymondcamden.com Twitter: raymondcamden

[fenomas]

Using surge is the same as serving from harp, right? In that case using a 200.jade router would let you not worry about folders or index files. (Though I assume most people won't serve their blog from surge, or at least I wasn't planning to.)

[cfjedimaster]

Um... I don't know if harp serve is the EXACT same as Surge. In fact, I'm pretty sure it isn't - but for this particular feature I think it does work in Harp. It should take 2 seconds to take - I have to run the kids to school though. I do NOT think you need a router at all - I think it just works (again, 100% sure in Surge, not sure in Harp)

On Fri, Dec 4, 2015 at 6:51 AM, Andy Hall notifications@github.com wrote:

Using surge is the same as serving from harp, right? In that case using a 200.jade router would let you not worry about folders or index files. (Though I assume most people won't serve their blog from surge, or at least I wasn't planning to.)

— Reply to this email directly or view it on GitHub https://github.com/andyhall/wp2harp/issues/1#issuecomment-161960721.

Raymond Camden, Developer Advocate for MobileFirst at IBM

Email : raymondcamden@gmail.com Blog : www.raymondcamden.com Twitter: raymondcamden

[fenomas]

No, what I mean is that if you serve from surge/harp then a router lets you handle whatever link formats your wordpress blog used without needing dozens/hundreds of subfolders to match the structure of the links. Plus, you can handle /2015/11/foo in parallel with /?p=1234 style links, which in general will be necessary if one wants to convert a wordpress to a static blog without breaking links, I expect.

[cfjedimaster]

Hmm - that makes sense - but as much as I love Surge, I'm not sure I'd want to require you to use the router to get nice urls. I still think if you did /yyy/mm/ddd/slug.html it would be better for more people. But I'm not terribly upset if you don't agree. ;)

On Fri, Dec 4, 2015 at 7:06 AM, Andy Hall notifications@github.com wrote:

No, what I mean is that if you serve from surge/harp then a router lets you handle whatever link formats your wordpress blog used without needing dozens/hundreds of subfolders to match the structure of the links. Plus, you can handle /2015/11/foo in parallel with /?p=1234 style links, which in general will be necessary if one wants to convert a wordpress to a static blog without breaking links, I expect.

— Reply to this email directly or view it on GitHub https://github.com/andyhall/wp2harp/issues/1#issuecomment-161962865.

Raymond Camden, Developer Advocate for MobileFirst at IBM

Email : raymondcamden@gmail.com Blog : www.raymondcamden.com Twitter: raymondcamden

[fenomas]

But do you agree that some sort of routing or URL rewriting is necessary to convert a wordpress blog to static without breaking links, or am I missing something? Creating lots of subfolders covers the case where you serve from surge and don't want to use a router, but it doesn't help if you don't use surge - plus, to avoid broken links you basically need to cover /category/foo and /tag/bar and /?p=123 links anyway, as I understand it.

So I'm thinking this tool would be better off giving you a canonical list of links that wordpress probably used for each page, and letting the user decide what to do with it.

(Incidentally if you personally need the subfolder-writing code, I can put it in a branch if you like)

[cfjedimaster]

"wordpress probably used for each page" But don't we know exactly what it used? I mean from the XML import. I don't want to touch my live site to test, but if you have a test WP install, try changing the URL format and comparing a new export to see if it changes.

In static, we can support /x/y/z format, but supporting /?p=123 would be impossible w/o a server side router of some sort (Surge or .htaccess). We could say, for this tool, that we will support /x/y/z out of the box and anything ?x based would need to have a custom router.

On Fri, Dec 4, 2015 at 8:08 AM, Andy Hall notifications@github.com wrote:

But do you agree that some sort of routing or URL rewriting is necessary to convert a wordpress blog to static without breaking links, or am I missing something? Creating lots of subfolders covers the case where you serve from surge and don't want to use a router, but it doesn't help if you don't use surge - plus, to avoid broken links you basically need to cover /category/foo and /tag/bar and /?p=123 links anyway, as I understand it.

So I'm thinking this tool would be better off giving you a canonical list of links that wordpress probably used for each page, and letting the user decide what to do with it.

(Incidentally if you personally need the subfolder-writing code, I can put it in a branch if you like)

— Reply to this email directly or view it on GitHub https://github.com/andyhall/wp2harp/issues/1#issuecomment-161976330.

Raymond Camden, Developer Advocate for MobileFirst at IBM

Email : raymondcamden@gmail.com Blog : www.raymondcamden.com Twitter: raymondcamden

[fenomas]

The XML tells us (I assume) which link style you have chosen in the settings, but you can still use all the others - like I can go to your blog and enter /slug or /y/m/d or /?p=123 and they work fine. So the only way to really know what links you need to support is to know what links are being used. In my case I use long urls by default but I've always used ?p=123 in tweets, so that's why I want to cover both cases. (Also, in both XML dumps I have, there are one or two posts that only have ?p=123 links. Not sure why.)

But anyway, like I said I don't think we can support /x/y/z format, without assuming either surge.sh or htaccess rewrites. On a generic server all existing wordpress links will break no matter what we do, right? (Unless I export /y/m/d/slug as /y/m/d/slug/index.html that is, which smells pretty bad)

[cfjedimaster]

Sure, WP supports the others, but if the user chose style X, I'd argue we only need to support X when converting to Harp. Anything else is extra and not really necessary. (My 2 cents anyway.)

To your second point, yeah, that is an issue. I know I keep on flip flopping here between saying require Surge and not, but... I'd say require Surge/.htaccess support.

Or, tell users that if they don't use Surge/.htaccess, then when they link to a post, they must include the .html. In theory, thats not too bad as long as it is documented?

On Fri, Dec 4, 2015 at 8:41 AM, Andy Hall notifications@github.com wrote:

The XML tells us (I assume) which link style you have chosen in the settings, but you can still use all the others - like I can go to your blog and enter /slug or /y/m/d or /?p=123 and they work fine. So the only way to really know what links you need to support is to know what links are being used. In my case I use long urls by default but I've always used ?p=123 in tweets, so that's why I want to cover both cases. (Also, in both XML dumps I have, there are one or two posts that only have ?p=123 links. Not sure why.)

But anyway, like I said I don't think we can support /x/y/z format, without assuming either surge.sh or htaccess rewrites. On a generic server all existing wordpress links will break no matter what we do, right? (Unless I export /y/m/d/slug as /y/m/d/slug/index.html that is, which smells pretty bad)

— Reply to this email directly or view it on GitHub https://github.com/andyhall/wp2harp/issues/1#issuecomment-161982813.

Raymond Camden, Developer Advocate for MobileFirst at IBM

Email : raymondcamden@gmail.com Blog : www.raymondcamden.com Twitter: raymondcamden

[fenomas]

Well, this is me-centric but I figure the main goal of fiddling with folders is to try and avoid breaking old links (most importantly the ones already in my old posts). Since there's no simple way to do that, and since there doesn't seem to be any One Obvious Way of setting up partials and layouts to achieve wordpress-style links in harp, I'm leaning towards writing out everything the user needs in the simplest way.

Incidentally I am decorating the outputted JSON metadata with something like wp_subfolders = /2015/11/01/ as I parse, so it shouldn't be too painful for someone to shuffle them into folders if they like. Less painful than for someone else to do the reverse, anyway.

[cfjedimaster]

KK - fair enough.

On Fri, Dec 4, 2015 at 9:25 AM, Andy Hall notifications@github.com wrote:

Well, this is me-centric but I figure the main goal of fiddling with folders is to try and avoid breaking old links (most importantly the ones already in my old posts). Since there's no simple way to do that, and since there doesn't seem to be any One Obvious Way of setting up partials and layouts to achieve wordpress-style links in harp, I'm leaning towards writing out everything the user needs in the simplest way.

Incidentally I am decorating the outputted JSON metadata with something like wp_subfolders = /2015/11/01/ as I parse, so it shouldn't be too painful for someone to shuffle them into folders if they like. Less painful than for someone else to do the reverse, anyway.

— Reply to this email directly or view it on GitHub https://github.com/andyhall/wp2harp/issues/1#issuecomment-161994027.

Raymond Camden, Developer Advocate for MobileFirst at IBM

Email : raymondcamden@gmail.com Blog : www.raymondcamden.com Twitter: raymondcamden

[fenomas]

Thank you for walking me through all the setup stuff about how to structure the directories and metadata!