Tool and Tutorial: migration wordpress to Markdown for Flowershow

[rufuspollock]

When on wordpress i want to switch to flowershow / nextjs because it is awesome so that i can maintain my site more easily etc

After running output is something like this:

about.md
/assets/
/blog/ # blog posts go here
/blog/2018/04/03/xxx-yyy.md

Acceptance

• [x] Analysis with e.g. libraries used and / or script Going with adjustments to https://github.com/lonekorean/wordpress-export-to-markdown
  • [x] Specification of frontmatter fields and specific items e.g. migration of cover image info etc
• [x] Step by step howto 🚧2022-11-21 see https://hackmd.io/Uq2yzonrQn6z2x8WfqjcIA
• [x] Pages ✅ this is working!
  • script outputs page folder with an images folder within it along with all pages.
• [x] Posts
  • [x] frontmatter needs adjustments Done
• [x] Assets
  • [x] Configurable can be e.g. assets/images/ or my-assets/images/ the name of the assets folder is configurable
• [x] Categories added in frontmatter incase we need it

Tasks

Tutorial

• [x] Write out how to do this with a concrete case in mind (e.g. datopian). Bullet points or more that could turn into a tutorial 🚧2022-11-21 https://hackmd.io/Uq2yzonrQn6z2x8WfqjcIA
• [x] Review
• [x] Turn into a tutorial/howto (draft)
• [ ] Finalize tutorial 2022/12/01 In review

Tool

• [x] Identify a script to use ✅2022-11-21 https://github.com/lonekorean/wordpress-export-to-markdown.
  • [x] Fork repo to flowershow (in case we need to make mods) 2022-11-21 https://github.com/flowershow/wordpress-export-to-markdown-plus
• [x] Try it out and make notes on issues
• [x] Create a basic test (e.g. for datopian) - for some parts e.g. frontmatter probably run against "library" part of script
  • [x] Where should posts be located
  • [x] what frontmatter should we have
• [x] Get test passing with e.g. datopian xml export
  • [x] Check flowershow runs with that folder
• [ ] Rinse and repeat

Bonus

• [ ] Add some tests with an example xml export to exporter script (and submit a PR!)

Notes

• what about links to images
• which frontmatter do we need
• any other mistakes in the pages created?
• Do we want the user to do npx flowershow --convert-wordpress ? ❌2022-11-21. No assume this is a standalone script for now so we don't "pollute" flowershow cli with misc stuff

Blog setup in flowershow

Current fields:

• title
• image
• description
• created (date) ?
• layout
• authors (list type ?)
• isDraft (Boolean)

Question

Change authors field type from https://www.contentlayer.dev/docs/reference/source-files/field-types#list

---
authors:
  - philippe
  - ...
---

to

---
authors: ['Philippe', '...']
---

Needed fields

• ...

Other changes

Current script output:

Changes in script

• Either add coverImage field in blog definition type or modify script to change coverImage to image
• categories and tags list has quotes. modify script to change this without quotes ? (follow consistent frontmatter syntax)

• When converting from wordpress, heading links are added in markdown content as [#](...link to heading). convert to only headings, because we have them added by default in flowershow using rehypeAutolinkHeadings.

Assets

Current folder output

Modify script to below changes:

• prepend "assets/" to all image links in markdown
• put all images in one assets folder
• change folder name post to blog
• move everything from page to ./

[PhilippeduPreez]

https://hackmd.io/Uq2yzonrQn6z2x8WfqjcIA

[PhilippeduPreez]

Used this so far Use https://github.com/lonekorean/wordpress-export-to-markdown

[PhilippeduPreez]

Current CLI setup:

[PhilippeduPreez]

@khalilcodes Just some problems that came up while testing the current tool

• Html in some titles (But the OG titles actually have html in them for some reason)
• hardcoded values to OG website like https://www.datopian.com/wp-content/uploads/2021/01/Group-4298_desktop.svg
• Some custom stuff like tables do not convert

[PhilippeduPreez]

@khalilcodes also my headings seems to display correctly when using flowershow, so I think the problem lies in the new datopian version and not the exports.

[rufuspollock]

@PhilippeduPreez

hardcoded values to OG website like https://www.datopian.com/wp-content/uploads/2021/01/Group-4298_desktop.svg

This is why i think brute force migrating wp-content/uploads into the public folder is good idea (i've had this experience before). Can you get a wp-content/uploads export out of wordpress?

I'd also frankly do that with the images if you could and not bother rewriting them.

Some custom stuff like tables do not convert

this is fine. Just flag that in the tutorial / docs.

[khalilcodes]

@rufuspollock @PhilippeduPreez Since wordpress users are already aware of plugins, how about just using a free export wordpress plugin like https://wordpress.org/plugins/export-media-library/ and just putting that in the tutorial section eg. "exporting your wordpress assets" ?

[PhilippeduPreez]

@khalilcodes @rufuspollock I agree that we might need to go the export wordpress plugin route to match the correct folder structure. Exporting the media in xml and using https://github.com/lonekorean/wordpress-export-to-markdown will not produce any images as it only work on exporting "All content"

[khalilcodes]

@PhilippeduPreez pushed the changes to script we worked on. Note that the images folder now generates within the (output) directory. Also please confirm .svg path outputs.

Although we discussed, I'm not convinced if we should be prepending assets like so [](assets/images/...) since the assets folder could be named anything else in flowershow. What do you think ?

Other than that I think the html in frontmatter's title field would be left to look at and then we may have a ready to go script.

[PhilippeduPreez]

@khalilcodes another option to [](assets/images/...) could be to turn image links into wikilinks since they should be picked up anywhere in content folder as far as I understand. Then we don't need to specify an assets folder

[khalilcodes]

another option to could be to turn image links into wikilinks

@PhilippeduPreez that would be digging in more especially since the script uses an xml to markdown parser. Let's keep it default and go with [](images/...) imo.

[PhilippeduPreez]

@khalilcodes just and idea for future users we could ask the name of their assets folder in the wizard and append that before like [ ](name-of-assets-folder/images/...)

[khalilcodes]

@PhilippeduPreez pushed the tests.

[PhilippeduPreez]

@khalilcodes The date frontmatter ordering works, so we don't need to make any changes to the type

[PhilippeduPreez]

• Replaced titles with HTML in them with their post name instead
• Most content seem to convert decently with all image links working
• @khalilcodes will it work if we add /pdf/ to the list of downloaded attachments? Otherwise they will still be linked wrong

[PhilippeduPreez]

@khalilcodes @rufuspollock Links like these [on the value of data](https://www.datopian.com/2022/04/07/on-the-value-of-data/) will still cause problems atm. The tool is able to currently output year and month folders, so maybe we can add a day folder so these links will be functional?

[PhilippeduPreez]

@rufuspollock I confirmed with @olayway that the new flowershow blog setup should handle this correctly, then these links should still work

[khalilcodes]

@PhilippeduPreez I came across two changes and opened a PR for your review on these.

1. Fix to titles being converted directly as slugs as seen below:

   

2. Renaming the coverImage field to image in frontmatter. Since an image field is already present in flowershow blog/page definition, let's use that rather than separately adding a coverImage field in contentlayer.

[PhilippeduPreez]

@khalilcodes I opened a PR for your review on authors being added in frontmatter

[olayway]

@PhilippeduPreez, @khalilcodes please update the tasks in the description and add relevant comments :) @PhilippeduPreez if the PR you created will close this issue you can link the PR by adding Closes #<number of this issue> to its description.

[khalilcodes]

@PhilippeduPreez @olayway @rufuspollock One issue with images I have faced during migration is with the paths to any image files.

In Obsidan, we would write the path to the images like so - ![](assets/images/example.png). This then shows up the image but fails to load on the web since the path resolves as relative. for eg. <img src="https://next.lifeitself.org/<sub-dir>/assets/images/example.png" />

For it to load on the web, we would need to add a leading slash - ![](/assets/images/example.png), but then this fails to load in Obsidian.

To fix this issue we can modify the img tag in MDX.js to add a leading slash for local images (if they don't already have one) like below:

// in MDX.js

const components = {
  img: ({ src, ...rest }) => {
    // check if the image path is external or already has a leading slash
    const imageSrc = src.startsWith("http") || src.startsWith("/") ? src : `/${src}`
    return <img src={imageSrc} {...rest} />
  }
  ...
}

I believe we should backport this which will then allow users consistency as to being able to view the images in Obsidian as well as on the website.

Note: that wiki-links-plus already prepends a slash and that's why we probably never noticed this. This issue is more likely to happen when either converting from wordpress or for users who use the markdown image syntax. IMO consistency across platforms remain.

Let me know your thoughts :)

[olayway]

In Obsidan, we would write the path to the images like so -

Hmmm... but aren't we using this syntax for embedding local images in Obsidian? ![[<image file name>]]

[olayway]

Oh, right, but we probably don't want to convert it to Obsidian wiki links since the user might not be using Obsidian.

[rufuspollock]

@PhilippeduPreez @khalilcodes can you confirm how we handled assets other than images (e.g. pdfs etc) in the wp-content folder? Does that stuff get migrated (especially relevant for life itself migration #261 )

[khalilcodes]

@rufuspollock I had updated the code locally to download pdfs into the assets/ folder (not in assets/images/) and parse the paths in markdown. If that sounds right, I can push the code to the tool and include the pdfs in the PR here https://github.com/life-itself/community/pull/212

[olayway]

Closed with #336