i18n: Add a folder structure option where locale is the root

[chriskirknielsen]

Currently, internationalisation options allow us to store localised files in three structures:

• multiple_folders - persists files in <folder>/<locale>/<slug>.<extension>
• multiple_files - persists files in <folder>/<slug>.<locale>.<extension>
• single_file - persists a single file in <folder>/<slug>.<extension>

Would there be an interest in adding an option such as locale_folders to specify the following structure: <locale>/<folder>/<slug>.<extension>?

For context, I am working with 11ty, and having /en and /fr folders for all of my localised content makes more sense with my configuration than having /pages and /posts folders that contain a folder for each locale. With locales as top-level folders, they can directly pass data down (e.g. /en/en.json data is available in for any file in my /en/pages/* folder, and I can adjust everything for one language in a single folder), which isn't possible with the multiple_folders structure, as far as I know.

I cloned this repo locally and did a quick test, editing the i18n.ts file with the added support I am aiming for — this edit, adding locale_folders into structure options, seems to provide the solution I am looking for, but I am not savvy enough to create all the tests required to form a clean PR for this, though the edits I made are minimal and seem to cover the bases.

Please let me know if I should provide additional details.

[chriskirknielsen]

Took some of my time off to look into this again. The source code has of course changed since my Gist was posted, so I had to juggle around more, as it seems the accessing of files is not as straightforward as I initially thought.

Fork is here — definitely not there yet but I think I got it at a good starting point: https://github.com/chriskirknielsen/netlify-cms/tree/feature/i18n-locale-root

Notable edits:

• i18n.ts: Implemented the LOCALE_FOLDERS options
• config.ts@527: Kept the i18n item in the collection object as it was consistently removed even for an i18n-enabled collection, thus allowing to parse that value with collection.get('i18n').

Notes

I only set this up for the backend-test implementation. I did try adding the locales argument to the other providers but I'm not sure of the best strategy there so I did not commit those changes.

I managed to display the posts but:

• pagination is broken
• when editing an i18n-enabled collection, the localised version is not loaded on the right
• please help out a noob: in listEntries and listAllEntries, I used collection.get('i18n').get('locales')._tail.array to get the array of locales instead of a Redux prop (I assume, likely incorrectly?), but that's not the right way to do it, I'm sure.

I might be understanding the architecture wrong, hence why I'm breaking everything. This is a huge beast, so hats off to the team maintaining this project. 👏

Hoping this might give someone familiar with the project a starting point to get this closer to working!

[erezrokah]

Thanks for digging into this @chriskirknielsen, based on the up votes is seems like a valid use case.

• config.ts@527: Kept the i18n item in the collection object as it was consistently removed even for an i18n-enabled collection, thus allowing to parse that value with collection.get('i18n').

It only removes it from the collection if you don't configure it at the root level.

I only set this up for the backend-test implementation. I did try adding the locales argument to the other providers but I'm not sure of the best strategy there so I did not commit those changes.

The logic for handling i18n should not be implemented per backend (mostly) as those are designed to only handle file operations. i18n logic is common to all backends, hence most of it is in backend.ts

Most of the work is to transform the flat list of files returned from each backend to a structure the CMS can handle and display.

• please help out a noob: in listEntries and listAllEntries, I used collection.get('i18n').get('locales')._tail.array to get the array of locales instead of a Redux prop (I assume, likely incorrectly?), but that's not the right way to do it, I'm sure.

You should be able to use https://github.com/netlify/netlify-cms/blob/4237150beaa4bb08dd6c70caba7ef53dab1dbf25/packages/netlify-cms-core/src/lib/i18n.ts#L33 to get locales information. What you're after is get('locales').toJS() or get('locales'). toArray() since we're dealing with Immutable.js objects.

TLDR: Most of the changes should be done in https://github.com/netlify/netlify-cms/blob/4237150beaa4bb08dd6c70caba7ef53dab1dbf25/packages/netlify-cms-core/src/lib/i18n.ts

[cleeyv]

I have also been working on this problem, in order to make netlify-cms i18n compatible with the multilingual structure of the wowchemy theme for hugo: https://github.com/wowchemy/wowchemy-hugo-modules/issues/2223

I worked on it with a friend who is a javascript dev (@elhil) and the resulting branch is here: https://github.com/cleeyv/netlify-cms/tree/wowchemy-i18n with two commits so far. The first commit is based on the initial work from the i18n.ts gist by @chriskirknielsen and was able to publish new files correctly but not access existing ones: https://github.com/cleeyv/netlify-cms/commit/c49e136d4d57befed7d0dc7dc5ee2458042f4fb6 The second commit makes further changes to i18n.ts to implement a getPathWithI18n function and then uses it in listEntries and listAllEntries of backend.ts to allow access to the existing files: https://github.com/cleeyv/netlify-cms/commit/989a896584c3b288308ea01975b61a4e37e7dc99 @chriskirknielsen, we both seem to have been coincidentally working on the same problem at the same time this past weekend 😊

This achieves the primary goal of accessing/editing existing multilingual files, though I'm still testing it to see if there are other less obvious problems that remain.

One important consideration here is that the structure proposed by @chriskirknielsen in this issue (<locale>/<folder>/<slug>.<extension>) is slightly different than the one required by wowchemy (content/<locale>/<content_type>/<slug>.<extension>). I explain this in more detail in a comment on the wowchemy issue: https://github.com/wowchemy/wowchemy-hugo-modules/issues/2223#issuecomment-828125357 even though the proposed solution here would not actually be compatible with wowchemy, at least a 4 of the 12 upvotes of this issue are from wowchemy users. As far as I can tell the best solution would be to add two new i18n structures, one for 11ty and one for wowchemy/hugo: locale_folders - persists files in <locale>/<folder>/<slug>.<extension> content_folders - persists files in content/<locale>/<content_type>/<slug>.<extension>

Finally, the main developer of wowchemy, @gcushen, has also done some overlapping work on this issue: https://github.com/wowchemy/netlify-cms/tree/feat/hugo-multilingual-content including many tests in i18n.spec.js.

[chriskirknielsen]

Thank you both so much for your input on this!

@erezrokah I have updated my branch and I think it works, aside from the listing of files. But if I create a new post and save it, the window object repoFiles adds the file I created in both the default locale and my added one (fr) — promising! Just seems like I need to figure out how to make it work with list(all)Entries to pull in the default locale posts, currently doesn't like it, despite me adjusting for depth, maybe you have pointers on that…

@cleeyv Looks like I might be able to, in turn, borrow some of your code to get my initial particular case to work. :) Thank you! I will have another look at this. I think we can avoid adding an additional case if we could set the i18n object's folder value to content/{locale}/posts/ for example, and detect that interpolation variable, opening it up to any folder structure we could think of. Food for thought — not sure it's possible with the current way Netlify CMS works!

[chriskirknielsen]

Hi, me again! The updated code is available here: https://github.com/netlify/netlify-cms/compare/master...chriskirknielsen:feature/i18n-locale-root

Thanks to @cleeyv's code I was able to simplify things!

@erezrokah I did make a change in my latest commit — maybe it doesn't make sense, but it does make all the expect files show up when I go to the Posts section. Hoping that this is an edit that won't be necessary for other backend implementations — not sure how one goes about testing those locally.

The unit tests are not written as of yet, as that's a bit outside my comfort zone, but I will try to take a look at it if need be. This thing might break in other contexts!

Don't know how helpful my code is but regardless it's fun to hack at and I appreciate everyone's contributions on this. 😄

[madsem]

actually this already works @chriskirknielsen . I just started a new website on 11ty with multi lingual setup, the folders in my _site dir are setup according to Netlify CMS suggested folder setup for multiple_folders, ie: <collection>/<locale>/

This way I can use the CMS, and then in the build step everything is saved according to the recommended folder structure for i18n setups with 11ty. I.E: <locale>/<collection>/

I was also scratching my head in the beginning, but then realized when you add a permalink key to the content files, they are outout to where the permalink points to.

So what i'm doing now is to add the language data files like _site/pages/en/en.json

With this content: en.json

{
  "dir": "ltr",
  "locale": "en"
}

In the _sites/pages/ directory, I'll add this data file: pages.json

{
  "layout": "layouts/page.njk",
  "permalink": "{{ locale }}/{{ page.fileSlug }}/"
}

index.md files of your collections / category pages, get a locale permalink

permalink: "/{{ locale }}/"

The index.md file for your default languages homepage

permalink: /

Then everything is built into exactly the dir structure we need, and the apex domain mydomain.com shows content in your default language, without /<locale>/ appended to it.

[Anastasiia-XFR]

I do this like:

- label: "NEWS" name: "news" folder: "content" path: 'news/{{slug}}' format: 'frontmatter' create: true i18n: true slug: "news-{{year}}-{{month}}-{{day}}-{{hour}}-{{minute}}"

The post has been added and saved correctly, but it appears with a poor view inside the CMS.