Autogenerated sidebar items: allow more control on output

Josh-Cena commented 3 years ago

🚀 Feature

Several things:

Add exclude option in category metadata to filter out docs;
Add additionalItems option to include more items in autogenerated categories, including link, ref, etc.;
Exclude entire subdirectories when declaring autogenerated sidebar items.

Have you read the Contributing Guidelines on issues?

Yes

Has this been requested on Canny?

No, but there are inline comments asking if they should be allowed, and the answer is yes: I've been asked about this feature.

Motivation

Sometimes we want a fully autogenerated sidebar, but occasionally want to add a few external links in categories. Sometimes we have a legacy directory structure and we only want to generate the sidebar from part of that directory.

API Design

In _category_.json, add the following options:

type CategoryMetadatasFile = {
  label?: string;
  position?: number;
  collapsed?: boolean;
  collapsible?: boolean;
  className?: string;
+ additionalItems?: WithPosition<SidebarItem>[];
+ exclude?: {
+   paths: string[];
+   docIDs?: string[];
+ };
};

paths accepts folder paths (I don't know if file paths would work well; from my experience with the autogenerator code, seems it's not easy since the doc metadata only includes sourceDir?), while docIDs accepts... doc IDs. It's assumed that the members in these two arrays are otherwise included in the category; if they are never included (non-existent IDs/paths not in the autogen dir...), maybe throw an error, or maybe do nothing.

additionalItems accepts doc, ref, link, and even category, but not autogenerated (at least I think it doesn't make much sense, and opens up holes to infinite recursion). Because doc items already come with their own sidebarPosition, they will be sorted well with the rest of the items. However, all items can have an additional sidebarPosition attribute (hence WithPosition<SidebarItem>) to override this behavior.

The autogenerated sidebar item will also accept the exclude and additionalItems properties, because the metadata file in the autogen dir root is not read.

Have you tried building it?

No

slorber commented 3 years ago

I'd like to include more options yes, however this API design is not so simple and I'm not very fan of your current proposal 😅

The API you propose seems influenced by the current implementation details, which IMHO are not great and should be refactored first. Would you design this API the same if autogen sidebars were not implemented?

The last time I thought about it, my idea what just to have an items attribute in _category_.json and it would allow to completely override the autogen for this specific dir for which you would need to be exhaustive, and it wouldn't attempt to more clever things like include/exclude/additional/merge. This may help delay the need for more powerful APIs and could be good enough or at least will give us the opportunity to understand user use-cases better.

Josh-Cena commented 3 years ago

Would you design this API the same if autogen sidebars were not implemented?

Hard to say now I do know what's going on behind the scenes:P But even before I went down that rabbit hole, I was hoping for an include and exclude glob pattern; and in order for that to be extensible to links, I changed include to additionalItems. Hence this proposal.

it would allow to completely override the autogen for this specific dir for which you would need to be exhaustive

What if I want some additional links but keep the rest of the 10 categories + 50 doc links auto generated?

slorber commented 3 years ago

What if I want some additional links but keep the rest of the 10 categories + 50 doc links auto generated?

If we could make the sidebar generation logic more recursive it would allow overriding some behavior on a per-category level more easily.

You can still generate the autogenerated sidebar, and modify/filter it after generation (ie remove subitems, add link in-between 2 existing items etc...).

This is a bit verbose to do for the end-user though. Maybe we should just expose some additional hooks or sidebar modification API, encapsulating the traversal logic?

A powerful API could be to let user add any kind of useful metadata in _category_.json and provide some callbacks like processCategory in the generation logic where the user could read the metadata and add/filter the category items accordingly: the user can creates his own API that suits the use-case

Josh-Cena commented 3 years ago

My mind always explodes when I work on recursion in our sidebar structure :P E.g. do we process the category's children prior to processing itself?

I'll revisit this proposal later since we recently have/will have quite a few changes with the sidebar architecture. Maybe category metadata would be read ahead of time. After a few weeks we may be in a better position to think about the API

slorber commented 3 years ago

Maybe category metadata would be read ahead of time.

Yes, that's also what I think. An empty folder with a _category_.json should probably lead to an empty sidebar category (on which you could add extra links?), but with the current algorithm, it doesn't because there's no doc in this folder and we use doc's paths to infer the sidebar structures.

I figured out the algo impl was not very good a bit later, but as the feature was already working correctly and the public API was fine I shipped it.

Refactoring this algo remains needed 🤯 so that we are more comfortable improving the autogen logic.

lorenzolewis commented 2 years ago

Keeping an eye on this one. Just to help with my understanding:

I have an autogenerated sidebar slice
Inside the directory being pointed to I have an index.md file
Right now that index file is generated in the sidebar (and no way to remove this via the autogenerated sidebar type)
With this feature, I'd be able to add an exclude option (or something similar) to the autogenerated sidebar object to exclude that index.md from being rendered as a child link.

Referencing https://github.com/tauri-apps/tauri-docs/issues/347

slorber commented 2 years ago

@lorenzolewis that looks like a similar case to https://github.com/facebook/docusaurus/pull/5830#issuecomment-1017146250 (except your doc is named index)

You should be able to do something like:

[
  {
    type: "category",
    label: "JS",
    link: {
      type: "doc",
      id: "index"
    },
    items: [
      {
        type: "autogenerated",
        dirName: "js",
        exclude: ["index"]
      }
    ]
  }
];

Eventually, we could figure a way to exclude it automatically, as it looks like a convenient default

We could also allow an include option so that you can generate the sidebar slice one level upper:

[
  {
    type: "autogenerated",
    dirName: ".",
    include: ["js"]
  }
];

Does it make sense?

lorenzolewis commented 2 years ago

@slorber makes perfect sense, thanks!

KatherineWhan commented 2 years ago

@slorber I like the idea of being able to add docs too.

I have just had a read through the whole issue. Will there be any way to sort the sidebar items other than using position in the _category_.json? Ideally sidebar_label alpha would be perfect by default!

I have a workflow where kb articles will be scraped into my repo from up to 20 'source' repos and deposited into the kb folder. It is impossible to get the sidebar_position right and the current default sort is filename alpha which creates 'interesting results' as we have no control over how the teams name their files.

slorber commented 2 years ago

@slorber I like the idea of being able to add docs too.

Add docs where?

I have just had a read through the whole issue. Will there be any way to sort the sidebar items other than using position in the category.json? Ideally sidebar_label alpha would be perfect by default!

I have a workflow where kb articles will be scraped into my repo from up to 20 'source' repos and deposited into the kb folder. It is impossible to get the sidebar_position right and the current default sort is filename alpha which creates 'interesting results' as we have no control over how the teams name their files.

I don't understand what kb is and what you mean exactly here

Please give a very concrete example by providing a file-system tree with files/folders, the config you want to provide, and what the sidebar should look like.

Note that in any case, you can control the sidebar generation logic, including the order, by providing your own generator, or "enhancing" the existing one.

The doc provides an example where the order of the sidebar items is reversed:

https://docusaurus.io/docs/sidebar#customize-the-sidebar-items-generator

// Reverse the sidebar items ordering (including nested category items)
function reverseSidebarItems(items) {

  // Reverse child items of categories
  const result = items.map((item) => {
    if (item.type === 'category') {
      return {...item, items: reverseSidebarItems(item.items)};
    }
    return item;
  });

  // Reverse items at current level
  result.reverse();

  return result;
}

module.exports = {
  plugins: [
    [
      '@docusaurus/plugin-content-docs',
      {
        async sidebarItemsGenerator({defaultSidebarItemsGenerator, ...args}) {
          const sidebarItems = await defaultSidebarItemsGenerator(args);

          // Here, it's up to you to do whatever you want to re-order the sidebar items  
          return reverseSidebarItems(sidebarItems);
        },
      },
    ],
  ],
};

Now it's up to you to provide your own ordering logic:

you can create exceptions and decide to reverse only a category that has a specific label
you can invent your own frontmatter and use args.docs + sidebarItems to compute the new order based on your custom frontmatter

Somehow, this issue can already be worked around by writing your own sidebarItemsGenerator that include/exclude generated items. We are just looking at providing convenient config so that you don't have to write such logic yourself.

dwmkerr commented 2 years ago

Can I suggest we also look at this comment from @slorber:

https://github.com/facebook/docusaurus/issues/3464#issuecomment-820336747

tl;dr

module.exports = {
  mySidebar: [
    {
      type: "autogenerated",
      dirName: "tutorials"
      include: "**/easy-*.md"
    },
    {
      type: "autogenerated",
      dirName: "tutorials"
      include: "**/easy-*.md"
    }
  ]
};

This would be amazing for me - my book Effective Shell uses a folder per article:

Unless I have missed something, this nesting structure means my sidebars have to look like this:

anotherSidebar: [
    'index',
    {
      type: 'category',
      label: 'Transitioning to the Shell',
      link: { type: 'doc', id: 'transitioning-to-the-shell/index'},
      items: [
        'transitioning-to-the-shell/getting-started/index',
        'transitioning-to-the-shell/navigating-your-system/index',
        'transitioning-to-the-shell/managing-your-files/index',
        'transitioning-to-the-shell/clipboard-gymnastics/index',
        'transitioning-to-the-shell/getting-help/index',
        'transitioning-to-the-shell/the-renaissance-of-the-shell/index',
      ],
    },

If instead I could do this:

      type: 'category',
      label: 'Transitioning to the Shell',
      link: { type: 'doc', id: 'transitioning-to-the-shell/index'},
      include: "**/index.md"

That would be amazing

slorber commented 2 years ago

@dwmkerr I'm not sure to understand: what's the result you currently have, and what's the result you want?

If you are using autogenerated sidebars, why do you care about include? do you want to use it through the category.json file or through explicit config?

It looks like you display twice the same sidebar in your comment, maybe a typo?

dwmkerr commented 2 years ago

Sorry there was mistake - copy/paste error. I've fixed it now. Because each page is in a subfolder, I cannot autogenerate the pages without each one sitting in it's own category. What I'd like to be able to do is have a single category, such as "Transitioning to the Shell" and have a flat set of pages in this category which are populated using a globbing pattern that finds all of the pages in the child folders

Josh-Cena commented 2 years ago

@dwmkerr Have you tried beta.16? In the latest version, a folder containing only an index.md should already generate a single doc link instead of a category.

alexfornuto commented 2 years ago

I would like to see exclude added to either _category_.json or sidebar configs. I have directories in /docs that I want to be excluded from the main sidebar. I link to them in the header and they use their own sidebar once loaded.

dwmkerr commented 2 years ago

@Josh-Cena yes I'm actually on beta 20, but still have the issues of the 'index' file being included when we auto-generate items and then also the subfolder issues, so the wildcard option plus the 'exclude' options could still be a massive improvement for me!

Josh-Cena commented 2 years ago

@dwmkerr Are you running into this bug? https://github.com/facebook/docusaurus/issues/7348 Try with canary?

dwmkerr commented 2 years ago

No I'm afraid not - I have a category but it there are also items

Josh-Cena commented 2 years ago

Ah, so it's the problem we are talking above. No problem—it will be the top use-case for this feature.

azinit commented 2 years ago

Still actual!

lukaszch commented 2 years ago

Is the exclude from autogenerated sidebar option already implemented? I tried it, but it doesn't work in the 2.1 version. I have only a docs-based site, where one of the docs pages is the initial page, which works great. I only cannot find a way to hide it from the autogenerated sidebar.

slorber commented 2 years ago

If this issue is not closed, this means it's not implemented not released

I have only a docs-based site, where one of the docs pages is the initial page, which works great. I only cannot find a way to hide it from the autogenerated sidebar.

Interesting use-case to consider, thanks for reporting

blessanm86 commented 2 years ago

We have docs in multiple repos. So we use git submodules Maybe I'm doing things wrong

This is our folder structure

docs/
├─ guides/
│  ├─ getting-started.md
│  ├─ contributing.md
├─ submodules/
│  ├─ request/
│  │  ├─ README.md
│  ├─ product-analytics/
│  │  ├─ README.md
├─ awesome-lib/
│  ├─ README.md

I am trying to generate a sidebar structure of

Guides
├─ Getting Started
├─ Contributing
Request
Product Analytics
@internal/awesome-lib

I am just trying to remove the submodules folder from showing up. My sidebar config looks like this

{ type: 'autogenerated', dirName: '.' },
{
   type: 'doc',
   id: 'submodules/request/README',
   label: 'request,
}

I was hoping for something like { type: 'autogenerated', dirName: '.' , exclude: ['./submodules/**/*.md']}

@internal/awesome-lib is a docs generated via typedoc plugin.

Everything works as expected, just want to filter out the submodules category from the sidebar.

hrumhurum commented 1 year ago

In one of the projects I have a page that should not be visible in a sidebar, but the page still should be associated with it. Allowing a sidebar label to be an empty string in frontmatter of the page perfectly communicates the intent:

sidebar_label: ""

This is not strictly an exclusion, it just makes the corresponding sidebar item invisible while keeping the page <-> sidebar association active.

A use case: some pages should not be indexable or observable in navigation because they are shared by direct links only. At the same time, those "secret" pages belong to a larger resource such as Knowledge Base, meaning that the corresponding sidebar should remain visible for such pages to allow further navigation.

MattiasBuelens commented 7 months ago

You can implement something like this using a custom sidebar items generator:

async function postProcess({ item, ...args }) {
  if (item.type === 'category') {
    // Recurse through children
    for (const childItem of item.items) {
      await postProcess({ item: childItem, ...args });
    }
    // Add additional items
    if (item.customProps?.additionalItems) {
      for (const { position, ...additionalItem } of item.customProps.additionalItems) {
        if (position !== undefined) {
          item.items.splice(position - 1, 0, additionalItem);
        } else {
          item.items.push(additionalItem);
        }
      }
    }
  }
}

export default async function sidebarItemsGenerator({ defaultSidebarItemsGenerator, item, ...args }) {
  const sidebarItems = await defaultSidebarItemsGenerator({ item, ...args });
  for (const item of sidebarItems) {
    await postProcess({ item, defaultSidebarItemsGenerator, ...args });
  }
  return sidebarItems;
}

See sidebarItemsGenerator.ts for the full code, and this _category_.json file for an example usage. It's not perfect (the position handling is very hacky), but it does the job. 😁

facebook / docusaurus