Automatically generate TOC from folder structure

tulsileathers commented 6 years ago

DocFX Version Used: 2.34 Template used: (default or statictoc or contain custom template) default with small customizations

Actual Behavior: Is it possible to have the toc automatically generated from the folder structure of my documentation? I have a lot of markdown documentation already organized in folders. I would like docfx to simply build a site that uses the existing organization as the toc. It seems like unnecessary busy work to have to add each of my markdown files to a toc file so that they are visible in the UI.

StevenTCramer commented 6 years ago

When docfx generates the following error:

https://github.com/dotnet/docfx/blob/55311684642506280bc8daf02e65745ffa515d76/src/Microsoft.DocAsCode.Build.TableOfContents/TocResolver.cs#L111

Instead of that error it would be nice if we could auto generate the toc from the folder structure.

yufeih commented 6 years ago

How would you like to order the files in TOC?

StevenTCramer commented 6 years ago

@yufeih typically a directory lists alphabetical. Would be fine with me.

yufeih commented 6 years ago

Grav uses numbers in file names determine the order, these numbers are removed in URL:

01. page-1.md
02. page-2.md

Flogex commented 5 years ago

How would you like to order the files in TOC?

What about specifying the position of each item in the TOC using a property in the YAML Header?

---
index: 2
---

And if there are two files with the same index or it is not set, they are ordered alphabetically.

dampee commented 5 years ago

Hi, we are doing something for the Umbraco documenation here: https://github.com/dampee/DocFxTocGenerate

vtouati commented 4 years ago

I'd really like to see this feature added to DocFx. :-)

SpocWeb commented 3 years ago

MS DevOps Wikis generate a File named '.order' per Folder where the File Names are listed URL-escaped, one per Line. I don't know what happens when there are Files missing in this List, by personally I would place those missing last and in alphabetic order.

mtirionMSFT commented 3 years ago

From a large project we've been working on, we've published tools that were created to address these kinds of things. They are on github and can be found on the DocFx Tools page: https://dotnet.github.io/docfx/templates-and-plugins/tools-dashboard.html

segunak commented 2 years ago

MS DevOps Wikis generate a File named '.order' per Folder where the File Names are listed URL-escaped, one per Line. I don't know what happens when there are Files missing in this List, by personally I would place those missing last and in alphabetic order.

It's been years but was about to bring this up. Currently working on a migration project from Azure DevOps Wiki's to DocFx generated web pages and one of the more tedious parts is having to define every single file in a toc.yml for it to render. ADO Wiki's implicitly take the folder structure of your project and render your documents in the same structure. DocFx certainly gives you more granular control but for large projects and teams, having to manage several toc.yml files in several sub folders can be tedious.

avylove commented 2 years ago

I don't mind adding files at the top level, but I find for basic page navigation I have to add entries for each section. TOCs are already implied through headers and a tree can be generated with levels specified by which header designation is used. All that should be required from the user is include files and set the depth of the TOC. This is how it works in Sphinx.

nlyu commented 2 years ago

is there any update in this feature? Would this be a feature in v3? Microsoft Enghub is using docfx and they have implemented their own version of auto toc generation. Would like to see this feature in open source.

saipramod commented 1 year ago

Background

We use a forked version of docfx, with the added implementation of auto toc generation. Its especially helpful for our customers that they donot need to author their own.

the toc's are generated based on the underlying folder structure
.order file is supported and the format of the file is same as that of azure devops wiki

How to use this feature

Update your docfx.json. Create a new element at the bottom named tocGeneration with autoGenerateToc set to true. You may optionally set overrideExistingToc to true to overwrite any existing toc.yml files in the directory structure.

{
    "build": {
      "content": [
        {
          "files": ["**/*.yml", "**/*md", "**/*ipynb"]
        }
      ],
      "resource": [
        {
          "files": [
            "**/*.{png,PNG}",
            "**/*.{jpg,JPG}",
            "Scripts/**"
          ],
          "exclude": ["obj/**", "_site/**"]
        }
      ],
      "dest": "_site",
      "markdownEngineName": "markdig",
      "globalMetadata": {
        "_appTitle": "<example>",
        "_enableSearch": "true",
        "_enableNewTab": "true",
        "_gitContribute": {
          "branch": "master"
        }
      },
      "fileMetadataFiles": [],
      "template": [
        "default"
      ],
      "postProcessors": [],
      "noLangKeyword": false,
      "keepFileLink": false,
      "cleanupCacheHistory": false,
      "disableGitFeatures": false,
      "tocGeneration": {
        "autoGenerateToc": true,
        "overrideExistingToc": false, // stop generating toc.yml for a folder and its subfolder, if one if already present
        "writeToDisk": false, // setting to true would produce the files on disk for customization.
      }
    }
  }

implementation details

the toc's are generated as a part of prebuild for BuildTocDocument and in memory.

yufeih commented 1 year ago

Love the idea of alphabetic by default and a .order file for customization same way as Azure DevOps Wiki. It seems we can break this into some smaller features that is useful on its own:

Autogen TOC title

If we can auto-gen link titles in TOC.yml, it would achieve the same effect as the .order file and simplifies authoring for TOC files in general without introducing additional concepts:

# TOC.yml
- href: a.md # name default to the title of a.md
- href: sub-folder/TOC.yml # name derived from sub-folder name for nested TOCs
- href: sub-folder/ # name derived from sub-folder name for sub-folder reference

For link to content files, fallback the name to markdown title metadata and then h1
For nested TOCs and sub-folder references, calculate the fallback name based on sub-folder name, by replacing special chars such as _ or - with whitespace.

The idea can also be extended to links in markdown files in the future.

Auto TOC

Instead of using a global switch in docfx.json, switch at TOC.yml enables fine-granular control. We can start with a single boolean switch, when enabled, automatically generates the hierarchy for said TOC:

# TOC.yml
auto: true

auto recursively visits sub-directories and generate a single hierarchy based on the directory structure
When a directory does not have a TOC.yml, auto generates items for that directory, sort items alphabetically and embed the generated TOC like nested TOCs
When a directory contains a TOC.yml, auto skips auto-generation and embed the existing TOC hierarchy like nested TOCs
It is illegal for both auto and items to appear at the same time.

saipramod commented 1 year ago

Hello @yufeih,

Auto gen TOC title is a nice idea, so to clarify, generate the name not based on the file name but use the content with in the file like file metadata or h1, like you proposed. That makes sense, just wanted to make sure this feature called so far is nothing to with processing the .order file.
On this point you mentioned "When a directory does not have a TOC.yml, auto generates items for that directory, sort items alphabetically and embed the generated TOC like nested TOCs", i was hoping to clarify.

As far as generating a toc.yml goes, in my very limited experience, users hate having to manually write toc.yml. They make so many errors, hence in my proposal i moved the switch to generate toc to docfx.json.

i have seen generating toc.yml at the level as docfx.json and just 1 level deep seems to worked for a very population of my product users. The reason being the maintenance aspect of multiple tocs and the resulting navigation is pretty clean.

Example of generating toc.yml just one level in and the corresponding navigation

Autotoc gen one level :

navigation:

Please notice how the all the navigation for the folder 1 sticks around and doesnot change since we declared all the subsequent child navigation as children in the same toc.yml and opposed to having individual toc.yml (i.e one for each folder)

is this what you are proposing as well ?

yufeih commented 1 year ago

is this what you are proposing as well ?

Yes! I was hoping in your example:

|- toc.yml
|- docfx.json
|- folder1
    |- toc.yml
    |- index.md
    |- folder1.1
        |- index.md

When the content of folder1/toc.yml is auto: true, it would act as if the content of folder1/toc.yml is:

- name: folder1.1
  items:
  - name: <title of folder1.1/index.md>
    href: folder1.1/index.md
- name: <title of index.md>
  href: index.md

If there is need to reorder a folder:

|- toc.yml
|- docfx.json
|- folder1
    |- toc.yml
    |- index.md
    |- folder1.1
        |- index.md
    |- folder1.2
        |- toc.yml
        |- a.md
        |- b.md

We can do so by adding a folder1/folder1.2/toc.yml with this content:

- href: b.md
- href: a.md

Then when folder1/toc.yml is auto: true, it would act as if the content of folder1/toc.yml is:

- name: folder1.1
  items:
  - name: <title of folder1.1/index.md>
    href: folder1.1/index.md
- name: folder1.2
  href: folder1.2/toc.yml
- name: <title of index.md>
  href: index.md

where href: folder1.2/toc.yml would be expanded by Nested TOC into:

- name: folder1.1
  items:
  - name: <title of folder1.1/index.md>
    href: folder1.1/index.md
- name: folder1.2
  items:
  - name: <title of folder1.2/b.md>
    href: folder1.2/b.md
  - name: <title of folder1.2/a.md>
    href: folder1.2/a.md
- name: <title of index.md>
  href: index.md

saipramod commented 1 year ago

cool ! that sounds like a plan. follow up questions -

if i as a user want to enable this feature for my entire project i.e i want ALL toc.yml to be auto generated. All i have to do is i set auto: true in the root toc.yml and thats it right ? do we have an override flag somewhere, folks who typically want to use this feature will have to otherwise delete all their existing toc's or update their existing toc.yml to say auto:true
Say auto:true NOT in the root toc.yml, but in a toc.yml nested two levels deep. I am assuming based on this proposal, we would populate the toc.yml for that nested folder and recursively populate its children toc,yml as applicable..

yufeih commented 1 year ago

if i as a user want to enable this feature for my entire project i.e i want ALL toc.yml to be auto generated. All i have to do is i set auto: true in the root toc.yml and thats it right ?

Yes!

do we have an override flag somewhere, folks who typically want to use this feature will have to otherwise delete all their existing toc's or update their existing toc.yml to say auto:true

That matches my understanding.

Say auto:true NOT in the root toc.yml, but in a toc.yml nested two levels deep. I am assuming based on this proposal, we would populate the toc.yml for that nested folder and recursively populate its children toc,yml as applicable..

Yes!

saipramod commented 1 year ago

@yufeih sounds good, i understand and agree with the functional aspects of the feature. I will put out a draft pr in the coming week or two and we will go from there.

yufeih commented 1 year ago

@yufeih sounds good, i understand and agree with the functional aspects of the feature. I will put out a draft pr in the coming week or two and we will go from there.

Love it!

connorstorer-kbxcom commented 11 months ago

@yufeih sounds good, i understand and agree with the functional aspects of the feature. I will put out a draft pr in the coming week or two and we will go from there.

update? :D

dotnet / docfx