Check in post-commit docs assets as Markdown to docs repo for docs team to control styling

[tt-rkim]

As discussed with @jwilde-tenstorrent , we will be transferring docs and styling ownership to the docs team. Their publishing model is:

• building our assets as Markdown (https://pypi.org/project/sphinx-markdown-builder/)
  • will need to consider what to do about versioning
  • will need to consider fixing and merging #5037
• Move youtube videos onto Tenstorrent youtube and embed rather than using checked in file
• checking in these assets using versioning script that docs team provides via GitHub write token

Docs pipeline for pybuda as reference: https://yyz-gitlab.local.tenstorrent.com/tenstorrent/pybuda/-/blob/main/docs/CI/.gitlab-ci.build-docs.yml?ref_type=heads

cc: @adifrancescoTT @TT-billteng @dimitri-tenstorrent

[tt-rkim]

Sent in two tickets to IT:

• re-generate token for write / re-work the publishing model
• give dev access to metalium-admins for docs repo

cc: @warthog9

[tt-rkim]

@adifrancescoTT @jwilde-tenstorrent

Current action items are:

• create markdown output for metal sphinx build
• @jwilde-tenstorrent and @tt-rkim to check in End of week

[tt-rkim]

https://github.com/tenstorrent/docs-test is the repo we need to use

[tt-rkim]

We've run into a blocking issue with the open source markdown file builder for Sphinx. Looks like someone with the exact same build flow is running into the same issue: https://github.com/liran-funaro/sphinx-markdown-builder/issues/21

Such is the woes of working in a feature-rich, language-rich (C++ and Python) API like tt-metal.

As referenced in: https://github.com/liran-funaro/sphinx-markdown-builder/issues/21#issuecomment-2106151098 This is a feature request, and likely won't be done any time soon. Let us know if we should devote more resources into trying to get this working, or if docs team and metal infra team can change either/both of our docs build processes to get around this issue.

cc: @jwilde-tenstorrent @adifrancescoTT

[tt-rkim]

@dimitri-tenstorrent For your future eyes.

[tt-rkim]

Pushed interim work to branch rkim/8534-markdown

1. build_metal.sh w/ python env
2. cd docs
3. make markdown

You see the error.

This is because of the above.

[dimitri-tenstorrent]

I will try to replicate and see if I can identify a path towards the fix.

[dimitri-tenstorrent]

I have identified the issue and will work on figuring out if I can create a patch for Breathe to handle the missing definitions:

/Users/dgnidash/projects/tt-metal/docs/source/tt-metalium/tt_metal/apis/host_apis/buffers/AssignGlobalBufferToProgram.rst:5: WARNING: unknown node type: <desc_signature_line: <target "host__api_8hpp_1a4e29ca9df8b6f13424f2a392d89714 ...>

Exception occurred:
  File "/Users/dgnidash/projects/.tt/lib/python3.9/site-packages/sphinx_markdown_builder/contexts.py", line 339, in make
    assert len(content) > 0, "Empty title"
AssertionError: Empty title

[dimitri-tenstorrent]

I have been able to generate markdown and integrate it into the docs site. I need to do these additional tasks

• Verify and fix issues
• Build a tt-metal doc deployment pipeline
• See if the C++ function highlighting needs to be fixed (see screenshot)

[tt-rkim]

General

• There are no dropdowns at all. I think @dsklavos mentioned this. First pic is docs test, second pic is original docs.

[1] Docs test

[2] original

TT-Metalium

• Change the upper corner to "TT-Metalium"

---

• We could consider getting rid of this link to models in ttnn. In fact, I might even consider getting rid of that whole page. It has models using the tt_lib API, which I think we should not advertise widely to users:

---

• For these overloaded functions with multiple "versions", we may need something more clearly showing that these two different sections. They're a little close together. On the original docs, shown in 2nd pic, they have blue headings.

[1] docs test

[2] original

TT-NN

• General point: It is TT-NN, not TTNN, if using the name in regular text. In code, it's ttnn. Sorry for the confusion 😅 You're going to have change it in certain places like the upper left hand corner.

---

• The original documentation has the jupyter notebooks in ttnn/tutorials rendered. This is not the case in docs-test. First pic docs-test, second pic original. Unfortunately, this is likely something we didn't take into account when considering this original team docs -> markdown -> build sphinx again flow.

[1] docs test

[2] original

---

• The original documentation is able to render tt_lib functions, their arguments, and docstrings. The new doesn't seem to render anything at all. Again a markdown builder plugin issue. First pic is docs test, 2nd is original

[1] docs test

[2] original

---

• The original documentation is able to render the original docstrings. This is the "C++ bindings" issue I discussed earlier, and is reminiscent of this issue: https://github.com/liran-funaro/sphinx-markdown-builder/issues/19. I think it may be the same issue as the above one.

[1] docs test

[2] original

[dimitri-tenstorrent]

@tt-rkim @TT-billteng

Thank you so much for reviewing the result! This is a great review. I have done a lot of experimentation on the weekend and was able to make progress with the tutorials and I believe there is a path to fix the first issue with expanding Table of Contents menu.

I am recording all the tickets in the doc-test repo and will cross mention the issues here when they are created.

[dimitri-tenstorrent]

@tt-rkim Take a look at the tutorial 2 here - https://tenstorrent.github.io/docs-test/ttnn/ttnn-latest/ttnn/tutorials/ttnn_tutorials/002.html

[tt-rkim]

Looks fantastic - how did you do this?

[dimitri-tenstorrent]

Lots of trial-and-error on the weekend with the sphinx-markdown-builder I had to implement something called CodeAreaNode. I do believe there are a few things missing from the tutorials due to lack of implementation of FancyCodeNode :)

writing output... [100%] ttnn_sweeps/index
/Users/dgnidash/projects/tt-metal/docs/source/ttnn/ttnn/tensor.rst:23: WARNING: unknown node type: <caption: <#text: 'Tensor with 16 ...'>>
/Users/dgnidash/projects/tt-metal/docs/source/ttnn/ttnn/tutorials/ttnn_tutorials/004.ipynb:45: WARNING: unknown node type: <FancyOutputNode: <container...>>
/Users/dgnidash/projects/tt-metal/docs/source/ttnn/ttnn/tutorials/ttnn_tutorials/006.ipynb:297: WARNING: unknown node type: <FancyOutputNode: <container...>>
/Users/dgnidash/projects/tt-metal/docs/source/ttnn/ttnn/tutorials/ttnn_tutorials/007.ipynb:238: WARNING: unknown node type: <FancyOutputNode: <container...>>
build succeeded, 568 warnings.

[tt-rkim]

@dimitri-tenstorrent can we close this?