LLMs are better at reading some kinds of content than others. In particular, they are highly optimized to read markdown. Making MyST documents easier to read by LLMs may allow our authors / readers to more effectively utilize MyST content as part of their workflows. Currently, there's not an easy way to do this without manually wrapping up a book's content into a Markdown file.
User story
As an author, I want to expose (and select) my MyST content so that it can be easily discovered and used in LLM workflows by readers.
An effort to standardize LLM text files for static sites
The Answer.AI (and FastAI) team have a proposal to standardize the use of llm.txt to signal to web scrapers the content that is "meant" for scraping by LLMs. We could follow this convention, or something similar to it, as part of the MyST build process.
LLMs are better at reading some kinds of content than others. In particular, they are highly optimized to read markdown. Making MyST documents easier to read by LLMs may allow our authors / readers to more effectively utilize MyST content as part of their workflows. Currently, there's not an easy way to do this without manually wrapping up a book's content into a Markdown file.
User story
As an author, I want to expose (and select) my MyST content so that it can be easily discovered and used in LLM workflows by readers.
An effort to standardize LLM text files for static sites
The Answer.AI (and FastAI) team have a proposal to standardize the use of
llm.txtto signal to web scrapers the content that is "meant" for scraping by LLMs. We could follow this convention, or something similar to it, as part of the MyST build process.