The "document" format for page content isn't documented

[benjamincburns]

Endpoints that return page content generally allow you to select either markdown or document formats. From what I can tell, the document format isn't actually documented anywhere, which limits its utility.

This is problematic in my case because the AI search sources refer to page sections by their keys. I can't get the key for a particular page section unless I get the doc back in document format. I want to be able to pull out the relevant sections from an AI search response and feed them back into an LLM, formatted as markdown for additional context. However in order to do that I'd need to reverse engineer your document format, or go find the code that defines it in the main gitbook repo.

If I'm wrong about this, and the document format is documented somewhere, please make it more discoverable from the docs pages, as I've searched around and come up empty-handed.

[benjamincburns]

Looking further, the structure of the page content is documented in the API reference, but it's really difficult to consume it in this way. You have to toggle through all of the various item type dropdowns, and the deeply-nested nature of the structure means that after a few clicks you're finding yourself trying to read content that's rendered as in the screenshot below. Having a user guide for this structure (and something like a JSON schema that describes it fully) would be very helpful.

[addisonschultz]

Hey @benjamincburns - thanks for the feedback with the docs (I'm having a look at your other issues as well). Definitely great feedback for the way you're using the types in the API response sender. We can look into this with the Scalar team to improve to make it easier to use given the structure in place for the types/schema.

We have the entire OpenAPI spec hosted here - https://api.gitbook.com/openapi.json

Maybe that would help in the interim as we improve on this?

[benjamincburns]

Thanks @addisonschultz! The OpenAPI spec file will be a bit easier to review for sure. Is that linked somewhere in the dev docs and I missed it? It would be nice if that was linked from each of the reference pages as well, as a sort of "escape hatch" from the problem in the screenshot above.

I can raise a new issue for this if you'd like, but another bit of feedback is that it'd be nice if the responses shown in the reference docs were more realistic and reflective of real-world responses. The ones that are there currently are a bit more like stubs, and the content doesn't accurately reflect what's actually returned.

Finally, I apologize for this being a bit off topic, but while I have you - is there an existing library out there (ideally a node module) that can transform a Document object into markdown - or more ideally for my particular case - render a section of a Document?

[addisonschultz]

Hey @benjamincburns - it's not linked currently in the docs - so can understand how that might have been confusing. I've just opened a change request for us to add it in.

Also, thanks for the feedback on the API examples. They come from our OpenAPI spec directly, so our team can look into improving these.

I guess for your last question - is there a reason you can't just return the markdown response to get what you need?

[benjamincburns]

is there a reason you can't just return the markdown response to get what you need?

The ask endpoint returns sources in its response. Those sources contain a list of section references. From what I can see in the response, the only way for me to get the relevant sections is to fetch the page in document format.

I definitely can just request the markdown format for the whole page, but I'm feeding this content (along with the answer from the ask API) into an LLM, so I want to only provide the most relevant content to the question being asked, in part to help the LLM to focus on only the most relevant details, and in part to minimize input context token count.

What would really help is if the page and revisions endpoints allowed you to fetch individual (or maybe a list of?) sections, and control their format as you can now, by requesting either a markdown or document response.

Thanks again for the quick replies, @addisonschultz!