Improve HPy documentation

[fangerer]

resolves #323 resolves #320 resolves #149 addresses #342

There is, ofc, still more to document (in particular, we should add documentation to all API functions) but I think I'll make a cut here. I'll plan to spend a certain amount per week to write API doc but any help is, of course, welcome.

Major changes in this PR:

• Implemented verification of tuple/list builder usage in debug mode. This also needed a little refactoring of DHQueue. I did not (yet) implement tracking of open builders because it is possible to find them by their elements since they would also leak. Tracking leaks of empty builders didn't seem very useful to me but could still be nice.
• Updated debug mode documentation to include latest features.
• I tried to structure doc section API Reference in a way that it gives a good overview of all API elements. IMO, the HPy API consists of three parts:
  • Functions and handles derived from public_api.h. I called this the Core API.
  • The HPy helper functions (helpers.h, argparse.h, buildvalue.h)
  • The HPy inline helper functions (inline_helpers.h).
• I've implemented an autogen generator that creates a core API function index. I'm thinking of generating a full function index (including functions of all three parts) by using pycparser but I'll do that in a separate PR.
• I've added doc for HPyContext using hpy/cpython/autogen_ctx.h. The CPython ABI context just contains handles, the name, and the ABI version and I think these are the only user-facing things.
• I added doc about types in HPy API introduction (= api.rst). This is not an API reference doc but describes why and how to use shapes, legacy types, etc. Also includes examples.
• I've removed api.md. I didn't even realize that it was there but it contained some out-dated doc and someone on GitHub found it and complained about that. There was one section about HPy protocols that I moved to misc/protocols.rst.
• I've invested also some effort into section Porting Guide; in particular, I've written another autogen generator that generates a C API - HPy API function mapping.
• Last but not least, I've documented many API elements.

[mattip]

Hmm. This conflates things that might be easier to review in separate PRs:

• documentation improvements
  • lots more API and code documentation in comments in the code
  • rst improvements
  • additional porting help
• Implemented verification of tuple/list builder usage in debug mode.
• an autogen generator that creates a core API function index
• an autogen generator that creates a C-API to HPy mapping

Some CI is failing.

[fangerer]

The PR mainly contains doc changes. The debug mode changes are just there because #320 asks for both, tuple/list builder docs and debug mode features. I tried to make useful commits and so it should be possible to review commit-by-commit. For example, the tuple/list builder changes are:

• Document list/tuple builder
• Make DHQueue more generic
• Track builder handles in debug mode

Also, the autogen changes are nicely encapsulated in commits:

• Auto-generate API function index for doc
• Generate documentation for struct _HPyContext_s
• Use autogen to generate C API - HPy function mapping

Anyway, if you still like to have the PR split, then I can do that as well. In that case, I already have a question regarding your suggestion:

• documentation improvements
  • lots more API and code documentation in comments in the code
  • rst improvements
  • additional porting help

Does that mean that you want to have three PRs?

[mattip]

I can do commit-by-commit, no need for a new PR.

[mattip]

Can you make the commit history reflect the division you mention in your comment (no need to split all the doc ones) - I see 32 commits.

[mattip]

Also CI is failing

[fangerer]

Also CI is failing

Yes, looking into the CI problems right now.

[mattip]

I went to the summary page to download the docs artifact. There are some weird failures there: it seems the python3.10 job died?

[fangerer]

@mattip I fixed the CI issues (except of cppcheck but that failed before and is a separate story).

I went to the summary page to download the docs artifact. There are some weird failures there: it seems the python3.10 job died?

Yes, I've seen that and re-ran the job. Now it passes. Also re-running jobs that were cancelled because of that.

Anyway, I changed the commit history for nicer commit-by-commit review. Here is an overview:

Debug mode implementation changes (tuple/list builder)

• Make DHQueue more generic
• Track builder handles in debug mode
• Add tuple/list builder debug mode tests

Documentation Changes

Section HPy API Introduction

• Write documentation for HPy types

Section API Reference

• Documentation of HPy types, modules, and definitions: Basically everything that is needed for defining types, modules, members, methods, etc.
• Documentation of list/tuple builder: This commit also contains a renaming of parameter initial_size to size (functions HPyListBuilder_New and HPyTupleBuilder_New). This change is trivial but noisy because of the auto-generated sources.
• Structure API reference into core API, helpers, and inline helpers: Sorry, this commit should be before others. Unfortunately, reordering basically means to rewrite everything. However, this commit is pretty simple.
• Document inline helpers and related core API funcs
• Auto-generate API function index
• Document struct _HPyContext_s: This commit also includes an autogen class which is currently not used. I thought it might be useful in future but currently, the doc for struct _HPyContext_s is generated using sphinx-c-autodoc.
• Auto-generate C API - HPy function mapping
• Restructure and refine porting guide

Section Debug Mode

• Update debug mode doc (bytes data ptr, tuple/list builder)

Misc

• Move section 'Specialized protocols' to 'Misc': This rescues a section from the (later) removed api.md.
• Minor syntax fix in overview.rst
• Remove obsolete api.md
• Sphinx conf improvements

[mattip]

Thanks! This is soooo much better, let's put this in as-is and iterate where needed.

[fangerer]

@mattip Thank you for reviewing and merging. I plan to do more improvements and API docs anyway. I will address the open issues in follow-up PRs.