Merge API documentation into main docs site

[sarayourfriend]

Problem

This is based on an issue that @obulat reported:

Add Openverse colors/customize the design slightly to make it clear that it's still Openverse.

Description

The page in question is https://api.openverse.engineering/v1/

Redoc, the library used for the Django documentation, has essentially zero customisation options. Additionally, it uses React, which we could SSR but would require us invest way more time into this.

Rather than having to maintain two separate documentation websites built on different and hardly related technologies, we could use something like https://github.com/sphinx-contrib/openapi to add our OpenAPI spec into the main documentation site. That would solve the issue Olga mentioned and reduce the number of disparate dependencies we rely on for documentation generation.

The generate_swagger management command can already be used to create an openapi.yml file, as seen in the check to validate generated specs: https://github.com/WordPress/openverse/blob/697f62f01a32cb7fcf2f4a7627650a113cba40da/api/catalog/management/commands/validateopenapischema.py#L37

CI would need to generate that spec and output it into the documentation site before building the final site. We would also need to make it easy to iterate on the documentation locally.

[dhruvkb]

I agree with the idea of consolidating as much documentation as possible into one location but for some reasons, I still prefer the API documentation to be separate.

Fundamentally, I consider our documentation to be of the following types based on who it is targeted towards.

• Developers building Openverse itself: https://docs.openverse.org, the Sphinx site
• Developers using the Openverse API: https://api.openverse.org, the ReDoc site
• Users of the Openverse search engine: https://openverse.org/search-help, the frontend page

While having them all in central location is good, having them where people will look for them during their usage is better.

In any case, maybe a "💬 talk: discussion" label applies before we take a call or start work on this.

[sarayourfriend]

I can see your point for search users. For people consuming the API directly, I don't see a significant difference in usability/discoverability between hosting the documentation directly on the API or using the docs site, provided we linked to the docs site's API documentation from the API itself (or redirected to it). It does, however, present a maintenance burden. The different technology used to render documentation is redundant, but also prevents us from being able to create a sense of cohesion in our documentation. The API docs and the docs site feel disjointed at the moment due to having similar but ultimately very different presentation styles.

If it was possible to make the Redoc site look like the other docs site, then I would feel differently, because at least then it would achieve a visual cohesion between the two. Even then, though, it would be really nice to be able to write documentation that doesn't need to be embedded into Swagger and have them in the same place. API registration, for example, is just a bunch of Markdown embedded directly in the OAuth view code: https://github.com/WordPress/openverse/blob/697f62f01a32cb7fcf2f4a7627650a113cba40da/api/catalog/api/views/oauth2_views.py#L40-L72

Another example: https://github.com/WordPress/openverse/blob/697f62f01a32cb7fcf2f4a7627650a113cba40da/api/catalog/api/docs/media_docs.py#L21-L37

That means the Markdown isn't linted according to our documentation standards.

Additionally, to slightly problematise your premise a little, if docs.openverse.org is only developer documentation, then it is odd to have it in such a prominent position in our domain. The URL implies it is documentation for all of Openverse, not just for development. If those docs are truly just for development, then the subdomain should be dev-docs, or we should just write the documentation in GitHub Markdown and leave it in the repository rather than publishing it. To be clear, I don't think we should do either of those, I just want to point out that docs.openverse.org sounds to me like where I would go to find documentation about Openverse generally, not specifically for developing Openverse.

In my imaginary ideal world, I would even prefer that we did something like this:

• Move all descriptive API consumption documentation (registration guide, general notes about features of the API, things to keep in mind, etc) into the documentation site
• Use Sphinx to generate the swagger documentation.
  • Originally I thought we could use the Swagger UI, but it is super heavy with heaps of JavaScript. Our API doesn't really need that, I don't think?
• Update the root view of the API to have a note like the Language Tool API: https://api.languagetoolplus.com/

  Error: Missing arguments for LanguageTool API. Please see https://languagetool.org/http-api/swagger-ui/#/default

• Leverage the DRF "Browsable" view to link to real requests. We can update the template of these views to have Openverse styles and could modify the forms to be safe (cf #973)
  • DRF Browsable view isn't exactly light: it relies on bootstrap JS which in turn relies on jquery, etc. But it's built into the technology we already use, rather than something external. Plus, we can actually change the way it looks to give the documentation visual cohesion.
  • Alternatively, do away with the fancy DRF browsable API view and render uncompressed (pretty-printed) JSON on our API routes whenever the request isn't application/json content type.

This has the benefit of removing the responsibility from Django to serve Redoc and unifies our documentation both in location and presentation.

Yet another alternative, if we really want to have separate locations for the documentation but still want to achieve visual cohesion, replace Redoc with a simple Sphinx site served as static content in the API and use the sphinx OpenAPI extension to render it. We can use the same Furo configuration as for the documentation site. At least then they will look and feel the same and would still have the different domains (if folks feel that is indeed worth keeping).

[sarayourfriend]

I think we've effectively decided not to do this, because the API documentation needs to be updated per-environment, and the docs site does not have such a concept. We cannot build the docs from main for the API and publish them to the docs site, because those would represent staging, not production, and our canonical docs should be for the production application.

@dhruvkb Please reopen this if you have any ideas for how to do this that I'm forgetting. I skimmed over our discussions and my impression is that we basically said we cannot do this reliably, but just forgot to close the issue.

[dhruvkb]

@sarayourfriend I remember that there were certain approaches we could take but none of them were straightforward or low enough on effort to justify this. Let's close this and we can reopen it if some new idea occurs that makes this process much simpler.