Add documentation for development environment configuration

[sarayourfriend]

Problem

The documentation that exists only describes the bare minimum for a working development environment. It does not cover the steps required for setting up, e.g., a Python development environment that works with jedi or pylance to get auto-complete suggestions in editors such as Vim or VSCode.

Description

There are two options for configuring this. Both have trade-offs as far as ease of use goes:

1. Use the Python interpreter inside the docker container. This can take two forms. Either you can use something like https://marketplace.visualstudio.com/items?itemName=ms-vscode-remote.remote-containers or IntelliJ's own version which is built in. Or you can mount the Python installation from the container to the local machine and point your editor to that executable. The latter only works if the Python executable in the container will run on your local machine (i.e., you run Linux and happen to have a similar enough execution environment to the Python base image). You could use a local interpreter but mount the lib directory into the virtual environment to avoid needing to download dependencies twice (in the second option below), but it's bound to be flaky. The first option is almost certainly the most reliable, but requires proprietary editors/editor extensions and is therefore not free. It is probably the "easiest" though, especially for VSCode users if we supplied a devcontainers.json file in our monorepo. PyCharm/IntelliJ may have a similar auto-configuration tool. This option would not work for vim users, as far as I can tell, as Jedi doesn't support remote interpreters.
2. Run pipenv install --dev inside each project directory and configure your editor to use the appropriate interpreter depending on the directory you're working in. This can be done in VSCode (both free and non-free versions) using multi-directory workspaces. I'm sure PyCharm/IntelliJ and vim-jedi have similar capabilities. This is almost certainly the most reliable option, but it does duplicate dependency installation (once on the local and once more inside the container).

Either way, one of these is required to get completions working locally. My preference is the second because it doesn't require any special tools aside from the ones we already require. However, it does warrant some special documentation. For example, the API depends on psycopg2, which requires pg_config to exist locally. On macOS and Linux you'll need to install something like Arch's postgresql-libs package or just install Postgres in its entirety (it is usually distributed with the userland libraries).

[dhruvkb]

It is probably the "easiest" though, especially for VSCode users if we supplied a devcontainers.json file in our monorepo.

We do supply a devcontainers.json file so this might be even more feasible compared to the other option.

Personally, I exclusively use the Docker container for API and ingestion server development, but still install all Python dependencies locally for autocomplete and highlighting in IDEs.

[sarayourfriend]

Personally, I exclusively use the Docker container for API and ingestion server development, but still install all Python dependencies locally for autocomplete and highlighting in IDEs.

That's what I do as well for Openverse. On a project I worked on long ago I used the remote interpreter in PyCharm to great success and I was disappointed that VSCode makes it so much more complicated and hides it behind a proprietary plugin that cannot be installed on non-MS distributions of VSCode. vim-jedi doesn't appear to support it at all :confused:

[dhruvkb]

Just confirming that on macOS, you need to setup Homebrew and then use it to get postgresql@13 (entirely) to have pg_config.