Add the Scoop REST API to the development environment

[rebeccacremona]

This PR arranges for Perma's development environment to spin up a running instance of the Scoop REST API. See Linear ENG-269.

By default

Now by default, running docker compose up will:

• spin up a Scoop REST API container and its MySQL database,
• create the necessary tables and columns if starting with a fresh DB,
• start cron and xvfb in the background,
• launch the Flask dev server
• launch six parallel capture processes, and
• create a new Scoop API key and it put somewhere the Perma Django app can find it.

The API is ONLY reachable from the web container, e.g. docker compose exec web curl http://scoop-rest-api:5000.

It is NOT exposed to the host. Which is to say, curl http://localhost:5000 from your shell will not work.

Dev only, with an easy opt-out

I added the config to docker-compose.override.yml for now so that you can opt out (for instance, if you want to run Perma and the Scoop REST API separately from their own repos, to more easily make alterations to both at the same time), and so that CI won't run these containers.

We can rearrange as desired when we have something for CI to test for us. We might consider trying to come up with something clever, rather than moving the config to docker-compose.yml... the ability to seamlessly run both apps from their own repos is pretty nice.

Tweaking

You can change the environment variables in scoop-rest-api > environment to get your desired behavior if the defaults don't suit your current task.

We can also consider adding a copy of config.py to this repo (e.g., in services/docker/scoop-rest-api) and mounting that into the container, to allow for freer experimentation with Scoop REST API settings... but I figured let's hold off until we need it, so that we don't have to worry about keeping config.py up-to-date in this repo while the API is still so new and fluid.

API Keys

I am fairly pleased with this strategy for automatically communicating a Scoop API key to the Perma Django application.

It does have its downsides:

• access_key.txt doesn't necessarily describe keys that are actually in the Scoop API's database; a person could have deleted the database but not deleted access_key.txt
• creating a new access key every time means access_key.txt could get long... but I have a hard time imagining it will get so long that it becomes a problem
• it can be hard to remember that, because info in volumes persists until you delete the volume, that access_key.txt will be available to the Django app once created even if, at the moment, you aren't running the Scoop API container
• it uses a regex to parse the message... which could change.

But, I think this will be pretty convenient, and is about as close as we can get given how the Scoop API currently handles the issuing of API keys. We can readdress if that changes, or if this is proving clunkier than I anticipate.

[codecov[bot]]

Codecov Report

Merging #3381 (c0ea939) into develop (0bc6e7e) will not change coverage. Report is 2 commits behind head on develop. The diff coverage is n/a.

@@           Coverage Diff            @@
##           develop    #3381   +/-   ##
========================================
  Coverage    69.75%   69.75%           
========================================
  Files           53       53           
  Lines         7098     7098           
========================================
  Hits          4951     4951           
  Misses        2147     2147