search

mediacloud / news-search-api

Internal API server that offers search access to the Media Cloud Online News Archive (in Elasticsearch).
https://mediacloud.org
GNU Affero General Public License v3.0
1 stars 3 forks source link

Issues with deployment #96

Open philbudne opened 2 hours ago

philbudne commented 2 hours ago

I did development and deployment of new-search-api for this first time this week and ran into the following issues:

  1. When doing "make dev" (deploy a dev stack), it expects configuration in the USER environment!
  2. There is no .env-template file for outside docker settings, OR to serve as examples for the above. The rss-fetcher/web-search push.sh & story-indexer deploy.sh scripts do not take user environment for configuration and I don't think this should be the case here either. The push.sh scripts source the .env template file, and THEN a per-user file for overrides (story-indexer doesn't have a .env file).
  3. Developer stacks use a fixed port offset, so only one can run on a server. Production (naturally) uses the base port and (in story-indexer) the next increment up is for (one) staging stack, and above that is for user stacks (increment can be controlled by an environment variable). deploy.sh here uses a larger offset for staging than for user stacks, and allows no control over the user offset.
  4. deploy.sh creates a docker-compose.yml.dump file as the story-indexer does (for testing the syntax of the file), but I didn't use the .dump file, in case someone introduces a bug in the dumper, it's better not to trust it. This deploy.sh uses the .dump file for the final deployment. [I've been thinking the .dump file should be (re)named with the same tag as the image, so they stay around, and you can redeploy an older stack easily that way if needed]).
  5. Since the .dump file may contain parameter values from private repos, it should be protected against world & group access, as the story-indexer deploy.sh does.
  6. The docker-compse.yml file mounts the current working directory OVER the /app directory baked into the image!!
  7. The docker-compose.yml file does not restrict containers to running only on the swarm leader node (ramos) as the story-indexer compose file does. As a result, some containers ended on bradley. I removed bradley from the swarm (since we never want anything to run there), and the containers are now all on ramos.
  8. This deploy.sh does not check that the current directory is "clean" and corresponds to the release being deployed: this means the container and deployment files may not match up.
  9. This deploy.sh does not tag the -config repo (as the other scripts do). The mcweb & rss-fetcher push scripts will use the current version (extracted from the source, since tagging is not magical), and if that config tag already exists, they use version + date to show the config used by later deployments [maybe ALL config tags should be version + date?].
philbudne commented 2 hours ago
  1. Does anyone use config.yml? If not remove it (and reading it, from utils.py). I was taught it's best practice NOT to bake config into any files put into the image.
  2. This repo does not use the usual staging and prod branches, and the script doesn't enforce it!