search

mediacloud / story-indexer

The core pipeline used to ingest online news stories in the Media Cloud archive.
https://mediacloud.org
Apache License 2.0
1 stars 4 forks source link

architect approach to building/hosting container images #101

Open rahulbot opened 9 months ago

rahulbot commented 9 months ago

I think I heard this go past as a question: where do we hold our built images for deployment. We need to decide if there is a local store or we use one of the image hosting services that exist. Please comment if I misunderstood or this need more urgency.

philbudne commented 9 months ago

By tweaking a docker configuration file, I was able to make ramos.angwin:5000 an acceptable insecure image repository (tho I may have had to manually pull the image to other nodes). There may have been some grumbling about image signing.

My $0.02:

Disclaimer: I'm not a Docker purist, so I have no religion about this.

Using a real/secure repo is probably a pain (wouldn't want to check the keys into a public git repo), and since we're working in a closed/hidden cluster it doesn't seem (to me) like there is risk in using an insecure, local repo.

BUT, if all the containers using the indexer-worker (and any web-search-api) images run on one server (ie; bernstein), it may all be a moot point.

thepsalmist commented 9 months ago

Using Github's CI could work for this case, we could have a CI to build and publish the Images to the repo of choice (Dockerhub/AWS ECR) one one of the actions such as pushing to staging/release. The authentication should be a one off process. Ref https://docs.github.com/en/actions/publishing-packages/publishing-docker-images

rahulbot commented 6 months ago

We are effectively using GitHub CI to build and push news-search-index images to DockerHub whenever a new tag is pushed to GH. So I think that process (as described above) could work well here too.

rahulbot commented 2 months ago

(potentially related to #200)