search

Seneca-CDOT / telescope

A tool for tracking blogs in orbit around Seneca's open source involvement
https://telescope.cdot.systems
BSD 2-Clause "Simplified" License
96 stars 189 forks source link

Set up docker.cdot.systems Docker Registry #2801

Closed humphd closed 2 years ago

humphd commented 2 years ago

Following-up from https://github.com/Seneca-CDOT/telescope/issues/1743#issuecomment-1026157483, we need a solution for pushing docker images from CI to a registry. Our options seem to be:

  1. use the GitHub Package Registry
  2. host our own Docker registry on staging/production

I don't think there is another (free) option. Docker Hub is not going to work for a Team, and Amazon ECR will cost us. Are there other options I'm missing?

If we go with 1., which is probably the simpler of the two, we need to figure out what would break, and what has to be re-created. That includes:

I have no idea what else would break. If we do this, we'd have to plan it carefully and execute it in such a way that it wouldn't cause major downtime for the project/community.

Thoughts?

humphd commented 2 years ago

We're discussing doing this on the Thursday before study week, and then fix things over the break.

humphd commented 2 years ago

According to https://docs.github.com/en/repositories/creating-and-managing-repositories/transferring-a-repository, this should mostly work:

People will need to update their upstream remote with git remote set-url origin new_url.

I think people with access will still retain it too, though, we might need to tweak permissions and teams. I'm not sure what's going to happen with Projects, especially those that are attached to the Seneca-CDOT org.

humphd commented 2 years ago

I looked to see what we could create. The following team names (e.g., XXXX/telescope) are (currently) available:

Other ideas?

Kevan-Y commented 2 years ago

I looked to see what we could create. The following team names (e.g., XXXX/telescope) are (currently) available:

  • SenecaOpen
  • OpenSourceCourse
  • LearningOpen
  • PlanetSeneca
  • SenecaCodes
  • code4seneca
  • osd-dps
  • SenecaBlogs

Other ideas?

I love PlanetSeneca name

sirinoks commented 2 years ago
  • SenecaOpen
  • OpenSourceCourse
  • LearningOpen
  • PlanetSeneca
  • SenecaCodes
  • code4seneca
  • osd-dps
  • SenecaBlogs

I don't really like having Seneca in name, OpenSourceCourse sounds so generic to be available.. Voting for that

DukeManh commented 2 years ago

I really want to stick with Seneca-CDOT, but I'll go with SenecaOpen if I have to choose.

DukeManh commented 2 years ago

@humphd Why doesn't Docker Hub work for us? I think they have a free plan for education/open-source.

image
humphd commented 2 years ago

We could try. What I don't love about is the fact that it would be tied to my account vs. a team (blocked on me to do any fixes), and that we get rate limited "200 image pulls per 6 hours." Without good caching, this is going to mean that CI builds, deployments, local dev, etc. will just randomly fail.

I don't know, maybe I'm wrong and we should try it first.

humphd commented 2 years ago

We could also try https://www.docker.com/blog/expanded-support-for-open-source-software-projects/

TDDR commented 2 years ago

We could also try https://www.docker.com/blog/expanded-support-for-open-source-software-projects/

This seems like the easier solution, but is easier better?

Do we meet the other two requirements, or should issues be made to address them?

humphd commented 2 years ago

We use an OSI approved license, https://opensource.org/licenses/BSD-2-Clause.

Our images would be under the same license. I think we are good for all of this.

humphd commented 2 years ago

Docker Open Source Community Application Thank you for completing the Docker Open Source Survey. We will review your application, and get back to you as quickly as we can. We have received a large of applications, so please bear with us, as we go through the review process

OK, I filled out the form. I'll see what they say.

humphd commented 2 years ago

Another follow-up. I had a good meeting today with Chris Tyler. He and I agreed that if Docker says 'no', or it doesn't make sense to use Docker Hub, we'll look at hosting our own Docker Registry in CDOT. He has a machine with 1/2 TB SSD that we could use pretty quickly, and a lot more NAS storage that we could expand into later if needed.

Running our own registry is well documented at https://docs.docker.com/registry/deploying/. Essentially, we need to run the a registry Docker container, and configure it for proper storage volumes, access, and security.

We might be able to do this using our existing infrastructure, which would get us the TLS and security for free. If not, we'll have to do Let's Encrypt, DNS, etc ourselves.

cindyorangis commented 2 years ago

Oh thank god, we don't have to move out of Seneca-CDOT, I was having an identity crisis over what name Telescope's org was gonna get.

humphd commented 2 years ago

This is interesting, re: docker build:

By default, the build cache is based on results from previous builds on the machine on which you are building. The --cache-from option also allows you to use a build-cache that’s distributed through an image registry refer to the specifying external cache sources section in the docker build command reference.

With our own registry, we could re-use the build cache in CI or on development machines too.

humphd commented 2 years ago

I've reached out to Chris to start the process of creating our own Docker Registry.

Those that are interested in being involved should begin by reading https://docs.docker.com/registry/deploying/. Once I talk to him and get more details, I'll file specific issues that we can start fixing.

humphd commented 2 years ago

I've just finished a meeting regarding this, and here's the update:

TODO items for this work:

I'm sure there's more to do, but that's the list I have right now. Let me know if and how you want to be involved.

TDDR commented 2 years ago

I've just finished a meeting regarding this, and here's the update:

* Chris has to go to campus to move some SSDs around so we can take over one of his servers.

* He's going to create a VM that we can use to host the Docker Registry, and we'll initially have access to a few hundred Gigs of storage.

* Later, he's hoping to move this into a rack with faster network and access to a larger set of network storage, but that won't happen this term.

* He hopes to have this set up done by Friday afternoon

* We'll run the registry at https://docker.cdot.systems

TODO items for this work:

* we need to get SSL certs setup for the `docker.cdot.systems` domain with Let's Encrypt, and have them auto-update.  Ideally we can steal/learn from what @manekenpix has done on Telescope (I don't know all the steps)

* we should probably run this behind nginx, which is documented at https://docs.docker.com/registry/recipes/nginx/#setting-things-up

* setup Basic Auth username/password pairs for any service (e.g., GitHub Actions) or person who needs to `push` to the registry.

* we should see if we can set-up pull through mirroring https://docs.docker.com/registry/recipes/mirror/

I'm sure there's more to do, but that's the list I have right now. Let me know if and how you want to be involved.

I would like to be involved. I don't have many issues assigned right now for anticipation of this, so load me up with a decent workload for over the next couple of weeks if possible.

DukeManh commented 2 years ago

What will be the network bandwidth of the current system?

Kevan-Y commented 2 years ago

Just wondering, I read a tutorial, about self-hosting docker registry. Is it something similar to this https://www.digitalocean.com/community/tutorials/how-to-set-up-a-private-docker-registry-on-ubuntu-18-04 we gonna do?

humphd commented 2 years ago

What will be the network bandwidth of the current system?

Eventually 10 Gigabit between the docker registry and staging/prod, but not until we move it into the server rack later in the term. For now, I don't know the answer, but should be pretty fast.

humphd commented 2 years ago

Just wondering, I read a tutorial, about self-hosting docker registry. Is it something similar to this https://www.digitalocean.com/community/tutorials/how-to-set-up-a-private-docker-registry-on-ubuntu-18-04 we gonna do?

Yes, very similar, but on Fedora vs. Ubuntu.

humphd commented 2 years ago

@TDDR, @rclee91, @Kevan-Y for the docker registry, here is what I think we need to do:

@TDDR I've put your name on all of this, but we can add more people and should split this up into separate issues. I know @rclee91 and @Kevan-Y want to join in as well, so maybe they can speak up. I'm happy to take any of it you don't want and do reviews.

I would suggest trying to create all this for your local fork without SSL to learn how the proxy works (i.e., try to get it setup so you have to authenticate with GitHub on your local machine)

humphd commented 2 years ago

OK, server is setup, docker.cdot.systems created. Thank you @ctyler!

Now the fun begins.

DukeManh commented 2 years ago

This is exciting, I added my name to a task.

Kevan-Y commented 2 years ago

Assigned myself to some tasks too.

humphd commented 2 years ago

I think oauth2-proxy might be hard to use for automated login scenarios (e.g., pushing to the registry from CI). I think we should look at https://github.com/cesanta/docker_auth too, which should let us do it with tokens, see https://github.com/cesanta/docker_auth/blob/main/examples/reference.yml#L122-L163

humphd commented 2 years ago

Auth discussion happening in https://github.com/Seneca-CDOT/docker.cdot.systems/issues/3

tpmai22 commented 2 years ago

@TDDR if there by chance you uploaded the PR by tonight, please also file following up issues for 2.9 and close this.