search

azukiapp / azk

azk is a lightweight open source development environment orchestration tool. Instantly & safely run any environment on your local machine.
http://azk.io
Apache License 2.0
898 stars 63 forks source link

Add support for quay.io #581

Closed cweagans closed 8 years ago

cweagans commented 8 years ago

If it's supported, I couldn't figure out how to use it. Using this:

    image: {"docker": "quay.io/myemployer/thecontainer"},

tried to download library/quay.io/myemployer/thecontainer, and obviously failed.

Am I missing something? Or are alternate docker image hosts not supported?

gullitmiranda commented 8 years ago

@cweagans to be honest, not yet tested with other registries. We will be performing tests and updating here. Thanks for the report.

cweagans commented 8 years ago

@gullitmiranda Is there any way that I can test with other registries without modifying azk? These are the three scenarios that I'd expect to work for an image name:

In addition, I'm not sure that registry authentication is being handled properly. If I use a private image and I'm not authenticated with the registry that the image is coming from, azk should prompt me to authenticate before complaining about a 404 (which is what Docker Hub will return for private images if you're not authenticated) or a 403 (which is what Quay.io will return). Right now, though, if I'm not logged into a registry, I get this:

azk: Lost internet connection:
azk:   Error: 404: cweagans/myimage >> url: https://index.docker.io/v1/repositories/cweagans/myimage/images
azk: Due to the above error azk will stop all instances already running.

If this is not a priority for the azk team, can you please point me in the right direction? If you can point to relevant files/functions where these changes would need to happen, I can spend some time on making it work.

gullitmiranda commented 8 years ago

I did some testing. The problem occurred because an azk dependency could not handle private registries.

https://github.com/azukiapp/docker-registry-downloader/blob/master/src%2Fdocker-hub%2Findex.js#L3

paralin commented 8 years ago

Would it not make sense to, rather than call Docker API directly, use the installed docker client to download images?

nuxlli commented 8 years ago

@paralin one of the things we value most in azk is its usability. Not only in the azk commands, but also in how the arguments can be used (different from Docker, azk respects Unix standards and the arguments can be passed in any order) and, mostly, in the feedback to the user.

The idea of using Docker CLI isn't good because it doesn't give us any control on what's going on. Also, it would require Docker to be installed on Mac, making azk's installation process a little more complicated (and it wasn't an option when azk was first published).

But when azk was first developed, there were three big problems with pulling images using the Docker API:

1) Docker didn't do parallel pull of an image layers;

2) It wasn't possible to resume a running pull;

3) It wasn't possible to get the total of bytes to be downloaded and display an overall progress bar to the user.

Since it's important for us to have control of those issues, we've developed the docker-registry-downloader to solve the issues (1) and (2).

However, right after publishing it, the first issue was fixed in Docker, so we don't use the parallel download feature from docker-registry-downloader.

The issue (2) never had an easy solution, because it depends on inner changes from Docker, which only became available recently: https://github.com/docker/docker/issues/6928 and https://github.com/docker/docker/pull/18353 (however we still have to update azk to use this improvement).

Due to the the issue (3), we haven't implemented support to other repositories beside Docker Hub (we could have done this, but we prioritize other features).

Besides all of that, Docker has recently announced that the Registry API v1 will be deprecated (and the whole docker-registry-downloader is based on that version).

Since we have to do this upgrade, this is priority now and it should be solved by adding and using the lib https://github.com/joyent/node-docker-registry-client

cweagans commented 8 years ago

The idea of using Docker CLI isn't good because it doesn't give us any control on what's going on

Why do you need control beyond "Download this image and let me know when it's done"? Re-implementing the Docker client in Node seems wasteful.

when azk was first developed, there were three big problems with pulling images using the Docker API

Those issues are now resolved. Why not just use the Docker client directly and nuke this entire part of the codebase from orbit?

It is not so useful to see a new feature implemented in Docker and then have to wait for it to be implemented/fixed/whatever in azk before it can be used on a project.

paralin commented 8 years ago

Also the security auditing on the registry logic in this code is iffy at best, compared to using the official client. And you're already using the Docker runtime locally anyway.

I think azk currently has a lot of logic built into it that really doesn't need to be there. This is one example. Another is the inability to generate application images that don't need the azk runtime in place to execute.

nuxlli commented 8 years ago

Also the security auditing on the registry logic in this code is iffy at best, compared to using the official client. And you're already using the Docker runtime locally anyway.

Not all of azk user is a necessarily a Docker user, specially if you are using azk on Mac OS X, you don't Docker runtime locally, only inside the virtual machine. (And in this case a bunch of ssh requests is not the answer).

Which is the security point you've mentioned? azk yet doesn't handle private images and isn't responsible for downloading images (Docker does that). It only looks for every image's layer size to provide a global vision/perception of the process. In this way, we are talking about user experience.

I think azk currently has a lot of logic built into it that really doesn't need to be there. This is one example. Another is the inability to generate application images that don't need the azk runtime in place to execute.

To answer this question it important to highlight an issue: azk isn't a tool to ease the Docker experience. It is a tool to manage and orchestrate development environments. It offers an approach completly different from docker-compose when it comes to handle application's code and it's necessary images.

If by one side of the coin azk doesn't offer a way to build images (even because our deployment process is still an experiment) to be executed in an environment free of azk, on the other side it offers things such application live-reload (very important feature for rails developers for example) and the capacity to resolve dependencies without the necessity of building new images.

nuxlli commented 8 years ago

Back to the issue's topic and the initial @cweagans' questioning, the steps to accomplish the support for quay.io are:

It is certainly not a simple process but it is important according to problem pointed out by this issue and the incoming Registry's update.

Thanks for the interest and we really appreciate if you could make some efforts on this implementation. :) We are also at Gitter, reach us if you have any questions about the process.

cweagans commented 8 years ago

I definitely won't be working on this, btw. To be blunt, azk creates more problems than it solves for me at this point. Docker Compose did the job for my team.

paralin commented 8 years ago

I would say Azk is a promising tool but the direction needs to change a bit, I think.

The goal should be, 1 command development environment, and easy ways to translate this development environment to real life deployment scenarios (like Kubernetes, for example). I think it should use existing tools like Docker as much as possible and avoid rolling its own internal behaviors because the Azk agent most certainly will not be available in production.

It's up to the creators of the tool, though.