search

GoogleCloudPlatform / cloud-sdk-docker

Google Cloud CLI Docker Image - Docker Image containing the gcloud CLI and its bundled components.
https://cloud.google.com/sdk/docs/downloads-docker
Apache License 2.0
745 stars 230 forks source link

Some Components Not Installed in `latest` Image #94

Closed timfallmk closed 4 years ago

timfallmk commented 7 years ago

The Readme states that all components are installed in the latest version of the image (as opposed to the slim or alpine versions). Running a fresh image shows that some components are not installed in the image, such as kubectl and docker-gcr-credential-helper for example.

As expected, components cannot be installed into the container after startup.

Tims-MacBook-Pro-15:~ timfall$ docker run -i -t --name google -v /root/.config google/cloud-sdk
root@59f399a68bb8:/# gcloud components list

Your current Cloud SDK version is: 165.0.0
The latest available version is: 165.0.0

+-------------------------------------------------------------------------------------------------------------+
|                                                  Components                                                 |
+---------------+------------------------------------------------------+--------------------------+-----------+
|     Status    |                         Name                         |            ID            |    Size   |
+---------------+------------------------------------------------------+--------------------------+-----------+
| Not Installed | Cloud Datastore Emulator (Legacy)                    | gcd-emulator             |  38.1 MiB |
| Not Installed | Emulator Reverse Proxy                               | emulator-reverse-proxy   |  14.5 MiB |
| Not Installed | Google Container Local Builder                       | container-builder-local  |   3.6 MiB |
| Not Installed | Google Container Registry's Docker credential helper | docker-credential-gcr    |   2.2 MiB |
| Not Installed | kubectl                                              | kubectl                  |  16.0 MiB |
| Installed     | App Engine Go Extensions                             | app-engine-go            |  98.0 MiB |
| Installed     | BigQuery Command Line Tool                           | bq                       |   < 1 MiB |
| Installed     | Cloud Bigtable Command Line Tool                     | cbt                      |   4.1 MiB |
| Installed     | Cloud Bigtable Emulator                              | bigtable                 |   3.5 MiB |
| Installed     | Cloud Datalab Command Line Tool                      | datalab                  |   < 1 MiB |
| Installed     | Cloud Datastore Emulator                             | cloud-datastore-emulator |  15.4 MiB |
| Installed     | Cloud Pub/Sub Emulator                               | pubsub-emulator          |  21.0 MiB |
| Installed     | Cloud SDK Core Libraries                             | core                     |   6.4 MiB |
| Installed     | Cloud Storage Command Line Tool                      | gsutil                   |   3.0 MiB |
| Installed     | Default set of gcloud commands                       | gcloud                   |           |
| Installed     | gcloud Alpha Commands                                | alpha                    |   < 1 MiB |
| Installed     | gcloud Beta Commands                                 | beta                     |   < 1 MiB |
| Installed     | gcloud app Java Extensions                           | app-engine-java          | 128.1 MiB |
| Installed     | gcloud app Python Extensions                         | app-engine-python        |   6.5 MiB |
+---------------+------------------------------------------------------+--------------------------+-----------+
To install or remove components at your current SDK version [165.0.0], run:
  $ gcloud components install COMPONENT_ID
  $ gcloud components remove COMPONENT_ID

To update your SDK installation to the latest version [165.0.0], run:
  $ gcloud components update

root@59f399a68bb8:/# gcloud components update
You cannot perform this action because this Cloud SDK installation is 
managed by an external package manager.  If you would like to get the 
latest version, please see our main download page at:

https://cloud.google.com/sdk/

ERROR: (gcloud.components.update) The component manager is disabled for this installation
ahmetb commented 7 years ago

cc: @salrashid123 should we install at least kubectl? I think docker credential doesn't help much as we don't expect people to build/push images with docker-in-docker. Same for local builder. Not sure about the others.

@timfallmk thanks for reporting. There's still a way to install the components via apt-get. See step 5 here: https://cloud.google.com/sdk/downloads#apt-get kubectl has its own apt package you can install directly via apt-get.

timfallmk commented 7 years ago

@ahmetb thanks for mentioning the install. For the record, I use all three examples you just mentioned, although in order of priority kubectl is the top.

Is it worth "unlocking" the gcloud install so it can be modified inside the container afterwards?

ahmetb commented 7 years ago

For the record, I use all three examples you just mentioned, although in order of priority kubectl is the top.

Do you push images using the containerized gcloud? How do you do that? Is it by mounting the docker socket?

Is it worth "unlocking" the gcloud install so it can be modified inside the container afterwards?

I don't think we intentionally lock it down, my guess is that when it's installed via apt-get it gets locked down. This probably means you should not unlock it.

We have an alpine-based image flavor too, give that a try. You should be able to install additional components there.

timfallmk commented 7 years ago

Do you push images using the containerized gcloud? How do you do that? Is it by mounting the docker socket?

Correct, mounting the docker socket and copying the docker daemon allows you to run direct docker commands (we build and push to gcr from here).

I don't think we intentionally lock it down, my guess is that when it's installed via apt-get it gets locked down. This probably means you should not unlock it.

This is also correct. The "lock" comes from gcloud being installed from a package manager. If it's installed via the curl command (or compiled manually), this configuration file is not set to unmodifiable.

As it currently stands, we essentially build our own gcloud docker image by using the curl install method and adding components manually. I'd love to pull directly from this image though.

ahmetb commented 7 years ago

Got it thanks for putting it to a perspective.

I just recalled that kubectl is present in this image:

$ docker run -i -t google/cloud-sdk kubectl version
Client Version: version.Info{Major:"1", Minor:"7", GitVersion:"v1.7.0", GitCommit:"d3ada0119e776222f11ec7945e6d860061339aad", GitTreeState:"clean", BuildDate:"2017-06-29T23:15:59Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}

It's just not installed "via" gcloud, that's why it's reporting as Not Installed. We have an internal bug tracking this. However, I think it is technically correct to report it as Not Installed if it's not installed via gcloud. For other components that are missing, we'll be adding those. @salrashid123

salrashid123 commented 7 years ago

yes, as @ahmetb stated for kubctl

for the other components, yes, its a bug (its outright not installable by apt- this is a bug, i'll file one internally today). 

/# apt-get install docker-credential-gcr container-builder-local emulator-reverse-proxy gcd-emulator 
Reading package lists... Done
Building dependency tree       
Reading state information... Done
E: Unable to locate package docker-credential-gcr
E: Unable to locate package container-builder-local
E: Unable to locate package emulator-reverse-proxy
E: Unable to locate package gcd-emulator
theist commented 6 years ago

any news?

seems that docker-credential-gcr still is not installable using apt

salrashid123 commented 6 years ago

Sorry, no updates on the bugs filed for those packages

@ahmetb perhaps we can refactor the image and just use the non-package installer (i.,e just untar somewhere and set $PATH)?

ahmetb commented 6 years ago

I’m open to anything. As long as we can add to the bin/ dir that it would normally reside, and the version of the docker-credentials-helper is up to compatible with the gcloud, it is fine.

Can you cc me on the internal bug? It should be made available as an apt pkg.

An alternative solution here might be force-overriding the package manager bit to “off” here temporarily and using “gcloud components install” for some components, and then turning it back on.

andreyluiz commented 6 years ago

For those who urgently need to get this running, use an older tag. I'm using 198.0.0-alpine.

mickeyreiss commented 5 years ago

Is there a workaround for the cloud-firestore-emulator component?

fixl commented 5 years ago

@mickeyreiss I resorted to building my own emulator images: https://hub.docker.com/r/fixl/google-cloud-sdk-emulators

salrashid123 commented 5 years ago

Sorry for the late update. As you know, the core issue is there are no package installers for gcloud which w're using in the images so far to install the other componetns. THere's no real reason to use the installers here since we've got named version tags for each release..(i.e the apt- installers dont' provide any benefit in this case)

Interms of what we'd do here is just not use the apt based installers and just do the install based on the .tar.gz here https://cloud.google.com/sdk/docs/quickstart-linux

right after
https://github.com/GoogleCloudPlatform/cloud-sdk-docker/blob/master/Dockerfile#L20

(just like the alpine image):

https://github.com/GoogleCloudPlatform/cloud-sdk-docker/blob/master/alpine/Dockerfile#L15

this would be pretty easy to do, we can get a PR for this going now

salrashid123 commented 5 years ago

the additional components will be live in the next SDK release

salrashid123 commented 5 years ago

Reopen: had to rollback on the PR since some dependencies were missed:

We'll need another PR like 158 but with the additional components highlighted installed via apt

(eg, apt-get install python2.7 openjdk-8-jdk)

$ apt-cache depends google-cloud-sdk
google-cloud-sdk
  Depends: python2.7   <<<<<<<<<<<<<<<<<<<<<
  Conflicts: gsutil
  Recommends: python-crcmod
  Suggests: google-cloud-sdk-app-engine-java
  Suggests: google-cloud-sdk-app-engine-python
  Suggests: google-cloud-sdk-pubsub-emulator
  Suggests: google-cloud-sdk-bigtable-emulator
  Suggests: google-cloud-sdk-datastore-emulator
  Suggests: kubectl

$ apt-cache depends google-cloud-sdk-app-engine-python
google-cloud-sdk-app-engine-python
  Depends: python2.7
  Depends: google-cloud-sdk

$ apt-cache depends google-cloud-sdk-app-engine-python-extras
google-cloud-sdk-app-engine-python-extras
  Depends: google-cloud-sdk
  Depends: google-cloud-sdk-app-engine-python

$ apt-cache depends google-cloud-sdk-app-engine-java
google-cloud-sdk-app-engine-java
  Depends: google-cloud-sdk-app-engine-python
 |Depends: <openjdk-7-jdk>
  Depends: openjdk-8-jdk

$ apt-cache depends google-cloud-sdk-app-engine-go
google-cloud-sdk-app-engine-go
  Depends: google-cloud-sdk
  Depends: google-cloud-sdk-app-engine-python

$ apt-cache depends google-cloud-sdk-datalab
google-cloud-sdk-datalab
  Depends: python2.7
  Depends: google-cloud-sdk

$ apt-cache depends google-cloud-sdk-datastore-emulator
google-cloud-sdk-datastore-emulator
  Depends: google-cloud-sdk
  Depends: openjdk-8-jdk   <<<<<<<<<<<<<<<<<<<<<<<

$ apt-cache depends google-cloud-sdk-pubsub-emulator
google-cloud-sdk-pubsub-emulator
  Depends: google-cloud-sdk
 |Depends: <openjdk-7-jdk>
  Depends: openjdk-8-jdk   

$ apt-cache depends google-cloud-sdk-bigtable-emulator
google-cloud-sdk-bigtable-emulator
  Depends: google-cloud-sdk

$ apt-cache depends google-cloud-sdk-cbt
google-cloud-sdk-cbt
  Depends: google-cloud-sdk

@gannett-ggreer

willfaris commented 4 years ago

The README now lists what component is installed for each image, ":latest" never had all components installed as not all components were|are packaged as deb packages and :latest is a deb based installation. https://github.com/GoogleCloudPlatform/cloud-sdk-docker#components-installed-in-each-tag

"Is it worth "unlocking" the gcloud install so it can be modified inside the container afterwards?" The install was never locked, but when you install google-cloud-sdk from a deb file the component subsystem is disabled, you need to install other components with apt.

the openjdk 8 issue was fixed by allowing the sid repo, hopefully that can be removed as soon as the pubsub-emulator verifies that it works on Java 11.