google_notebooks_instance using container image does not work.

mikhail-khodorovskiy commented 3 years ago

Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request.
Please do not leave +1 or me too comments, they generate extra noise for issue followers and do not help prioritize the request.
If you are interested in working on this issue or have submitted a pull request, please leave a comment.
If an issue is assigned to the modular-magician user, it is either in the process of being autogenerated, or is planned to be autogenerated soon. If an issue is assigned to a user, that user is claiming responsibility for the issue. If an issue is assigned to hashibot, a community member has claimed the issue already.

Terraform Version

Terraform v0.13.5

Affected Resource(s)

google_XXXXX

Terraform v0.13.5 Google beta provider 3.52.0

Terraform Configuration Files

# Copy-paste your Terraform configurations here.
#
# For large Terraform configs, please use a service like Dropbox and share a link to the ZIP file.
# For security, you can also encrypt the files using our GPG public key:
#    https://www.hashicorp.com/security
#
# If reproducing the bug involves modifying the config file (e.g., apply a config,
# change a value, apply the config again, see the bug), then please include both:
# * the version of the config before the change, and
# * the version of the config after the change.

resource "google_notebooks_instance" "ai_notebook" {
  provider           = google-beta
  name               = "${var.common_data.environment_name}-${var.name}"
  location           = var.zone

  machine_type = "e2-medium"

  container_image {
    repository = "gcr.io/deeplearning-platform-release/base-cpu"
    tag        = "latest"
  }
  network = data.google_compute_network.datalab_network.self_link
  subnet  = data.google_compute_network.datalab_network.subnetworks_self_links[0]

  service_account = data.google_service_account.datalab_service_account.email
  labels            = var.common_data.common_labels
  instance_owners   = ["user:${var.ai_notebook_user_email}"]
  project           = var.common_data.project_id
  no_public_ip      = true
  no_proxy_access   = true
}

Debug Output

On the VM for the ai notebook VM, it looks like the startup script is missing formatting of the mount

Feb 8 21:12:33 hsq-dev-thomas-notebook-2 systemd[1]: Started containerd container runtime. Feb 8 21:12:33 hsq-dev-thomas-notebook-2 c2d-startup[369]: Add jupyter user: jupyter to docker Feb 8 21:12:33 hsq-dev-thomas-notebook-2 containerd[442]: time="2021-02-08T21:12:33.767995517Z" level=info msg=serving... address=/run/containerd/containerd.sock Feb 8 21:12:33 hsq-dev-thomas-notebook-2 containerd[442]: time="2021-02-08T21:12:33.779269570Z" level=info msg="containerd successfully booted in 0.222144s" Feb 8 21:12:33 hsq-dev-thomas-notebook-2 systemd[1]: Starting Docker Application Container Engine... Feb 8 21:12:33 hsq-dev-thomas-notebook-2 c2d-startup[369]: Run: usermod -aG docker jupyter Feb 8 21:12:33 hsq-dev-thomas-notebook-2 c2d-startup[369]: Docker group docker:x:998:jupyter Feb 8 21:12:33 hsq-dev-thomas-notebook-2 c2d-startup[369]: Jupyter user home directory: /home/jupyter Feb 8 21:12:33 hsq-dev-thomas-notebook-2 c2d-startup[369]: mount: /home/jupyter: wrong fs type, bad option, bad superblock on /dev/sdb, missing codepage or helper program, or other error. Feb 8 21:12:33 hsq-dev-thomas-notebook-2 GCEGuestAgent[420]: 2021-02-08T21:12:33.8505Z GCEGuestAgent Info: Instance ID changed, running first-boot actions

Panic Output

Expected Behavior

Actual Behavior

Steps to Reproduce

terraform apply

Important Factoids

References

0000

venkykuberan commented 3 years ago

I believe the resource creation is successful, right ?. Please try using any of the images from here - https://cloud.google.com/ai-platform/notebooks/docs/images which suits your need.

mikhail-khodorovskiy commented 3 years ago

We actually want to use a custom image we build but we get the same error that's why we tried to use one of the existing images first to see if that would work. We want to use

  container_image {
    repository = "gcr.io/<ourproject>/<our-image>"
    tag        = "latest"
  }

is that possible that when container_image is used the startup script fails to format the disk?

venkykuberan commented 3 years ago

Possible, were you able to get the instance w/ startup script working fine when you tried with console/gcloud ?. If you see same behavior then the issue needs to raised against the API team.

mikhail-khodorovskiy commented 3 years ago

You are right, the gcloud call creating the VM produces the same error:

gcloud beta notebooks instances create custom-instance-gcp-owned  --container-repository=gcr.io/deeplearning-platform-release/r-cpu --container-tag=latest --machine-type=n1-standard-16  --location=us-west1-b --boot-disk-size=300 --boot-disk-type=PD_SSD --project=hsq-dev-analytics-f8a4  --no-public-ip  --subnet=analytics-subnet --subnet-region=us-west1 --network=hsq-dev-analytics-network --service-account=hsq-dev-datalab@hsq-dev-analytics-f8a4.iam.gserviceaccount.com

produces

mount: /home/jupyter: wrong fs type, bad option, bad superblock on /dev/sdb, missing codepage or helper program, or other error.

We filed a support ticket with GCP.

ghost commented 3 years ago

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. If you feel I made an error 🤖 🙉 , please reach out to my human friends 👉 hashibot-feedback@hashicorp.com. Thanks!

hashicorp / terraform-provider-google