Unable to run Cloud Run sample Java application without memory issues

willmruzek commented 2 years ago

Environment:

CPU: ARM64 (M1 Mac)
RAM: 16GB
Managed Dependencies: off
Cloud SDK Version: 374.0.0
Skaffold Version: v1.36.0
Minikube Version: 1.25.2

Version information

Cloud Code Extension version: v1.17.0

VSCode version: v1.64.2

OS: MacOS 12.2.1 (21D62)

Kubectl: v1.23.4

Description:

I can't seem to get the Cloud Run sample Java application working in VSCode Cloud Code's Cloud Run Emulator (say that 10x fast).

I keep running into despite allocating 8GB RAM in Docker and setting high memory for the service in launch.json:

Waiting for deployments to stabilize...
 - deployment/hello-world-7: 0/1 nodes are available: 1 Insufficient memory.
    - pod/hello-world-7-5c7fd9ffd8-ncjb9: 0/1 nodes are available: 1 Insufficient memory.

Repro step:

Set Docker resource RAM to 8GB
Cloud Code: New Application
Cloud Run application
Java: Cloud Run
Disable gcp auth plugin in User and Workspace settings (known bug)
Run on Cloud Run Emulator
Set up build using Jib
Change configurations[0].service.resources.limits.memory to 4096Mi in .vscode/launch.json
Relaunch "Run on Cloud Run Emulator"

Logs

cloud-run-dev-internal docker container

+ userns=
+ grep -Eqv '0[[:space:]]+0[[:space:]]+4294967295' /proc/self/uid_map
+ validate_userns
+ [[ -z '' ]]
+ return
+ configure_containerd
++ stat -f -c %T /kind
+ [[ overlayfs == \z\f\s ]]
+ configure_proxy
+ mkdir -p /etc/systemd/system.conf.d/
+ [[ ! -z '' ]]
+ cat
+ fix_kmsg
+ [[ ! -e /dev/kmsg ]]
+ fix_mount
+ echo 'INFO: ensuring we can execute mount/umount even with userns-remap'
INFO: ensuring we can execute mount/umount even with userns-remap
++ which mount
++ which umount
+ chown root:root /usr/bin/mount /usr/bin/umount
++ which mount
++ which umount
+ chmod -s /usr/bin/mount /usr/bin/umount
+++ which mount
++ stat -f -c %T /usr/bin/mount
+ [[ overlayfs == \a\u\f\s ]]
+ [[ -z '' ]]
+ echo 'INFO: remounting /sys read-only'
INFO: remounting /sys read-only
+ mount -o remount,ro /sys
+ echo 'INFO: making mounts shared'
INFO: making mounts shared
+ mount --make-rshared /
+ retryable_fix_cgroup
++ seq 0 10
+ for i in $(seq 0 10)
+ fix_cgroup
+ [[ -f /sys/fs/cgroup/cgroup.controllers ]]
+ echo 'INFO: detected cgroup v2'
INFO: detected cgroup v2
+ return
+ return
+ fix_machine_id
+ echo 'INFO: clearing and regenerating /etc/machine-id'
INFO: clearing and regenerating /etc/machine-id
+ rm -f /etc/machine-id
+ systemd-machine-id-setup
Initializing machine ID from D-Bus machine ID.
+ fix_product_name
+ [[ -f /sys/class/dmi/id/product_name ]]
+ fix_product_uuid
+ [[ ! -f /kind/product_uuid ]]
+ cat /proc/sys/kernel/random/uuid
+ [[ -f /sys/class/dmi/id/product_uuid ]]
+ [[ -f /sys/devices/virtual/dmi/id/product_uuid ]]
+ select_iptables
+ local mode=nft
++ grep '^-'
++ wc -l
+ num_legacy_lines=6
+ '[' 6 -ge 10 ']'
++ grep '^-'
++ wc -l
++ true
+ num_nft_lines=0
+ '[' 6 -ge 0 ']'
+ mode=legacy
+ echo 'INFO: setting iptables to detected mode: legacy'
INFO: setting iptables to detected mode: legacy
+ update-alternatives --set iptables /usr/sbin/iptables-legacy
+ echo 'retryable update-alternatives: --set iptables /usr/sbin/iptables-legacy'
+ local 'args=--set iptables /usr/sbin/iptables-legacy'
++ seq 0 15
+ for i in $(seq 0 15)
+ /usr/bin/update-alternatives --set iptables /usr/sbin/iptables-legacy
+ return
+ update-alternatives --set ip6tables /usr/sbin/ip6tables-legacy
+ echo 'retryable update-alternatives: --set ip6tables /usr/sbin/ip6tables-legacy'
+ local 'args=--set ip6tables /usr/sbin/ip6tables-legacy'
++ seq 0 15
+ for i in $(seq 0 15)
+ /usr/bin/update-alternatives --set ip6tables /usr/sbin/ip6tables-legacy
+ return
+ enable_network_magic
+ local docker_embedded_dns_ip=127.0.0.11
+ local docker_host_ip
++ head -n1 /dev/fd/63
++ cut '-d ' -f1
+++ getent ahostsv4 host.docker.internal
+ docker_host_ip=192.168.65.2
+ [[ -z 192.168.65.2 ]]
+ [[ 192.168.65.2 =~ ^127\.[0-9]+\.[0-9]+\.[0-9]+$ ]]
+ iptables-save
+ iptables-restore
+ sed -e 's/-d 127.0.0.11/-d 192.168.65.2/g' -e 's/-A OUTPUT \(.*\) -j DOCKER_OUTPUT/\0\n-A PREROUTING \1 -j DOCKER_OUTPUT/' -e 's/--to-source :53/--to-source 192.168.65.2:53/g'
+ cp /etc/resolv.conf /etc/resolv.conf.original
+ sed -e s/127.0.0.11/192.168.65.2/g /etc/resolv.conf.original
++ cut '-d ' -f1
++ head -n1 /dev/fd/63
++++ hostname
+++ getent ahostsv4 cloud-run-dev-internal
+ curr_ipv4=192.168.49.2
+ echo 'INFO: Detected IPv4 address: 192.168.49.2'
INFO: Detected IPv4 address: 192.168.49.2
+ '[' -f /kind/old-ipv4 ']'
+ [[ -n 192.168.49.2 ]]
+ echo -n 192.168.49.2
++ head -n1 /dev/fd/63
++ cut '-d ' -f1
++++ hostname
+++ getent ahostsv6 cloud-run-dev-internal
+ curr_ipv6=
+ echo 'INFO: Detected IPv6 address: '
INFO: Detected IPv6 address: 
+ '[' -f /kind/old-ipv6 ']'
+ [[ -n '' ]]
++ uname -a
+ echo 'entrypoint completed: Linux cloud-run-dev-internal 5.10.76-linuxkit #1 SMP PREEMPT Mon Nov 8 11:22:26 UTC 2021 aarch64 aarch64 aarch64 GNU/Linux'
entrypoint completed: Linux cloud-run-dev-internal 5.10.76-linuxkit #1 SMP PREEMPT Mon Nov 8 11:22:26 UTC 2021 aarch64 aarch64 aarch64 GNU/Linux
+ exec /sbin/init
Failed to find module 'autofs4'
systemd 245.4-4ubuntu3.13 running in system mode. (+PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=hybrid)
Detected virtualization docker.
Detected architecture arm64.

Welcome to Ubuntu 20.04.2 LTS!

Set hostname to <cloud-run-dev-internal>.
[  OK  ] Started Dispatch Password …ts to Console Directory Watch.
[UNSUPP] Starting of Arbitrary Exec…Automount Point not supported.
[  OK  ] Reached target Local Encrypted Volumes.
[  OK  ] Reached target Network is Online.
[  OK  ] Reached target Paths.
[  OK  ] Reached target Slices.
[  OK  ] Reached target Swap.
[  OK  ] Listening on Journal Audit Socket.
[  OK  ] Listening on Journal Socket (/dev/log).
[  OK  ] Listening on Journal Socket.
         Mounting Huge Pages File System...
         Mounting Kernel Debug File System...
         Mounting Kernel Trace File System...
         Starting Journal Service...
         Starting Create list of st…odes for the current kernel...
         Mounting FUSE Control File System...
         Starting Remount Root and Kernel File Systems...
         Starting Apply Kernel Variables...
[  OK  ] Mounted Huge Pages File System.
[  OK  ] Mounted Kernel Debug File System.
[  OK  ] Mounted Kernel Trace File System.
[  OK  ] Finished Create list of st… nodes for the current kernel.
[  OK  ] Mounted FUSE Control File System.
[  OK  ] Finished Remount Root and Kernel File Systems.
         Starting Create System Users...
         Starting Update UTMP about System Boot/Shutdown...
[  OK  ] Finished Apply Kernel Variables.
[  OK  ] Started Journal Service.
         Starting Flush Journal to Persistent Storage...
[  OK  ] Finished Update UTMP about System Boot/Shutdown.
[  OK  ] Finished Flush Journal to Persistent Storage.
[  OK  ] Finished Create System Users.
         Starting Create Static Device Nodes in /dev...
[  OK  ] Finished Create Static Device Nodes in /dev.
[  OK  ] Reached target Local File Systems (Pre).
[  OK  ] Reached target Local File Systems.
[  OK  ] Reached target System Initialization.
[  OK  ] Started Daily Cleanup of Temporary Directories.
[  OK  ] Reached target Timers.
[  OK  ] Listening on BuildKit.
[  OK  ] Listening on D-Bus System Message Bus Socket.
         Starting Docker Socket for the API.
         Starting Podman API Socket.
[  OK  ] Listening on Docker Socket for the API.
[  OK  ] Listening on Podman API Socket.
[  OK  ] Reached target Sockets.
[  OK  ] Reached target Basic System.
         Starting containerd container runtime...
[  OK  ] Started D-Bus System Message Bus.
         Starting minikube automount...
         Starting OpenBSD Secure Shell server...
[  OK  ] Started OpenBSD Secure Shell server.
[  OK  ] Finished minikube automount.
[  OK  ] Started containerd container runtime.
         Starting Docker Application Container Engine...
[  OK  ] Started Docker Application Container Engine.
[  OK  ] Reached target Multi-User System.
[  OK  ] Reached target Graphical Interface.
         Starting Update UTMP about System Runlevel Changes...
[  OK  ] Finished Update UTMP about System Runlevel Changes.

Cloud Run: Run/Debug Locally - Detailed output log


Running skaffold command: /opt/homebrew/bin/skaffold init --artifact {"image":"hello-world-7","builder":"Jib Maven Plugin","payload":{"path":"pom.xml","project":"org.springframework.boot:hello-world"}} --kubernetes-manifest /var/folders/q6/qq3g1dqn7t3ftpsmtpks57k80000gn/T/cloud-code-cloud-run-zrAkCO/pods_and_services.yaml --filename /var/folders/q6/qq3g1dqn7t3ftpsmtpks57k80000gn/T/cloud-code-cloud-run-zrAkCO/skaffold.yaml --force
Setting build tagPolicy to SHA256.
Setting statusCheckDeadlineSeconds to 5 minutes
Setting build environment to 'local'
Created the file /var/folders/q6/qq3g1dqn7t3ftpsmtpks57k80000gn/T/cloud-code-cloud-run-zrAkCO/skaffold.yaml.
Running gcloud command: /opt/homebrew/bin/gcloud beta code export --readiness-probe --service-name hello-world-7 --image hello-world-7 --no-skaffold-file --dockerfile= --kubernetes-file /var/folders/q6/qq3g1dqn7t3ftpsmtpks57k80000gn/T/cloud-code-cloud-run-zrAkCO/pods_and_services.yaml --env-vars K_SERVICE=hello-world-7,K_REVISION=local,K_CONFIGURATION=hello-world-7 --memory 4096Mi --quiet --verbosity debug
Created the file /var/folders/q6/qq3g1dqn7t3ftpsmtpks57k80000gn/T/cloud-code-cloud-run-zrAkCO/pods_and_services.yaml.
Running: skaffold dev -v info --port-forward --rpc-http-port 58589 --filename /var/folders/q6/qq3g1dqn7t3ftpsmtpks57k80000gn/T/cloud-code-cloud-run-zrAkCO/skaffold.yaml --kube-context cloud-run-dev-internal --minikube-profile cloud-run-dev-internal --status-check
Activated kube-context \"cloud-run-dev-internal\"
starting gRPC server on port 58665
starting gRPC HTTP server on port 58589 (proxying to 58665)
Skaffold &{Version:v1.36.0 ConfigVersion:skaffold/v2beta27 GitVersion: GitCommit:189a55291c18ac850277134d2b8f3eaa2c4f7a1d BuildDate:2022-02-10T00:55:37Z GoVersion:go1.17.6 Compiler:gc Platform:darwin/arm64 User:}
Loaded Skaffold defaults from \"/Users/mruzekw/.skaffold/config\"
map entry found when executing locate for &{hello-world-7 . <nil> {<nil> <nil> <nil> 0x140003b7310 <nil> <nil> <nil>} [] {[] []} []} of type *v1.Artifact and pointer: 1374390758016
Using kubectl context: cloud-run-dev-internal
Using minikube docker daemon at tcp://127.0.0.1:58593
build concurrency first set to 1 parsed from *local.Builder[0]
final build concurrency value is 1
Listing files to watch...
 - hello-world-7
List generated in 1.351 second
Generating tags...
 - hello-world-7 -> hello-world-7:latest
Tags generated in 175.041µs
Checking cache...
 - hello-world-7: Found Locally
Cache check completed in 8.450375ms
Tags used in deployment:
 - hello-world-7 -> hello-world-7:93c2dafd944f568cd22724d732b148acb9424419fb2524db30cfcfc582788337
Starting deploy...
 - Warning: spec.template.spec.containers[0].env[4].name: duplicate name "K_CONFIGURATION"
 - Warning: spec.template.spec.containers[0].env[5].name: duplicate name "K_REVISION"
 - Warning: spec.template.spec.containers[0].env[6].name: duplicate name "K_SERVICE"
 - deployment.apps/hello-world-7 created
 - service/hello-world-7 created
Deploy completed in 1.645 second
Waiting for deployments to stabilize...
 - deployment/hello-world-7: 0/1 nodes are available: 1 Insufficient memory.
    - pod/hello-world-7-5c7fd9ffd8-ncjb9: 0/1 nodes are available: 1 Insufficient memory.
Cleaning up...
 - deployment.apps "hello-world-7" deleted
 - service "hello-world-7" deleted
Cleanup completed in 208.99975ms
1/1 deployment(s) failed
Skaffold exited with code 1.
Cleaning up...
Deleted the temporary directory /var/folders/q6/qq3g1dqn7t3ftpsmtpks57k80000gn/T/cloud-code-cloud-run-zrAkCO.
``` my

ssmall commented 2 years ago

Hi @mruzekw,

Thanks for the report and for providing all the detailed repro steps! Following the steps you provided, I attempted to reproduce the issue on my own M1 Mac but was unable to do so. I have a couple follow-up questions / suggestions to see if we can figure out why it's not working.

Additional Details:

Can you please provide the full contents of .vscode/launch.json and .vscode/settings.json?
Can you please provide the output of docker inspect cloud-run-dev-internal | grep \"Memory\"?

Steps to Try: (please try running again after each step so that we know which, if any, resolves the issue)

Restart minikube with the following command: minikube delete --profile=cloud-run-dev-internal && minikube start --profile=cloud-run-dev-internal
Confirm CPU and Memory settings for Docker Desktop and then restart Docker desktop
Explicitly configure memory limit for minikube by running the following commands: minikube config set --profile cloud-run-dev-internal memory 4096 && minikube delete --profile=cloud-run-dev-internal && minikube start --profile=cloud-run-dev-internal

Hopefully one of these steps will work around the issue you're experiencing, however if it does not please let me know if the behavior changes in any observable way so that we can continue to investigate!

kschaab commented 1 year ago

Closing due to lack of response

GoogleCloudPlatform / cloud-code-vscode