mathieuruellan
commented
3 years ago Hello, i've you find a solution? I'm stuck with the same situation with a debian testing.
Open itendtolosealot opened 3 years ago
mathieuruellan
commented
3 years ago Hello, i've you find a solution? I'm stuck with the same situation with a debian testing.
itendtolosealot
commented
3 years ago No. The error messages are not helpful. Also not clear which shell script is failing for us to debug it better.
mathieuruellan
commented
3 years ago Here is a log file on debian testing/bulleyes with Docker version 20.10.7, build f0df350. Hope it can help to figure out what's going wrong!
minikube_logs_f1b3b67a9c7d37e42dc4dad238e1c3ded160efda_0.log
afbjorklund
commented
3 years ago It seems that for some reason, it is trying to execute the systemd unit rather than writing it...
I0519 09:45:25.786161 125527 provision.go:296] Updating docker unit: /lib/systemd/system/docker.service ...There is a bogus extra line in the output, and it is messing up the quoting for the printf:
Environment="HTTP_PROXY=http://10.64.215.254:8080/"
Environment="HTTPS_PROXY=http://10.64.215.254:8080/"
Environment="NO_PROXY=localhost,127.0.0.1,localaddress,.localdomain.com" "
Environment="NO_PROXY=localhost,127.0.0.1,localaddress,.localdomain.com"So updateUnit in provision could use some more escaping, but there is also a bug in the content from the ubuntu provisioner.
if _, err := p.SSHCommand(fmt.Sprintf("sudo mkdir -p %s && printf %%s \"%s\" | sudo tee %s.new", path.Dir(dst), content, dst)); err != nil {
return err
}GenerateDockerOptions
engineConfigTmpl += `
{{range .EngineOptions.Env}}Environment={{.}}
{{end}}
Setting the proxy environment variables to bogus values will reproduce this:
NO_PROXY="localhost,127.0.0.1,localaddress,.localdomain.com\" " minikube startThat is, if there is an extra quote in the variable it will destroy the setup later on.
[Service]
Type=notify
Restart=on-failure
Environment="NO_PROXY=localhost,127.0.0.1,localaddress,.localdomain.com" "
# This file is a systemd drop-in unit that inherits from the base dockerd configuration.So there needs to be some validation of input, and also escaping of the content...
Note that since this is not the log from the first start, it is missing some information.
Also, this output is from a different bug: "Your cgroup does not allow setting memory."
itendtolosealot
commented
3 years ago Hi, Are there some work-arounds that I can use to get around? In my environment, I am not modifying any ENV variables. It is the act of using a different repo for minikube images (not the Internet) that is causing the issue. If I deploy it with the Internet images, then everything works as expected.
The images are the same in both the Internet and local repo. I checked even the hashes and they match. Ashvin
afbjorklund
commented
3 years ago Can you share your proxy configuration ? (env | grep -i proxy)
Specifically the NO_PROXY, but also http_proxy/https_proxy. (replace details like internal server names, if necessary)
mathieuruellan
commented
3 years ago Hello @afbjorklund.
In my case i havn't proxy configuration.
export|grep -iF "proxy" or env|grep -iF "proxy" return nothing
afbjorklund
commented
3 years ago Need the logs from the first (initial) start, otherwise it just says "trying to start running container" or somesuch.
mathieuruellan
commented
3 years ago here is the log after a start form scratch (I have removed .minikube and .kube directories)
Feel free to ask more details
afbjorklund
commented
3 years ago Seems like this start was successful, and there are no environment variables at all it seems ?
[Service]
Type=notify
Restart=on-failure
# This file is a systemd drop-in unit that inherits from the base dockerd configuration.Hopefully minikube will soon stop this nonsense of replacing the docker systemd config.
It's much better when just adding the few settings that are actually modified (like with cri-o)
afbjorklund
commented
3 years ago Other places (containerd) do base64 encoding of the content, which is probably a better way to go.
c := exec.Command("/bin/bash", "-c", fmt.Sprintf("sudo mkdir -p %s && printf %%s \"%s\" | base64 -d | sudo tee %s", path.Dir(cPath), base64.StdEncoding.EncodeToString(b.Bytes()), cPath))
if _, err := cr.RunCmd(c); err != nil {
return errors.Wrap(err, "generate containerd cfg.")
} if _, err := p.SSHCommand(fmt.Sprintf("sudo mkdir -p %s && printf %%s \"%s\" | sudo tee %s.new", path.Dir(dst), content, dst)); err != nil {
return err
}
I wouldn't say that the start is sucessful minikube.log
(I'm just a user, I can't do more that doing my best to provide my context or traces).
afbjorklund
commented
3 years ago I wouldn't say that the start is sucessful
Maybe I should have said "printf successful" :-)
k8s-triage-robot
commented
3 years ago The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.
This bot triages issues and PRs according to the following rules:
lifecycle/stale is appliedlifecycle/stale was applied, lifecycle/rotten is appliedlifecycle/rotten was applied, the issue is closedYou can:
/remove-lifecycle stale/lifecycle rotten/closePlease send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale
k8s-triage-robot
commented
3 years ago The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.
This bot triages issues and PRs according to the following rules:
lifecycle/stale is appliedlifecycle/stale was applied, lifecycle/rotten is appliedlifecycle/rotten was applied, the issue is closedYou can:
/remove-lifecycle rotten/closePlease send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle rotten
Steps to reproduce the issue:
[Service] Type=notify Restart=on-failure
Environment="HTTP_PROXY=http://10.64.215.254:8080/" Environment="HTTPS_PROXY=http://10.64.215.254:8080/" Environment="NO_PROXY=localhost,127.0.0.1,localaddress,.localdomain.com" " Environment="NO_PROXY=localhost,127.0.0.1,localaddress,.localdomain.com"
This file is a systemd drop-in unit that inherits from the base dockerd configuration.
The base configuration already specifies an 'ExecStart=...' command. The first directive
here is to clear out that command inherited from the base configuration. Without this,
the command from the base configuration and the command specified here are treated as
a sequence of commands, which is not the desired behavior, nor is it valid -- systemd
will catch this invalid input and refuse to start the service with an error like:
Service has more than one ExecStart= setting, which is only allowed for Type=oneshot services.
NOTE: default-ulimit=nofile is set to an arbitrary number for consistency with other
container runtimes. If left unlimited, it may result in OOM issues with MySQL.
ExecStart= ExecStart=/usr/bin/dockerd -H tcp://0.0.0.0:2376 -H unix:///var/run/docker.sock --default-ulimit=nofile=1048576:1048576 --tlsverify --tlscacert /etc/docker/ca.pem --tlscert /etc/docker/server.pem --tlskey /etc/docker/server-key.pem --label provider=docker --insecure-registry 10.96.0.0/12 ExecReload=/bin/kill -s HUP \$MAINPID
Having non-zero Limit*s causes performance problems due to accounting overhead
in the kernel. We recommend using cgroups to do container-local accounting.
LimitNOFILE=infinity LimitNPROC=infinity LimitCORE=infinity
Uncomment TasksMax if your systemd version supports it.
Only systemd 226 and above support this version.
TasksMax=infinity TimeoutStartSec=0
set delegate yes so that systemd does not reset the cgroups of docker containers
Delegate=yes
kill only the docker process, not all processes in the cgroup
KillMode=process
[Install] WantedBy=multi-user.target " | sudo tee /lib/systemd/system/docker.service.new err : Process exited with status 1 output : [Unit] Description=Docker Application Container Engine Documentation=https://docs.docker.com BindsTo=containerd.service After=network-online.target firewalld.service containerd.service Wants=network-online.target Requires=docker.socket StartLimitBurst=3 StartLimitIntervalSec=60
[Service] Type=notify Restart=on-failure
Environment=HTTP_PROXY=http://10.64.215.254:8080/ Environment=HTTPS_PROXY=http://10.64.215.254:8080/ Environment=NO_PROXY=localhost,127.0.0.1,localaddress,.localdomain.com bash: line 31: -H: command not found bash: line 32: -s: command not found bash: line 51: [Install]: command not found bash: -c: line 53: unexpected EOF while looking for matching `"' bash: -c: line 54: syntax error: unexpected end of file
[Service] Type=notify Restart=on-failure
Environment="HTTP_PROXY=http://10.64.215.254:8080/" Environment="HTTPS_PROXY=http://10.64.215.254:8080/" Environment="NO_PROXY=localhost,127.0.0.1,localaddress,.localdomain.com" " Environment="NO_PROXY=localhost,127.0.0.1,localaddress,.localdomain.com"
This file is a systemd drop-in unit that inherits from the base dockerd configuration.
The base configuration already specifies an 'ExecStart=...' command. The first directive
here is to clear out that command inherited from the base configuration. Without this,
the command from the base configuration and the command specified here are treated as
a sequence of commands, which is not the desired behavior, nor is it valid -- systemd
will catch this invalid input and refuse to start the service with an error like:
Service has more than one ExecStart= setting, which is only allowed for Type=oneshot services.
NOTE: default-ulimit=nofile is set to an arbitrary number for consistency with other
container runtimes. If left unlimited, it may result in OOM issues with MySQL.
ExecStart= ExecStart=/usr/bin/dockerd -H tcp://0.0.0.0:2376 -H unix:///var/run/docker.sock --default-ulimit=nofile=1048576:1048576 --tlsverify --tlscacert /etc/docker/ca.pem --tlscert /etc/docker/server.pem --tlskey /etc/docker/server-key.pem --label provider=docker --insecure-registry 10.96.0.0/12 ExecReload=/bin/kill -s HUP \$MAINPID
Having non-zero Limit*s causes performance problems due to accounting overhead
in the kernel. We recommend using cgroups to do container-local accounting.
LimitNOFILE=infinity LimitNPROC=infinity LimitCORE=infinity
Uncomment TasksMax if your systemd version supports it.
Only systemd 226 and above support this version.
TasksMax=infinity TimeoutStartSec=0
set delegate yes so that systemd does not reset the cgroups of docker containers
Delegate=yes
kill only the docker process, not all processes in the cgroup
KillMode=process
[Install] WantedBy=multi-user.target " | sudo tee /lib/systemd/system/docker.service.new err : Process exited with status 1 output : [Unit] Description=Docker Application Container Engine Documentation=https://docs.docker.com BindsTo=containerd.service After=network-online.target firewalld.service containerd.service Wants=network-online.target Requires=docker.socket StartLimitBurst=3 StartLimitIntervalSec=60
[Service] Type=notify Restart=on-failure
Environment=HTTP_PROXY=http://10.64.215.254:8080/ Environment=HTTPS_PROXY=http://10.64.215.254:8080/ Environment=NO_PROXY=localhost,127.0.0.1,localaddress,.localdomain.com bash: line 31: -H: command not found bash: line 32: -s: command not found bash: line 51: [Install]: command not found bash: -c: line 53: unexpected EOF while looking for matching `"' bash: -c: line 54: syntax error: unexpected end of file
X Exiting due to GUEST_PROVISION: Failed to start host: provision: ssh command error: command : sudo mkdir -p /lib/systemd/system && printf %s "[Unit] Description=Docker Application Container Engine Documentation=https://docs.docker.com BindsTo=containerd.service After=network-online.target firewalld.service containerd.service Wants=network-online.target Requires=docker.socket StartLimitBurst=3 StartLimitIntervalSec=60
[Service] Type=notify Restart=on-failure
Environment="HTTP_PROXY=http://10.64.215.254:8080/" Environment="HTTPS_PROXY=http://10.64.215.254:8080/" Environment="NO_PROXY=localhost,127.0.0.1,localaddress,.localdomain.com" " Environment="NO_PROXY=localhost,127.0.0.1,localaddress,.localdomain.com"
This file is a systemd drop-in unit that inherits from the base dockerd configuration.
The base configuration already specifies an 'ExecStart=...' command. The first directive
here is to clear out that command inherited from the base configuration. Without this,
the command from the base configuration and the command specified here are treated as
a sequence of commands, which is not the desired behavior, nor is it valid -- systemd
will catch this invalid input and refuse to start the service with an error like:
Service has more than one ExecStart= setting, which is only allowed for Type=oneshot services.
NOTE: default-ulimit=nofile is set to an arbitrary number for consistency with other
container runtimes. If left unlimited, it may result in OOM issues with MySQL.
ExecStart= ExecStart=/usr/bin/dockerd -H tcp://0.0.0.0:2376 -H unix:///var/run/docker.sock --default-ulimit=nofile=1048576:1048576 --tlsverify --tlscacert /etc/docker/ca.pem --tlscert /etc/docker/server.pem --tlskey /etc/docker/server-key.pem --label provider=docker --insecure-registry 10.96.0.0/12 ExecReload=/bin/kill -s HUP \$MAINPID
Having non-zero Limit*s causes performance problems due to accounting overhead
in the kernel. We recommend using cgroups to do container-local accounting.
LimitNOFILE=infinity LimitNPROC=infinity LimitCORE=infinity
Uncomment TasksMax if your systemd version supports it.
Only systemd 226 and above support this version.
TasksMax=infinity TimeoutStartSec=0
set delegate yes so that systemd does not reset the cgroups of docker containers
Delegate=yes
kill only the docker process, not all processes in the cgroup
KillMode=process
[Install] WantedBy=multi-user.target " | sudo tee /lib/systemd/system/docker.service.new err : Process exited with status 1 output : [Unit] Description=Docker Application Container Engine Documentation=https://docs.docker.com BindsTo=containerd.service After=network-online.target firewalld.service containerd.service Wants=network-online.target Requires=docker.socket StartLimitBurst=3 StartLimitIntervalSec=60
[Service] Type=notify Restart=on-failure
Environment=HTTP_PROXY=http://10.64.215.254:8080/ Environment=HTTPS_PROXY=http://10.64.215.254:8080/ Environment=NO_PROXY=localhost,127.0.0.1,localaddress,.localdomain.com bash: line 31: -H: command not found bash: line 32: -s: command not found bash: line 51: [Install]: command not found bash: -c: line 53: unexpected EOF while looking for matching `"' bash: -c: line 54: syntax error: unexpected end of file
╭──────────────────────────────────────────────────────────────────╮ │ │ │ If the above advice does not help, please let us know: │ │ https://github.com/kubernetes/minikube/issues/new/choose │ │ │ │ Please attach the following file to the GitHub issue: │ │ - /home/administrator/.minikube/logs/lastStart.txt │ │ │ ╰──────────────────────────────────────────────────────────────────╯
Full output of
minikube logscommand:Full output of failed command:
Anythoughts on what I could be doing wrong? Is this a missing package?