search

wey-gu / nebula-up

One-liner NebulaGraph playground with allllllllll-in-one toolchain integrated on single Linux Server
https://siwei.io/nebula-up
63 stars 16 forks source link

docker change cn mirror seems to fail #24

Closed liuxiaocs7 closed 2 years ago

liuxiaocs7 commented 2 years ago

When I execute curl -fsSL nebula-up.siwei.io/all-in-one.sh | bash -s -- v3 spark in my terminal.

It seems not change the docker mirror to 163 mirror, bash: /etc/docker/daemon.json: 没有那个文件或目录

My System is CentOS7.6, login by user root.

Logs are as follows:

+ sh -c 'yum install -y -q docker-ce docker-ce-cli containerd.io docker-scan-plugin docker-compose-plugin docker-ce-rootless-extras'
warning: /var/cache/yum/x86_64/7/docker-ce-stable/packages/docker-ce-20.10.17-3.el7.x86_64.rpm: Header V4 RSA/SHA512 Signature, key ID 621e9f35: NOKEY
docker-ce-20.10.17-3.el7.x86_64.rpm 的公钥尚未安装
导入 GPG key 0x621E9F35:
 用户ID     : "Docker Release (CE rpm) <docker@docker.com>"
 指纹       : 060a 61c5 1b55 8a7f 742b 77aa c52f eb6b 621e 9f35
 来自       : https://download.docker.com/linux/centos/gpg

================================================================================

To run Docker as a non-privileged user, consider setting up the
Docker daemon in rootless mode for your user:

    dockerd-rootless-setuptool.sh install

Visit https://docs.docker.com/go/rootless/ to learn about rootless mode.

To run the Docker daemon as a fully privileged service, but granting non-root
users access, refer to https://docs.docker.com/go/daemon-access/

WARNING: Access to the remote API on a privileged Docker daemon is equivalent
         to root access on the host. Refer to the 'Docker daemon attack surface'
         documentation for details: https://docs.docker.com/go/attack-surface/

================================================================================

bash: /etc/docker/daemon.json: 没有那个文件或目录

 ℹ️    Waiting for Docker Engine to be up...
 ✔️    docker engine is up.

 ℹ️    Ensuring Linux Docker Permission

 ℹ️    Starting Instlation of Docker-Compose
wey-gu commented 2 years ago

Thanks @liuxiaocs7 I'll look into this :), it turned out that by default this folder/file doesn't exist, I will add this awareness and creation routines later!

wey-gu commented 2 years ago

ref: https://stackoverflow.com/questions/69464001/docker-compose-container-name-use-dash-instead-of-underscore

would you mind help share below version commands?

azureuser@nebula-up:~$ docker-compose version
docker-compose version 1.29.0, build 07737305
docker-py version: 5.0.0
CPython version: 3.7.10
OpenSSL version: OpenSSL 1.1.0l  10 Sep 2019
azureuser@nebula-up:~$ docker compose version
Docker Compose version v2.5.0
azureuser@nebula-up:~$ docker version
Client: Docker Engine - Community
 Version:           20.10.16
 API version:       1.41
 Go version:        go1.17.10
 Git commit:        aa7e414
 Built:             Thu May 12 09:17:23 2022
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.16
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.17.10
  Git commit:       f756502
  Built:            Thu May 12 09:15:28 2022
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.4
  GitCommit:        212e8b6fa2f44b9c21b2798135fc6fb7c53efc16
 runc:
  Version:          1.1.1
  GitCommit:        v1.1.1-0-g52de29d
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0
liuxiaocs7 commented 2 years ago

sure

[root@liuxiaocs01 ~]# docker-compose version
Docker Compose version v2.6.0
[root@liuxiaocs01 ~]# docker compose version
Docker Compose version v2.6.0
[root@liuxiaocs01 ~]# docker version
Client: Docker Engine - Community
 Version:           20.10.17
 API version:       1.41
 Go version:        go1.17.11
 Git commit:        100c701
 Built:             Mon Jun  6 23:05:12 2022
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.17
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.17.11
  Git commit:       a89b842
  Built:            Mon Jun  6 23:03:33 2022
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.6
  GitCommit:        10c12954828e7c7c9b6e0ea9b0c02b01407d3ae1
 runc:
  Version:          1.1.2
  GitCommit:        v1.1.2-0-ga916309
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0
liuxiaocs7 commented 2 years ago

I noticed that the folder /etc/docker exists after docker installed, but the file daemon.json does not exist.

The following statement should create the file, but it does not, puzzling.

sudo bash -c "cat > ${DOCKER_CONF_PATH}/daemon.json" << EOF
{
  "registry-mirrors": [
    "https://hub-mirror.c.163.com"
  ]
}
EOF
wey-gu commented 2 years ago

sure

[root@liuxiaocs01 ~]# docker-compose version
Docker Compose version v2.6.0
[root@liuxiaocs01 ~]# docker compose version
Docker Compose version v2.6.0
[root@liuxiaocs01 ~]# docker version
Client: Docker Engine - Community
 Version:           20.10.17
 API version:       1.41
 Go version:        go1.17.11
 Git commit:        100c701
 Built:             Mon Jun  6 23:05:12 2022
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.17
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.17.11
  Git commit:       a89b842
  Built:            Mon Jun  6 23:03:33 2022
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.6
  GitCommit:        10c12954828e7c7c9b6e0ea9b0c02b01407d3ae1
 runc:
  Version:          1.1.2
  GitCommit:        v1.1.2-0-ga916309
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

Thanks! This explains #25 that docker-compose leveraged the go implementation of compose when the python 1.x docker-compose is not installed, where it convents with - instead of _.

wey-gu commented 2 years ago

I noticed that the folder /etc/docker exists after docker installed, but the file daemon.json does not exist.

The following statement should create the file, but it does not, puzzling.

sudo bash -c "cat > ${DOCKER_CONF_PATH}/daemon.json" << EOF
{
  "registry-mirrors": [
    "https://hub-mirror.c.163.com"
  ]
}
EOF

oh, that's interesting, I thought the issue was caused by folder not exist, let me check and reopen it for now.

wey-gu commented 2 years ago

Could you please help copy this to test0.sh?

DOCKER_CONF_PATH="/etc/docker"
sudo bash -c "cat > ${DOCKER_CONF_PATH}/daemon_1.json" << EOF
{
  "registry-mirrors": [
    "https://hub-mirror.c.163.com"
  ]
}
EOF

And run this for a minimal reproduction?

ls -l /etc/docker
bash test0.sh
ls -l /etc/docker
liuxiaocs7 commented 2 years ago

Could you please help copy this to test0.sh?

DOCKER_CONF_PATH="/etc/docker"
sudo bash -c "cat > ${DOCKER_CONF_PATH}/daemon_1.json" << EOF
{
  "registry-mirrors": [
    "https://hub-mirror.c.163.com"
  ]
}
EOF

And run this for a minimal reproduction?

ls -l /etc/docker
bash test0.sh
ls -l /etc/docker

Sure, in fact i have already test just like it, i will try again.

Result are as follow, please ignore the daemon.json, because i create it manually.

[root@liuxiaocs01 ~]# vim test0.sh
[root@liuxiaocs01 ~]# cat test0.sh
DOCKER_CONF_PATH="/etc/docker"
sudo bash -c "cat > ${DOCKER_CONF_PATH}/daemon_1.json" << EOF
{
  "registry-mirrors": [
    "https://hub-mirror.c.163.com"
  ]
}
EOF
[root@liuxiaocs01 ~]# ll /etc/docker
总用量 8
-rw-r--r--. 1 root root  67 7月  16 14:30 daemon.json
-rw-------. 1 root root 244 7月  16 14:12 key.json
[root@liuxiaocs01 ~]# bash test0.sh
[root@liuxiaocs01 ~]# ll /etc/docker/
总用量 12
-rw-r--r--. 1 root root  67 7月  16 15:57 daemon_1.json
-rw-r--r--. 1 root root  67 7月  16 14:30 daemon.json
-rw-------. 1 root root 244 7月  16 14:12 key.json
[root@liuxiaocs01 ~]# cat /etc/docker/daemon_1.json
{
  "registry-mirrors": [
    "https://hub-mirror.c.163.com"
  ]
}
liuxiaocs7 commented 2 years ago

sure

[root@liuxiaocs01 ~]# docker-compose version
Docker Compose version v2.6.0
[root@liuxiaocs01 ~]# docker compose version
Docker Compose version v2.6.0
[root@liuxiaocs01 ~]# docker version
Client: Docker Engine - Community
 Version:           20.10.17
 API version:       1.41
 Go version:        go1.17.11
 Git commit:        100c701
 Built:             Mon Jun  6 23:05:12 2022
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.17
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.17.11
  Git commit:       a89b842
  Built:            Mon Jun  6 23:03:33 2022
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.6
  GitCommit:        10c12954828e7c7c9b6e0ea9b0c02b01407d3ae1
 runc:
  Version:          1.1.2
  GitCommit:        v1.1.2-0-ga916309
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

Thanks! This explains #25 that docker-compose leveraged the go implementation of compose when the python 1.x docker-compose is not installed, where it convents with - instead of _.

OK, got it, thanks.

wey-gu commented 2 years ago

Could you please help copy this to test0.sh?

DOCKER_CONF_PATH="/etc/docker"
sudo bash -c "cat > ${DOCKER_CONF_PATH}/daemon_1.json" << EOF
{
  "registry-mirrors": [
    "https://hub-mirror.c.163.com"
  ]
}
EOF

And run this for a minimal reproduction?

ls -l /etc/docker
bash test0.sh
ls -l /etc/docker

Sure, in fact i have already test just like it, i will try again.

Result are as follow, please ignore the daemon.json, because i create it manually.

[root@liuxiaocs01 ~]# vim test0.sh
[root@liuxiaocs01 ~]# cat test0.sh
DOCKER_CONF_PATH="/etc/docker"
sudo bash -c "cat > ${DOCKER_CONF_PATH}/daemon_1.json" << EOF
{
  "registry-mirrors": [
    "https://hub-mirror.c.163.com"
  ]
}
EOF
[root@liuxiaocs01 ~]# ll /etc/docker
总用量 8
-rw-r--r--. 1 root root  67 7月  16 14:30 daemon.json
-rw-------. 1 root root 244 7月  16 14:12 key.json
[root@liuxiaocs01 ~]# bash test0.sh
[root@liuxiaocs01 ~]# ll /etc/docker/
总用量 12
-rw-r--r--. 1 root root  67 7月  16 15:57 daemon_1.json
-rw-r--r--. 1 root root  67 7月  16 14:30 daemon.json
-rw-------. 1 root root 244 7月  16 14:12 key.json
[root@liuxiaocs01 ~]# cat /etc/docker/daemon_1.json
{
  "registry-mirrors": [
    "https://hub-mirror.c.163.com"
  ]
}

Seems to test fine, regarding:

The following statement should create the file, but it does not, puzzling.

"it does not" do you mean to test with curl xxx| bash with a similar failure? If that is the case, that's due to this file being cached and not synced in real-time, you could verify it with the URL of: "https://raw.githubusercontent.com/wey-gu/nebula-up/main/all-in-one.sh" to be called with cURL and bash instead?

liuxiaocs7 commented 2 years ago

Could you please help copy this to test0.sh?

DOCKER_CONF_PATH="/etc/docker"
sudo bash -c "cat > ${DOCKER_CONF_PATH}/daemon_1.json" << EOF
{
  "registry-mirrors": [
    "https://hub-mirror.c.163.com"
  ]
}
EOF

And run this for a minimal reproduction?

ls -l /etc/docker
bash test0.sh
ls -l /etc/docker

Sure, in fact i have already test just like it, i will try again. Result are as follow, please ignore the daemon.json, because i create it manually.

[root@liuxiaocs01 ~]# vim test0.sh
[root@liuxiaocs01 ~]# cat test0.sh
DOCKER_CONF_PATH="/etc/docker"
sudo bash -c "cat > ${DOCKER_CONF_PATH}/daemon_1.json" << EOF
{
  "registry-mirrors": [
    "https://hub-mirror.c.163.com"
  ]
}
EOF
[root@liuxiaocs01 ~]# ll /etc/docker
总用量 8
-rw-r--r--. 1 root root  67 7月  16 14:30 daemon.json
-rw-------. 1 root root 244 7月  16 14:12 key.json
[root@liuxiaocs01 ~]# bash test0.sh
[root@liuxiaocs01 ~]# ll /etc/docker/
总用量 12
-rw-r--r--. 1 root root  67 7月  16 15:57 daemon_1.json
-rw-r--r--. 1 root root  67 7月  16 14:30 daemon.json
-rw-------. 1 root root 244 7月  16 14:12 key.json
[root@liuxiaocs01 ~]# cat /etc/docker/daemon_1.json
{
  "registry-mirrors": [
    "https://hub-mirror.c.163.com"
  ]
}

Seems to test fine, regarding:

The following statement should create the file, but it does not, puzzling.

"it does not" do you mean to test with curl xxx| bash with a similar failure? If that is the case, that's due to this file being cached and not synced in real-time, you could verify it with the URL of: "https://raw.githubusercontent.com/wey-gu/nebula-up/main/all-in-one.sh" to be called with cURL and bash instead?

Yes, i execute curl -fsSL nebula-up.siwei.io/all-in-one.sh | bash -s -- v3 spark.

that's due to this file being cached and not synced in real-time

Thanks, i got it.

liuxiaocs7 commented 2 years ago

But it seems that when this command runs and ends, the file does not appear, how to sync to show it?

wey-gu commented 2 years ago

But it seems that when this command runs and ends, the file does not appear, how to sync to show it?

the .sh file in nebula-up.siwei.io will be synced in 10 min i think😛

do you mean this bash file or？

liuxiaocs7 commented 2 years ago

But it seems that when this command runs and ends, the file does not appear, how to sync to show it?

the .sh file in nebula-up.siwei.io will be synced in 10 min i think😛

do you mean this bash file or？

this file: /etc/docker/daemon.json, I don't see this file after the command done so i created it manually.

wey-gu commented 2 years ago

Oh, you meant even after curl -fsSL https://raw.githubusercontent.com/wey-gu/nebula-up/main/all-in-one.sh | bash -s -- v3 spark, there is no error complaining /etc/docker/daemon.json not exist but still, it's not created?

Another thing is that the trigger to create that file is that your server cannot reach google.com could you confirm this, too?

Thanks!

liuxiaocs7 commented 2 years ago

I have newly created a virtual machine and curl -fsSL nebula-up.siwei.io/all-in-one.sh | bash -s -- v3 spark seems to be working fine.

No log like bash: /etc/docker/daemon.json: 没有那个文件或目录 output.

[root@liuxiaocs01 ~]# cd /etc/docker
[root@liuxiaocs01 docker]# ls
daemon.json  key.json
[root@liuxiaocs01 docker]# cat daemon.json
{
  "registry-mirrors": [
    "https://hub-mirror.c.163.com"
  ]
}

this works, thank you.

sudo mkdir -p ${DOCKER_CONF_PATH}
wey-gu commented 2 years ago

OK, maybe you could verify this in the previous VM, I guess somewhere it cached an old version of .sh file :), in case not, there could be some other issues I need to look into.

curl -s nebula-up.siwei.io/all-in-one.sh | grep "DOCKER_CONF_PATH\|sparkmaster"
wey-gu commented 2 years ago

And thanks to you @liuxiaocs7 :D

liuxiaocs7 commented 2 years ago

OK, maybe you could verify this in the previous VM, I guess somewhere it cached an old version of .sh file :), in case not, there could be some other issues I need to look into.

curl -s nebula-up.siwei.io/all-in-one.sh | grep "DOCKER_CONF_PATH\|sparkmaster"

Now the script is updated and works fine. Thank you for your help, I have learned a lot.

[root@liuxiaocs01 ~]# curl -s nebula-up.siwei.io/all-in-one.sh | grep "DOCKER_CONF_PATH\|sparkmaster"
        *inux*)  DOCKER_CONF_PATH="/etc/docker" ;;
        *arwin*) DOCKER_CONF_PATH="$HOME/.docker" ;;
    sudo mkdir -p ${DOCKER_CONF_PATH}
    sudo bash -c "cat > ${DOCKER_CONF_PATH}/daemon.json" << EOF
sudo docker exec -it sparkmaster /spark/bin/pyspark --driver-class-path /root/download/nebula-spark-connector.jar --jars /root/download/nebula-spark-connector.jar
sudo docker exec -it sparkmaster /spark/bin/spark-submit --master local --class com.vesoft.nebula.exchange.Exchange /root/download/nebula-exchange.jar -c /root/exchange.conf
sudo docker exec -it sparkmaster /spark/bin/spark-submit --master "local" --conf spark.rpc.askTimeout=6000s \\
liuxiaocs7 commented 2 years ago

And thanks to you @liuxiaocs7 :D

My pleasure, :)

wey-gu commented 2 years ago

And thanks to you @liuxiaocs7 :D

My pleasure, :)

My pleasure, @liuxiaocs7 :)