search

0chain / blobber

A storage provider (blobber) interface to the blockchain and consumers of storage.
Other
19 stars 23 forks source link

Blobber Postgres Container doesn't start/stop. #226

Closed NoSkillGuy closed 3 years ago

NoSkillGuy commented 3 years ago 
Stopping blobber1_blobber_1       ... done
Stopping blobber1_validator_1     ... done
Stopping blobber1_postgres_1      ...

ERROR: for blobber1_postgres_1  UnixHTTPConnectionPool(host='localhost', port=None): Read timed out. (read timeout=130)
ERROR: An HTTP request took too long to complete. Retry with --verbose to obtain debug information.
If you encounter this issue regularly because of slow network conditions, consider setting COMPOSE_HTTP_TIMEOUT to a higher value (current value: 120).

After stopping all the blobbers, all the Postgres containers remain running 

❯ docker ps | grep blobber                                                                                                                                                                                                                 ─╯
7ec4e9ae2658   postgres:11        "docker-entrypoint.s…"   10 minutes ago   Up 10 minutes   5432/tcp                                        blobber4_postgres_1
9e2e044cdcee   postgres:11        "docker-entrypoint.s…"   10 minutes ago   Up 10 minutes   5432/tcp                                        blobber3_postgres_1
ae026df16d83   postgres:11        "docker-entrypoint.s…"   10 minutes ago   Up 10 minutes   5432/tcp                                        blobber2_postgres_1
43fe5d90f356   postgres:11        "docker-entrypoint.s…"   18 minutes ago   Up 10 minutes   5432/tcp                                        blobber1_postgres_1

Environment Details: OS: macOS Big Sur Docker Desktop: 3.3.3

Tried these approaches to solve this problem

  1. manual docker stop, but gets timeout 

    docker stop blobber1_postgres_1

  2. Increased the timeout from 60 to 180 seconds. still gets timeout 

    export DOCKER_CLIENT_TIMEOUT=180                                                                                                                                                                                                         
export COMPOSE_HTTP_TIMEOUT=180

  3. Docker Desktop restart, since the docker socket connection gets closed. containers got exited. but next time when I want to stop the containers, again back to the same issue. They get timeout

  4. removing the container by force, still no luck :(. It gets stuck without returning any error!

    docker rm -f blobber1_postgres_1

Looks like this ultimately points to this issue in docker-compose.

NoSkillGuy commented 3 years ago

Now i am not able to start the container too. :( 

❯ ../bin/blobber.start_bls.sh                                                                                      ─╯
Starting blobber3 ...
Creating network "blobber3_default" with driver "bridge"
Creating volume "blobber3_data" with default driver
Creating volume "blobber3_config" with default driver
Creating volume "blobber3_bin" with default driver
Pulling postgres (postgres:11)...
11: Pulling from library/postgres
aed007321795: Already exists
702c74af4354: Already exists
ef123e2ab09b: Already exists
c84418d8530c: Already exists
b579800618be: Already exists
6562ab9a134a: Already exists
1c9daa412a3b: Already exists
ddb4502f2937: Already exists
02e243399656: Pull complete
39572b0b2f6f: Pull complete
e11139725f03: Pull complete
43726a854ff9: Pull complete
26a0f4ddb332: Pull complete
Digest: sha256:7211ad97f0ee411fc15fdde06671c38f735273c3933b5ee8b9d169e07d82303f
Status: Downloaded newer image for postgres:11
Creating blobber3_postgres_1 ...

ERROR: for blobber3_postgres_1  UnixHTTPConnectionPool(host='localhost', port=None): Read timed out. (read timeout=60)

ERROR: for postgres  UnixHTTPConnectionPool(host='localhost', port=None): Read timed out. (read timeout=60)
ERROR: An HTTP request took too long to complete. Retry with --verbose to obtain debug information.
If you encounter this issue regularly because of slow network conditions, consider setting COMPOSE_HTTP_TIMEOUT to a higher value (current value: 60).
cnlangzi commented 3 years ago

Docker is stuck. It can be fixed after restarting docker engine

cnlangzi commented 3 years ago

Docker is very easy to get stuck when sharder, miner and blobber are running at the same time. You can check ./dev.local/cli.sh on branch dayi/main. It helps you to run blobbers outside docker

NoSkillGuy commented 3 years ago

@cnlangzi I get this error! 

 ~/workspace/0chain/blobber/dev.local │ on dayi/main 
❯ ./cli.sh                                                                                                                                                                                                   

**********************************************
  Welcome to blobber/validator development CLI
**********************************************

Hostname: 192.168.0.108

Please select which blobber/validator you will work on:
1) 1
2) 2
3) 3
4) clean all
5) install debugers on .vscode/launch.json
#? q
#? 1

**********************************************
            Blobber/Validator 1
**********************************************

Please select what you will do:
1) install postgres  3) start validator
2) start blobber     4) clean
#? 1
Installing blobber_postgres in docker...
Initializing database
docker: Error response from daemon: Cannot link to a non running container: /blobber_postgres AS /blobber_postgres_init/postgres.
blobber_postgres_init

 ~/workspace/0chain/blobber/dev.local │ on dayi/main 
❯ docker ps                                                                                                                                                                                                  
CONTAINER ID   IMAGE     COMMAND   CREATED   STATUS    PORTS     NAMES

 ~/workspace/0chain/blobber/dev.local │ on dayi/main 
❯ docker ps -a                                                                                                                                                                                               
CONTAINER ID   IMAGE         COMMAND                  CREATED              STATUS    PORTS     NAMES
28f55ee1f899   postgres:11   "docker-entrypoint.s…"   About a minute ago   Created             blobber_postgres
NoSkillGuy commented 3 years ago

when I ran again, i got port already in use 

 ~/workspace/0chain/blobber/dev.local │ on dayi/main 
❯ ./cli.sh --help                                                                                                                                                                                            

**********************************************
  Welcome to blobber/validator development CLI
**********************************************

Hostname: 192.168.0.108

Please select which blobber/validator you will work on:
1) 1
2) 2
3) 3
4) clean all
5) install debugers on .vscode/launch.json
#? 1

**********************************************
            Blobber/Validator 1
**********************************************

Please select what you will do:
1) install postgres  3) start validator
2) start blobber     4) clean
#? 1
Installing blobber_postgres in docker...
4dafebe32842b74cb5eabcedf68c4b32235557b69e2f3d28e156e209158005d1
docker: Error response from daemon: Ports are not available: listen tcp 0.0.0.0:5432: bind: address already in use.
Initializing database
docker: Error response from daemon: Cannot link to a non running container: /blobber_postgres AS /blobber_postgres_init/postgres.
blobber_postgres_init

 ~/workspace/0chain/blobber/dev.local │ on dayi/main 
❯ lsof -i tcp:5432
cnlangzi commented 3 years ago

Docker rm -f=“name=blobber*” first. And restart docker engine

cnlangzi commented 3 years ago

please use my branch code too. I have made some change on go for cli.

NoSkillGuy commented 3 years ago

closing this issue for now! as it is working as expected now.