pegasus-io / this-is-gravitee

Gravitee On earth
0 stars 0 forks source link

New Containers, New Error, bad me #nolatesttag #1

Open Jean-Baptiste-Lasselle opened 4 years ago

Jean-Baptiste-Lasselle commented 4 years ago

The Issue, and its Context

How to reproduce the issue

You 2 possibilities, to reproduce the issue :

export DESIRED_VERSION=0.0.1
git clone git@github.com:pegasus-io/this-is-gravitee.git gravitee-on-earth
cd gravitee-on-earth/
git checkout ${DESIRED_VERSION}
./operations.sh
export DESIRED_VERSION=TEST2_REPODUCTION_ISSUE_1
git clone git@github.com:pegasus-io/this-is-gravitee.git gravitee-earthling-issues
cd gravitee-earthling-issues/
git checkout ${DESIRED_VERSION}
./test-issue-1.sh

The Context, and the Issue stdout

The Launch Gravitee.io API Platform  Pulumi recipe is now generated...
Pulling mongodb         ... done
Pulling elasticsearch   ... done
Pulling apim_gateway    ... done
Pulling apim_management ... done
Pulling apim_portal     ... done
Pulling am_gateway      ... done
Pulling am_management   ... done
Pulling am_webui        ... done
Pulling traefik         ... done
Creating network "storage" with the default driver
Creating network "frontend" with the default driver
Creating this-is-gravitee_mongodb_1       ... done
Creating this-is-gravitee_elasticsearch_1 ... done
Creating this-is-gravitee_apim_management_1 ... error
Creating this-is-gravitee_apim_gateway_1    ... 
Creating this-is-gravitee_am_gateway_1      ... done
Creating this-is-gravitee_am_management_1   ... done
Creating this-is-gravitee_am_webui_1        ... 

Creating this-is-gravitee_apim_gateway_1    ... error
sed \"rootfs_linux.go:58: mounting \\\"/home/jbl/this-is-gravitee/gravitee/cacerts\\\" to rootfs \\\"/var/lib/docker/overlay2/3c638c91112e4f3cde752c3fa3906bd7b4268de8c210c54d24ccab3f3519b4c9/merged\\\" at \\\"/var/lib/docker/overlay2/3c638c91112e4f3cde752c3fa3906bd7b4268de8c210c54d24ccab3f3519b4c9/merged/etc/ssl/certs/java/cacerts\\\" caused \\\"not a directory\\\"\"": unknown: Are you trying to mount a directory onto a file (or vice-versa)? Check if the specified host path exists and is the expected type

ERROR: for this-is-gravitee_apim_gateway_1  Cannot start service apim_gateway: OCI runtime create failed: container_linux.go:349: starting container process caused "process_linux.go:449: container init caused \"rootfs_linux.go:58: mounting \\\"/home/jbl/this-is-gravitee/gravitee/cacerts\\\" to rootfs \\\"/var/lib/docker/overlay2/433282ae24e7cd9667210a8fe8b741dfd1339dbbf84b4d68f5fb37386f7ed1e6/merged\\\" at \\\"/var/lib/docker/overlay2/433282ae24e7cd9667210a8fe8b741dfd1339dbbf84b4d68f5fb37386f7ed1e6/merged/etc/ssl/certs/java/cacerts\\\" caused \\\"not a directory\\\"\"": unknown: Are you trying to mount a directory onto a fileCreating this-is-gravitee_am_webui_1        ... done

ERROR: for apim_management  Cannot start service apim_management: OCI runtime create failed: container_linux.go:349: starting container process caused "process_linux.go:449: container init caused \"rootfs_linux.go:58: mounting \\\"/home/jbl/this-is-gravitee/gravitee/cacerts\\\" to rootfs \\\"/var/lib/docker/overlay2/3c638c91112e4f3cde752c3fa3906bd7b4268de8c210c54d24ccab3f3519b4c9/merged\\\" at \\\"/var/lib/docker/overlay2/3c638c91112e4f3cde752c3fa3906bd7b4268de8c210c54d24ccab3f3519b4c9/merged/etc/ssl/certs/java/cacerts\\\" caused \\\"not a directory\\\"\"": unknown: Are you trying to mount a directory onto a file (or vice-versa)? Check if the specified host path exists and is the expected type

ERROR: for apim_gateway  Cannot start service apim_gateway: OCI runtime create failed: container_linux.go:349: starting container process caused "process_linux.go:449: container init caused \"rootfs_linux.go:58: mounting \\\"/home/jbl/this-is-gravitee/gravitee/cacerts\\\" to rootfs \\\"/var/lib/docker/overlay2/433282ae24e7cd9667210a8fe8b741dfd1339dbbf84b4d68f5fb37386f7ed1e6/merged\\\" at \\\"/var/lib/docker/overlay2/433282ae24e7cd9667210a8fe8b741dfd1339dbbf84b4d68f5fb37386f7ed1e6/merged/etc/ssl/certs/java/cacerts\\\" caused \\\"not a directory\\\"\"": unknown: Are you trying to mount a directory onto a file (or vice-versa)? Check if the specified host path exists and is the expected type
ERROR: Encountered errors while bringing up the project.
./operations.sh: line 104: docker-composer: command not found
jbl@pc-alienware-jbl:~/this-is-gravitee$ 
jbl@pc-alienware-jbl:~/gravitee-earthling-issues$ docker inspect graviteeio/gateway:latest| jq .[].Id
"sha256:8a70c1761deb912a20f3f86ba0cd8d2f4c6997775c35085b1f8099c24c35913f"
jbl@pc-alienware-jbl:~/gravitee-earthling-issues$ docker inspect graviteeio/am-management-api| jq .[].Id
"sha256:31267a14c09c3ec40f0dd2c13d51a2fb8ffe3bc2e134a6da81807556c1e42fbf"
jbl@pc-alienware-jbl:~/gravitee-earthling-issues$ docker images
REPOSITORY                                      TAG                 IMAGE ID            CREATED             SIZE
graviteeio/am-management-api                    latest              31267a14c09c        3 days ago          226MB
graviteeio/am-gateway                           latest              98a9cdb07109        3 days ago          215MB
graviteeio/am-management-ui                     latest              ff7f593358a3        3 days ago          46.1MB
graviteeio/management-ui                        latest              5e43ed1fad02        3 weeks ago         50.6MB
graviteeio/management-api                       latest              707364db8861        3 weeks ago         349MB
graviteeio/gateway                              latest              8a70c1761deb        3 weeks ago         325MB
traefik                                         v1.7                100389fa48aa        2 months ago        76.4MB
mongo                                           3.4                 f76f959b2a49        4 months ago        431MB
docker.elastic.co/elasticsearch/elasticsearch   6.3.1               fa7212eab151        23 months ago       783MB

My Diagnosis up til now

the Gravitee APIM Gateway

docker run -it --name testjbl_apim_gw graviteeio/gateway:latest ls -allh /etc/ssl/certs/java/cacerts docker run -it --name testjbl_apim_gw graviteeio/gateway:latest stat /etc/ssl/certs/java/cacerts

  * results : 
```bash
jbl@pc-alienware-jbl:~/gravitee-earthling-issues$ docker run -it --name testjbl_apim_api graviteeio/management-api:latest ls -allh /etc/ssl/certs/java/cacerts
-r--r--r--    1 root     root      171.2K Oct 24  2018 /etc/ssl/certs/java/cacerts
jbl@pc-alienware-jbl:~/gravitee-earthling-issues$ docker run -it --name testjbl_apim_api graviteeio/management-api:latest stat /etc/ssl/certs/java/cacerts
  File: /etc/ssl/certs/java/cacerts
  Size: 175337      Blocks: 344        IO Block: 4096   regular file
Device: 851h/2129d  Inode: 931034      Links: 1
Access: (0444/-r--r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2018-10-24 22:32:16.000000000
Modify: 2018-10-24 22:32:16.000000000
Change: 2020-05-29 15:06:41.000000000

jbl@pc-alienware-jbl:~/gravitee-earthling-issues$ docker run -it --name testjbl_apim_gw graviteeio/gateway:latest ls -allh /etc/ssl/certs/java/cacerts
-r--r--r--    1 root     root      171.2K Oct 24  2018 /etc/ssl/certs/java/cacerts
jbl@pc-alienware-jbl:~/gravitee-earthling-issues$ docker run -it --name testjbl_apim_gw graviteeio/gateway:latest stat /etc/ssl/certs/java/cacerts
  File: /etc/ssl/certs/java/cacerts
  Size: 175337      Blocks: 344        IO Block: 4096   regular file
Device: 851h/2129d  Inode: 931034      Links: 1
Access: (0444/-r--r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2018-10-24 22:32:16.000000000
Modify: 2018-10-24 22:32:16.000000000
Change: 2020-05-29 15:06:41.000000000

Inside the apim_management service :

Inside the apim_gateway service :

My first attempt to fix the issue on my own

Ok, so now, I know, that PathOutside is a directory, while /etc/ssl/certs/java/cacerts is a file. That is a first, and easy issue to solve. All I have to do, is to replace, in docker-compose.yml, "$PWD/gravitee/cacerts:/etc/ssl/certs/java/cacerts", by "$PWD/gravitee/:/etc/ssl/certs/java/".

# reproduce the issue

export DESIRED_VERSION=TEST2_REPODUCTION_ISSUE_1
git clone git@github.com:pegasus-io/this-is-gravitee.git gravitee-earthling-issues
cd gravitee-earthling-issues/
git checkout ${DESIRED_VERSION}
./test-issue-1.sh

# apply the fix

sed -i "s#\$PWD/gravitee/cacerts:/etc/ssl/certs/java/cacerts#$PWD/gravitee:/etc/ssl/certs/java#g" docker-compose.yml

# re-launch the docker-compose

docker-compose down --rmi all && docker system prune -f --all && docker-compose up -d 
jbl@pc-alienware-jbl:~/gravitee-earthling-issues$ docker-compose logs  apim_management|grep -B10 'Caused by:'|grep -B10 'Exception'|head -n 15
apim_management_1  | 19:30:16.817 [graviteeio-node] INFO  i.g.p.c.internal.PluginRegistryImpl - List of available alert: 
apim_management_1  | 19:30:16.817 [graviteeio-node] INFO  i.g.p.c.internal.PluginRegistryImpl - List of available service_discovery: 
apim_management_1  | 19:30:16.817 [graviteeio-node] INFO  i.g.p.c.internal.PluginRegistryImpl -     > consul-service-discovery [1.1.1] has been loaded
apim_management_1  | 19:30:16.817 [graviteeio-node] INFO  i.g.p.c.internal.PluginRegistryImpl - List of available protocol: 
apim_management_1  | 19:30:16.817 [graviteeio-node] INFO  i.g.p.c.internal.PluginEventListener - All plugins have been loaded. Installing...
apim_management_1  | 19:30:16.817 [graviteeio-node] INFO  i.g.p.c.internal.PluginEventListener - Installing REPOSITORY plugins...
apim_management_1  | 19:30:16.822 [graviteeio-node] INFO  i.g.m.r.p.RepositoryPluginHandler - Register a new repository: repository-elasticsearch [io.gravitee.repository.elasticsearch.ElasticsearchRepository]
apim_management_1  | 19:30:16.823 [graviteeio-node] INFO  i.g.m.r.p.RepositoryPluginHandler - Repository [ANALYTICS] loaded by elasticsearch
apim_management_1  | 19:30:17.470 [vert.x-eventloop-thread-3] ERROR i.g.e.client.http.HttpClient - Unable to get a connection to Elasticsearch
apim_management_1  | io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: elasticsearch/192.168.16.5:9200
apim_management_1  | Caused by: java.net.ConnectException: Connection refused
--
apim_management_1  |    at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:702)
apim_management_1  |    at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
apim_management_1  |    at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
jbl@pc-alienware-jbl:~/gravitee-earthling-issues$ 
sudo chmod a+rw -R ./data && docker-compose restart elasticsearch && docker-compose logs -f elasticsearch
# -- just restarting and logs of Gravitee APIM API

Enjoy

So once again, Thank you Gravitee Team, for your excellent, rigourous work.

Jean-Baptiste-Lasselle commented 4 years ago

Pour reproduire à l'identitque, l'issue

export DESIRED_VERSION=TEST2_REPODUCTION_ISSUE_1

git clone git@github.com:pegasus-io/this-is-gravitee.git gravitee-earthling-issues

cd gravitee-earthling-issues/

git checkout ${DESIRED_VERSION}

./test-issue-1.sh
Jean-Baptiste-Lasselle commented 4 years ago

TODO :

Jean-Baptiste-Lasselle commented 4 years ago

Note to the reader : I also ran the sudo sysctl -w vm.max_map_count=262144 on the docker host, before running all those tests.