lets-fiware / FIWARE-Big-Bang

The FIWARE Big Bang is a turnkey solution for setting up a FIWARE instance in the cloud.
https://fi-bb.letsfiware.jp/
MIT License
21 stars 7 forks source link

Sanity check for Cygnus returns 502 Bad Gateway #359

Open dkastl opened 4 months ago

dkastl commented 4 months ago

Checklist Put an x in the boxes that apply

Not sure I inspected all errors and logs, but unfortunately I already deleted my previous setup.

Describe the bug

When I run the sanity check for Cygnus I get a "502 Bad Gateway" response like

sudo ngsi version --host cygnus.mydomain.com --pretty
cbVersion002 error 502 Bad Gateway <html>
<head><title>502 Bad Gateway</title></head>
<body>
<center><h1>502 Bad Gateway</h1></center>
<hr><center>nginx</center>
</body>
</html>

This happens, when only 1 sink is set to true. When 2 or more sinks are enabled, it seems to be working.

Eventually there is a problem when CYGNUS_MULTIAGENT is not set true, see: https://github.com/lets-fiware/FIWARE-Big-Bang/blob/main/lets-fiware.sh#L2248-L2250

Your system environment Run make collect in a directory where you ran the lets-fiware.sh script and paste the results here.

Date: Tue May 21 15:54:24 UTC 2024
Version: 0.37.0
OS: Ubuntu 2404
Docker: Docker version 26.1.3, build b72abbb
Docker compose: Docker Compose version v2.27.0
uid=0(root) gid=0(root) groups=0(root)
Hash: 53423c9f29f4242ffa8131ec9a722288259b01e3787850763a26cb89e4cebdb2  lets-fiware.sh
git-hash: 
a849b38e7f164b8556e21721e175ce928b06f0a3
b4cf3f3e124b0d64d6aebd0853f30575962d2905
04c575519e1f47aa1491403b1ca927883ab2aa4d
App list: KEYROCK ORION CYGNUS COMET NODE_RED GRAFANA QUANTUMLEAP IOTAGENT_UL IOTAGENT_JSON IOTAGENT_HTTP MOSQUITTO 
Install: completed
WILMA_AUTH_ENABLED: true
Docker containers: 
make[1]: Entering directory '/srv/fiware-big-bang'
sudo ./setup/ps.sh
WARN[0000] /srv/fiware-big-bang/docker-compose.yml: `version` is obsolete 
NAME                                    IMAGE                                   COMMAND                  SERVICE               CREATED          STATUS                      PORTS
fiware-big-bang-comet-1                 telefonicaiot/fiware-sth-comet:2.10.0   "docker/entrypoint.s…"   comet                 12 minutes ago   Up 12 minutes (healthy)     8666/tcp
fiware-big-bang-comet-wilma-1           letsfiware/fiware-pep-proxy:8.4.0       "docker-entrypoint.s…"   comet-wilma           12 minutes ago   Up 12 minutes (unhealthy)   1027/tcp
fiware-big-bang-crate-1                 crate:4.6.6                             "/docker-entrypoint.…"   crate                 12 minutes ago   Up 12 minutes               4300/tcp, 127.0.0.1:4200->4200/tcp, 5432/tcp
fiware-big-bang-cygnus-1                telefonicaiot/fiware-cygnus:3.2.0       "/cygnus-entrypoint.…"   cygnus                12 minutes ago   Up 12 minutes (healthy)     5050/tcp, 5080/tcp
fiware-big-bang-cygnus-wilma-1          letsfiware/fiware-pep-proxy:8.4.0       "docker-entrypoint.s…"   cygnus-wilma          12 minutes ago   Up 12 minutes (unhealthy)   1027/tcp
fiware-big-bang-grafana-1               grafana/grafana:6.1.6                   "/run.sh"                grafana               12 minutes ago   Up 12 minutes               3000/tcp
fiware-big-bang-iotagent-http-wilma-1   letsfiware/fiware-pep-proxy:8.4.0       "docker-entrypoint.s…"   iotagent-http-wilma   12 minutes ago   Up 12 minutes (unhealthy)   1027/tcp
fiware-big-bang-iotagent-json-1         telefonicaiot/iotagent-json:2.4.2       "docker-entrypoint.s…"   iotagent-json         12 minutes ago   Up 12 minutes (healthy)     4041/tcp, 7896/tcp
fiware-big-bang-iotagent-json-wilma-1   letsfiware/fiware-pep-proxy:8.4.0       "docker-entrypoint.s…"   iotagent-json-wilma   12 minutes ago   Up 12 minutes (unhealthy)   1027/tcp
fiware-big-bang-iotagent-ul-1           telefonicaiot/iotagent-ul:2.4.2         "docker-entrypoint.s…"   iotagent-ul           12 minutes ago   Up 12 minutes (healthy)     4061/tcp, 7896/tcp
fiware-big-bang-iotagent-ul-wilma-1     letsfiware/fiware-pep-proxy:8.4.0       "docker-entrypoint.s…"   iotagent-ul-wilma     12 minutes ago   Up 12 minutes (unhealthy)   1027/tcp
fiware-big-bang-keyrock-1               letsfiware/fiware-idm:8.4.0             "docker-entrypoint.s…"   keyrock               12 minutes ago   Up 12 minutes (healthy)     3000/tcp
fiware-big-bang-mongo-1                 mongo:4.4                               "docker-entrypoint.s…"   mongo                 12 minutes ago   Up 12 minutes               27017/tcp
fiware-big-bang-mosquitto-1             eclipse-mosquitto:1.6                   "/docker-entrypoint.…"   mosquitto             12 minutes ago   Up 12 minutes               1883/tcp
fiware-big-bang-mysql-1                 mysql:8.1                               "docker-entrypoint.s…"   mysql                 12 minutes ago   Up 12 minutes               3306/tcp, 33060/tcp
fiware-big-bang-nginx-1                 nginx:1.25                              "/docker-entrypoint.…"   nginx                 12 minutes ago   Up 12 minutes               0.0.0.0:80->80/tcp, :::80->80/tcp, 0.0.0.0:443->443/tcp, :::443->443/tcp, 0.0.0.0:8883->8883/tcp, :::8883->8883/tcp
fiware-big-bang-node-red001-1           letsfiware/node-red:0.37.0              "./entrypoint.sh"        node-red001           12 minutes ago   Up 12 minutes (healthy)     1880/tcp
fiware-big-bang-node-red002-1           letsfiware/node-red:0.37.0              "./entrypoint.sh"        node-red002           12 minutes ago   Up 12 minutes (healthy)     1880/tcp
fiware-big-bang-node-red003-1           letsfiware/node-red:0.37.0              "./entrypoint.sh"        node-red003           12 minutes ago   Up 12 minutes (healthy)     1880/tcp
fiware-big-bang-orion-1                 telefonicaiot/fiware-orion:3.10.1       "sh -c 'rm /tmp/cont…"   orion                 12 minutes ago   Up 12 minutes (healthy)     1026/tcp
fiware-big-bang-orion-wilma-1           letsfiware/fiware-pep-proxy:8.4.0       "docker-entrypoint.s…"   orion-wilma           12 minutes ago   Up 12 minutes (unhealthy)   1027/tcp
fiware-big-bang-perseo-core-1           telefonicaiot/perseo-core:1.13.0        "/code/perseo_core-e…"   perseo-core           12 minutes ago   Up 12 minutes (healthy)     8080/tcp
fiware-big-bang-perseo-fe-1             telefonicaiot/perseo-fe:1.27.0          "docker-entrypoint.s…"   perseo-fe             12 minutes ago   Up 12 minutes (healthy)     9090/tcp
fiware-big-bang-perseo-fe-wilma-1       letsfiware/fiware-pep-proxy:8.4.0       "docker-entrypoint.s…"   perseo-fe-wilma       12 minutes ago   Up 12 minutes (unhealthy)   1027/tcp
fiware-big-bang-postfix-1               letsfiware/postfix:0.37.0               "docker-entrypoint.sh"   postfix               12 minutes ago   Up 12 minutes               25/tcp
fiware-big-bang-postgres-1              postgres:15                             "docker-entrypoint.s…"   postgres              12 minutes ago   Up 12 minutes               5432/tcp
fiware-big-bang-quantumleap-1           orchestracities/quantumleap:0.8.3       "python app.py"          quantumleap           12 minutes ago   Up 12 minutes (healthy)     8668/tcp
fiware-big-bang-quantumleap-wilma-1     letsfiware/fiware-pep-proxy:8.4.0       "docker-entrypoint.s…"   quantumleap-wilma     12 minutes ago   Up 12 minutes (unhealthy)   1027/tcp
fiware-big-bang-redis-1                 redis:6                                 "docker-entrypoint.s…"   redis                 12 minutes ago   Up 12 minutes               6379/tcp
fiware-big-bang-tokenproxy-1            letsfiware/tokenproxy:0.37.0            "docker-entrypoint.sh"   tokenproxy            12 minutes ago   Up 12 minutes               1029/tcp
make[1]: Leaving directory '/srv/fiware-big-bang'
Keyrock: 
{"keyrock":{"version":"8.4.0","release_date":"2023-06-09T13:19:25.000Z","uptime":"00:11:49.9","doc":"https://keyrock-fiware.github.io/","api":{"version":"v1","link":"https://keyrock.mydomain.com/v1"}}}

To Reproduce Steps to reproduce the behavior:

  1. Configure Cygnus with only one sink set to true. -> 502 Bad Gateway
  2. Configure Cygnus with more than one sink set to true. -> it works

Expected behavior A clear and concise description of what you expected to happen.

As descibed in the documentation it should return something like

{
  "success": "true",
  "version": "3.2.0.9b21eae9095e3250ba8799e61411905d1c756191"
}

Server (please complete the following information):

fisuda commented 4 months ago

I have not been able to reproduce this error. The hash value of the letsfiware.sh file is different from the original one of 0.37.0. Please make a FIWARE instance again using the official release file.

$ cd FIWARE-Big-Bang-0.37.0/

$ cp -a config.sh{,.bak}

$ vi config.sh

$ diff config.sh config.sh.bak
97c97
< CYGNUS=cygnus
---
> CYGNUS=
100c100
< CYGNUS_MONGO=true
---
> CYGNUS_MONGO=

$ ./lets-fiware.sh letsfiware.jp

$ ngsi version --host cygnus.letsfiware.jp
{"success":"true","version":"3.2.0.9b21eae9095e3250ba8799e61411905d1c756191"}

$ make collect
./config/script/collect.sh

Sat May 25 18:16:53 JST 2024
Version: 0.37.0
OS: Ubuntu 2204
Docker: Docker version 26.1.3, build b72abbb
Docker compose: Docker Compose version v2.27.0
uid=1000(letsfiware) gid=1000(letsfiware) groups=1000(letsfiware),4(adm),24(cdrom),27(sudo),30(dip),46(plugdev),122(lpadmin),135(lxd),136(sambashare),999(docker)
Hash: af89b60a9483d6a8ac4a0a709c6638d909cf3773f0b1bb035d1548e994b4ef7a  lets-fiware.sh
App list: KEYROCK ORION CYGNUS
Install: completed
WILMA_AUTH_ENABLED: false
Docker containers:
make[1]: Entering directory '/home/letsfiware/FIWARE-Big-Bang-0.37.0'
sudo ./setup/ps.sh
WARN[0000] /home/letsfiware/FIWARE-Big-Bang-0.37.0/docker-compose.yml: `version` is obsolete
NAME                                IMAGE                               COMMAND                  SERVICE      CREATED       STATUS                   PORTS
fiware-big-bang-0370-cygnus-1       telefonicaiot/fiware-cygnus:3.2.0   "/cygnus-entrypoint.…"   cygnus       2 hours ago   Up 2 hours (healthy)     5050/tcp, 5080/tcp
fiware-big-bang-0370-keyrock-1      letsfiware/fiware-idm:8.4.0         "docker-entrypoint.s…"   keyrock      2 hours ago   Up 2 hours (healthy)     3000/tcp
fiware-big-bang-0370-mongo-1        mongo:4.4                           "docker-entrypoint.s…"   mongo        2 hours ago   Up 2 hours               27017/tcp
fiware-big-bang-0370-mysql-1        mysql:8.1                           "docker-entrypoint.s…"   mysql        2 hours ago   Up 2 hours               3306/tcp, 33060/tcp
fiware-big-bang-0370-nginx-1        nginx:1.25                          "/docker-entrypoint.…"   nginx        2 hours ago   Up 2 hours               0.0.0.0:80->80/tcp, :::80->80/tcp, 0.0.0.0:443->443/tcp, :::443->443/tcp
fiware-big-bang-0370-orion-1        telefonicaiot/fiware-orion:3.10.1   "sh -c 'rm /tmp/cont…"   orion        2 hours ago   Up 2 hours (healthy)     1026/tcp
fiware-big-bang-0370-tokenproxy-1   letsfiware/tokenproxy:0.37.0        "docker-entrypoint.sh"   tokenproxy   2 hours ago   Up 2 hours               1029/tcp
fiware-big-bang-0370-wilma-1        letsfiware/fiware-pep-proxy:8.4.0   "docker-entrypoint.s…"   wilma        2 hours ago   Up 2 hours (unhealthy)   1027/tcp
make[1]: Leaving directory '/home/letsfiware/FIWARE-Big-Bang-0.37.0'
Keyrock:
{"keyrock":{"version":"8.4.0","release_date":"2023-06-09T13:19:25.000Z","uptime":"01:29:53.6","doc":"https://keyrock-fiware.github.io/","api":{"version":"v1","link":"https://keyrock.letsfiware.jp/v1"}}}
dkastl commented 4 months ago

Thank you for taking a look!

I can confirm that it works with only postgres enabled in a setup, where only "orion" and "cygnus" are configured with subdomain.

Before I tried with my original setup, that has many more services enabled, and the failure happend with main, v37.0 and an updated Cygnus as I saw in your pull request.

So the problem must occur with more services enabled, and I can try to enable one by one.

Which would be the best log file to look at? Or, is there eventually a minimum requirement for server hardware, that prevents Cygnus to start correctly? Eventually 4GB RAM is not sufficient?

dkastl commented 4 months ago

Just as a reference, the config.sh I used has the following modifications:

diff --git a/config.sh b/config.sh
index 67b7d60..67e74e4 100755
--- a/config.sh
+++ b/config.sh
@@ -94,12 +94,12 @@ IMAGE_TIMESCALE=timescale/timescaledb-postgis:1.7.5-pg12
 # Cygnus
 #
 # Set a sub-domain name of Cygnus
-CYGNUS=
+CYGNUS=cygnus

 # Use Cygnus sink (true or false) Default: false
 CYGNUS_MONGO=
 CYGNUS_MYSQL=
-CYGNUS_POSTGRES=
+CYGNUS_POSTGRES=true
 CYGNUS_ELASTICSEARCH=

 # Expose port (none, local, all) Default: none
@@ -129,7 +129,7 @@ IMAGE_ELASTICSEARCH_DB=elasticsearch:7.6.2
 # Comet
 #
 # Set a sub-domain name of Comet
-COMET=
+COMET=comet

 # Expose port (none, local, all) Default: none
 COMET_EXPOSE_PORT=
@@ -145,7 +145,7 @@ IMAGE_COMET=telefonicaiot/fiware-sth-comet:2.10.0
 #
 # Perseo
 #
-PERSEO=
+PERSEO=perseo

 # Perseo Max age (default: 6000)
 PERSEO_MAX_AGE=
@@ -189,7 +189,7 @@ IMAGE_DRACO=ging/fiware-draco:2.1.0
 # Quantumleap
 #
 # Set a sub-domain name of Quantumleap
-QUANTUMLEAP=
+QUANTUMLEAP=quantumleap

 # Expose port (none, local, all) Default: none
 QUANTUMLEAP_EXPOSE_PORT=
@@ -236,7 +236,7 @@ IMAGE_MEMCACHED=memcached:1
 # IoT Agent over Mosquitto
 #
 # Set a sub-domain name of Mosquitto
-MOSQUITTO=
+MOSQUITTO=mosquitto

 # Use MQTT (Port 1883). (true or false) Default: false
 MQTT_1883=
@@ -261,7 +261,7 @@ IMAGE_MOSQUITTO=eclipse-mosquitto:1.6
 # IoT Agent over HTTP
 #
 # Set a sub-domain name to use IoT Agent over HTTP.
-IOTAGENT_HTTP=
+IOTAGENT_HTTP=iotagent-http

 # Authorization for IoT Agent over HTTP. (none, basic or bearer) Default: bearer
 IOTA_HTTP_AUTH=
@@ -276,7 +276,7 @@ IOTA_HTTP_BASIC_PASS=
 # IoT Agent for UltraLight 2.0
 #
 # Set a sub-domain name of IoT Agent for UltraLight 2.0
-IOTAGENT_UL=
+IOTAGENT_UL=iotagent-ul

 IOTA_UL_DEFAULT_RESOURCE=/iot/ul
 IOTA_UL_TIMESTAMP=true
@@ -292,7 +292,7 @@ IMAGE_IOTAGENT_UL=telefonicaiot/iotagent-ul:2.4.2
 # IoT Agent for JSON
 #
 # Set a sub-domain name of IoT Agent for JSON
-IOTAGENT_JSON=
+IOTAGENT_JSON=iotagent-json

 IOTA_JSON_DEFAULT_RESOURCE=/iot/json
 IOTA_JSON_TIMESTAMP=true
@@ -308,11 +308,11 @@ IMAGE_IOTAGENT_JSON=telefonicaiot/iotagent-json:2.4.2
 # Node-RED
 #
 # Set a sub-domain name of Node-RED
-NODE_RED=
+NODE_RED=nodered

 # Node-RED multi instance
 # Number of Node-RED instance. default: 1
-NODE_RED_INSTANCE_NUMBER=
+NODE_RED_INSTANCE_NUMBER=3

 # username for Node-RED instance. default: node-red
 NODE_RED_INSTANCE_USERNAME=
@@ -338,7 +338,7 @@ IMAGE_NODE_RED=letsfiware/node-red:0.37.0
 # Grafana
 #
 # Set a sub-domain name of Grafana
-GRAFANA=
+GRAFANA=grafana

 # Logging level for Grafana
 #   https://grafana.com/docs/grafana/latest/administration/configuration/#configure-with-environment-variables
dkastl commented 4 months ago

Eventually Comet makes a difference, beside enabling all database adapters.

The 502 Bad Gateway error happens for Cygnus, when the default config.sh is modified like

diff --git a/config.sh b/config.sh
index 67b7d60..7051045 100755
--- a/config.sh
+++ b/config.sh
@@ -94,12 +94,12 @@ IMAGE_TIMESCALE=timescale/timescaledb-postgis:1.7.5-pg12
 # Cygnus
 #
 # Set a sub-domain name of Cygnus
-CYGNUS=
+CYGNUS=cygnus

 # Use Cygnus sink (true or false) Default: false
 CYGNUS_MONGO=
 CYGNUS_MYSQL=
-CYGNUS_POSTGRES=
+CYGNUS_POSTGRES=true
 CYGNUS_ELASTICSEARCH=

 # Expose port (none, local, all) Default: none
@@ -129,7 +129,7 @@ IMAGE_ELASTICSEARCH_DB=elasticsearch:7.6.2
 # Comet
 #
 # Set a sub-domain name of Comet
-COMET=
+COMET=comet

 # Expose port (none, local, all) Default: none
 COMET_EXPOSE_PORT=

This returns

$ sudo ngsi version --host cygnus.example.com | jq
cbVersion002 error 502 Bad Gateway <html>
<head><title>502 Bad Gateway</title></head>
<body>
<center><h1>502 Bad Gateway</h1></center>
<hr><center>nginx</center>
</body>
</html>

Cygnus works without comet enabled:

diff --git a/config.sh b/config.sh
index 67b7d60..92fd770 100755
--- a/config.sh
+++ b/config.sh
@@ -94,12 +94,12 @@ IMAGE_TIMESCALE=timescale/timescaledb-postgis:1.7.5-pg12
 # Cygnus
 #
 # Set a sub-domain name of Cygnus
-CYGNUS=
+CYGNUS=cygnus

 # Use Cygnus sink (true or false) Default: false
 CYGNUS_MONGO=
 CYGNUS_MYSQL=
-CYGNUS_POSTGRES=
+CYGNUS_POSTGRES=true
 CYGNUS_ELASTICSEARCH=

 # Expose port (none, local, all) Default: none

And Cygnus works with Comet if all databases are set to true.

diff --git a/config.sh b/config.sh
index 67b7d60..5d16b4d 100755
--- a/config.sh
+++ b/config.sh
@@ -94,12 +94,12 @@ IMAGE_TIMESCALE=timescale/timescaledb-postgis:1.7.5-pg12
 # Cygnus
 #
 # Set a sub-domain name of Cygnus
-CYGNUS=
+CYGNUS=cygnus

 # Use Cygnus sink (true or false) Default: false
-CYGNUS_MONGO=
-CYGNUS_MYSQL=
-CYGNUS_POSTGRES=
+CYGNUS_MONGO=true
+CYGNUS_MYSQL=true
+CYGNUS_POSTGRES=true
 CYGNUS_ELASTICSEARCH=

 # Expose port (none, local, all) Default: none
@@ -129,7 +129,7 @@ IMAGE_ELASTICSEARCH_DB=elasticsearch:7.6.2
 # Comet
 #
 # Set a sub-domain name of Comet
-COMET=
+COMET=comet

 # Expose port (none, local, all) Default: none
 COMET_EXPOSE_PORT=

Then the ngsi command return correctly

$ sudo ngsi version --host cygnus.example.com | jq
{
  "success": "true",
  "version": "3.2.0.9b21eae9095e3250ba8799e61411905d1c756191"
}
fisuda commented 4 months ago

This error seems to occur when Cygnus mongodb and postgresql sinks are enabled.

CYGNUS_MONGO=true
CYGNUS_POSTGRES=true
CYGNUS_MONGO= CYGNUS_MYSQL= CYGNUS_POSTGRES= OK/NG
true true NG
true true OK
true true OK
true true true OK

As a workaround, enable the cygnus mysql sink as well.

COMET=comet
CYGNUS=cygnus
CYGNUS_MONGO=
CYGNUS_MYSQL=true
CYGNUS_POSTGRES=true

Or disable STH-Comet.

COMET=
CYGNUS=cygnus
CYGNUS_MONGO=
CYGNUS_POSTGRES=true

Your configuration is bellow:

COMET=comet
CYGNUS=cygnus
CYGNUS_MONGO=
CYGNUS_POSTGRES=true

When enabled STH-Comet and Cygnus, the FIWARE Big Bang sets CYGNUS_MONGO to true automatically and support the formal mode (Cygnus + STH-Comet).

Therefore, your configuration is equivalent to:

COMET=comet
CYGNUS=cygnus
CYGNUS_MONGO=true
CYGNUS_POSTGRES=true
dkastl commented 4 months ago

I see. That makes sense. And thank you for the suggestions!

I usually prefer PostgreSQL, if I have the choice. That's why I initially disabled the other alternatives. But there is no problem to enable all.

That said, I thought it may be helpful to report the problem here.

fisuda commented 4 months ago

I put a question about this issue on the Stack Overflow site as shown: https://stackoverflow.com/questions/78565261/failed-to-connect-to-fiware-cygnus-3-8-0