search

open-horizon / devops

Devops processes to build and deploy horizon components
Apache License 2.0
10 stars 42 forks source link

Cannot start mgmt hub due to failing css-api container #174

Open scwhaley opened 3 months ago

scwhaley commented 3 months ago

When running the mgmt hub via deploy-mgmt-hub.sh, it fails with error code 5 due to a failing css-api container.

Looks like it is erroring because css-api fails to connect to mongo due to invalid connection string format.

Command:

export EXCHANGE_ROOT_PW=Horizon-Rul3s EXCHANGE_IMAGE_TAG=testing EXCHANGE_DATABASE=exchange; \
export EXCHANGE_HUB_ADMIN_PW=hubadminpw EXCHANGE_SYSTEM_ADMIN_PW=ibmadminpw; \
export AGBOT_ID=agbot AGBOT_TOKEN=Abcdefghijklmno1 AGBOT_IMAGE_TAG=e2edev; \
export CSS_IMAGE_TAG=e2edev CSS_INTERNAL_PORT=8080; \
export MONGO_IMAGE_TAG=4.0.6; \
export POSTGRES_IMAGE_TAG=13 POSTGRES_USER=admin; \
export VAULT_IMAGE_TAG=testing VAULT_PORT=8200 VAULT_DEV_LISTEN_ADDRESS=0.0.0.0:8200 VAULT_ROOT_TOKEN=vault_dev_root_token_id; \
export ANAX_SOURCE=~/repos/forks/anax ANAX_LOG_LEVEL=5; \
export HZN_TRANSPORT=http; 

mgmt-hub/deploy-mgmt-hub.sh

Script Output:


------- Checking input and initializing...
Management hub services will listen on http://127.0.0.1
Updating apt-get package index...
Installing prerequisites, this could take a minute...
Certificate /etc/horizon/keys/horizonMgmtHubFDO.crt already exists, so not receating it
------- Downloading template files...
Substituting environment variables into template files...
------- Downloading/starting Horizon management hub services...
Downloading management hub docker images...
Trying to pull openhorizon/amd64_agbot:e2edev ...
Pulling openhorizon/amd64_exchange-api:testing ...
Trying to pull openhorizon/amd64_cloud-sync-service:e2edev ...
Trying to pull postgres:13 ...
Trying to pull mongo:4.0.6 ...
Pulling openhorizon/fdo-owner-services:testing ...
Pulling openhorizon/amd64_vault:testing ...
Starting management hub containers...
[+] Running 15/15
 ✔ Network hzn_horizonnet                Created                                                                           0.1s 
 ✔ Volume "hzn_vaultvol"                 Created                                                                           0.0s 
 ✔ Volume "hzn_agbotmsgkeyvol"           Created                                                                           0.0s 
 ✔ Volume "hzn_postgres"                 Created                                                                           0.0s 
 ✔ Volume "hzn_mongovol"                 Created                                                                           0.0s 
 ✔ Volume "hzn_fdo-ocs-db"               Created                                                                           0.0s 
 ✔ Volume "hzn_postgresvolfdo"           Created                                                                           0.0s 
 ✔ Container postgres-fdo-owner-service  Started                                                                           0.4s 
 ✔ Container postgres                    Started                                                                           0.3s 
 ✔ Container mongo                       Started                                                                           0.4s 
 ✔ Container exchange-api                Started                                                                           0.1s 
 ✔ Container css-api                     Started                                                                           0.1s 
 ✔ Container vault                       Started                                                                           0.1s 
 ✔ Container fdo-owner-services          Started                                                                           0.1s 
 ✔ Container agbot                       Started                                                                           0.1s 
Waiting for the exchange....
------- Creating the user org, and the admin user in both orgs...
Creating exchange hub admin user, and the admin user and agbot in the system org...
------- Creating a Vault instance and performing all setup and configuration operations ...
Checking Vault service status, initialization, and seal...
A Vault instance has not been initialized. Initializing...
Vault instance is sealed. Unsealing...
Creating KV ver.2 secrets engine openhorizon...
Registering auth plugin openhorizon-exchange to Vault instance...
Generating SHA256 hash of openhorizon-exchange plugin...
Enabling auth method openhorizon-exchange for secrets engine openhorizon...
Configuring auth method openhorizon-exchange for use with the Exchange...
Success! Data written to: auth/openhorizon/config
------- Creating an agbot in the exchange...
[+] Restarting 1/1
 ✔ Container agbot  Started                                                                                               10.8s 
Creating exchange user org and admin user...
------- Downloading/installing/configuring Horizon agent and CLI...
Downloading the Horizon agent and CLI packages...
Installing the Horizon agent and CLI packages...
Configuring the Horizon agent and CLI...
Publishing /tmp/horizon-all-in-1/agent-install.cfg in CSS as public object agent-install.cfg in the IBM org...
Digital sign with SHA256 will be performed for data integrity. It will delay the MMS object publish.
Start hashing the file...
Data hash is generated. Start digital signing with the data hash...
Digital sign finished.
Error: Encountered HTTP error: Put "http://127.0.0.1:9443/api/v1/objects/IBM/agent_files/agent-install.cfg": dial tcp 127.0.0.1:9443: connect: connection refused calling Model Management Service REST API PUT http://127.0.0.1:9443/api/v1/objects/IBM/agent_files/agent-install.cfg. HTTP status: .
Error: exit code 5 from: publishing /tmp/horizon-all-in-1/agent-install.cfg in CSS as a public object

Docker containers:


CONTAINER ID   IMAGE                                         COMMAND                  CREATED         STATUS                          PORTS                                                NAMES
3e109afdec68   openhorizon/amd64_agbot:e2edev                "/bin/sh -c /usr/hor…"   2 minutes ago   Up 2 minutes (healthy)          127.0.0.1:3110->8080/tcp, 127.0.0.1:3111->8083/tcp   agbot
e30806a9e099   openhorizon/fdo-owner-services:testing        "/bin/sh -c $WORKDIR…"   2 minutes ago   Up 2 minutes (healthy)          0.0.0.0:8042->8042/tcp, 0.0.0.0:9008->9008/tcp       fdo-owner-services
91b08433718e   openhorizon/amd64_vault:testing               "entrypoint.sh server"   2 minutes ago   Up 2 minutes (healthy)          127.0.0.1:8200->8200/tcp                             vault
57f192b2cacd   openhorizon/amd64_cloud-sync-service:e2edev   "/usr/edge-sync-serv…"   2 minutes ago   Restarting (2) 48 seconds ago                                                        css-api
1fb73692c9d2   openhorizon/amd64_exchange-api:testing        "/bin/sh -c '/usr/bi…"   2 minutes ago   Up 2 minutes (healthy)          8083/tcp, 127.0.0.1:3090->8080/tcp                   exchange-api
10bbb39c6e89   postgres:13                                   "docker-entrypoint.s…"   2 minutes ago   Up 2 minutes (healthy)          0.0.0.0:5433->5432/tcp                               postgres-fdo-owner-service
d87d034c9bd4   mongo:4.0.6                                   "docker-entrypoint.s…"   2 minutes ago   Up 2 minutes (healthy)          27017/tcp                                            mongo
09a9538b52c1   postgres:13                                   "docker-entrypoint.s…"   2 minutes ago   Up 2 minutes (healthy)          0.0.0.0:5432->5432/tcp                               postgres

css-api container logs:


* CSS: 2024/06/18 21:10:22 Loaded configuration:
  NodeType  CSS
  DestinationType  Cloud
  DestinationID  Cloud
  OrgID  
  ListeningType  unsecure
  ListeningAddress  
  SecureListeningPort  8080
  UnsecureListeningPort  8080
  ServerCertificate  <...>
  ServerKey  <...>
  CSSOnWIoTP  false
  UsingEdgeConnector  false
  LeadershipTimeout  45
  AuthenticationHandler  dummy
  ObjectQueueBufferSize  1000
  VerifyQueueBufferSize  500
  CommunicationProtocol  http
  MQTTClientID  
  MQTTUserName  
  MQTTPassword  
  MQTTUseSSL  true
  MQTTCACertificate  <...>
  MQTTSSLCert  
  MQTTSSLKey  
  MQTTAllowInvalidCertificates  false
  MQTTBrokerConnectTimeout  300
  MQTTParallelMode  
  PersistenceRootPath  /var/edge-sync-service/persist/
  BrokerAddress  
  BrokerPort  0
  HTTPPollingInterval  10
  HTTPCSSHost  
  HTTPCSSPort  0
  HTTPCSSUseSSL  false
  HTTPCSSCACertificate  
  HTTPESSClientTimeout  120
  HTTPESSObjClientTimeout  600
  HTTPCSSObjDownloadConcurrencyMultiplier  1
  HTTPServerReadHeaderTimeout  60
  LogLevel  INFO
  LogRootPath  /var/edge-sync-service/log
  LogTraceDestination  stdout
  LogFileName  sync-service
  TraceLevel  INFO
  TraceRootPath  /var/edge-sync-service/trace
  TraceFileName  sync-service
  LogTraceFileSizeKB  20000
  MaxCompressedlLogTraceFilesNumber  50
  LogTraceMaintenanceInterval  60
  ResendInterval  10
  ESSCallSPIRetryInterval  2
  ESSSPIMaxRetry  1000
  ESSPingInterval  1
  RemoveESSRegistrationTime  30
  EnableDataChunk  true
  MaxDataChunkSize  5242880
  MaxInflightChunks  1
  MongoAddressCsv  mongo:27017
  MongoAuthDbName  admin
  MongoDbName  d_edge
  MongoUsername  
  MongoPassword  
  MongoUseSSL  false
  MongoCACertificate  
  MongoAllowInvalidCertificates  false
  MongoSessionCacheSize  1
  MongoSleepTimeBetweenRetry  4000
  DatabaseConnectTimeout  300
  StorageMaintenanceInterval  30
  ObjectActivationInterval  30
  StorageProvider  mongo
  ESSConsumedObjectsKept  1000
  MessagingGroupCacheExpiration  60
  ShutdownQuiesceTime  60
  ObjectsDataPath  
CSS: 2024/06/18 21:10:22 INFO: Horizon Authenticator starting with exchange authenticated identity
* CSS: 2024/06/18 21:10:22 INFO: CConnecting to mongo...
* CSS: 2024/06/18 21:10:22 INFO: connect to mongo...
* CSS: 2024/06/18 21:10:22 ERROR: Error connecting to mongo. Error was: error parsing uri: scheme must be "mongodb" or "mongodb+srv"
* CSS: 2024/06/18 21:10:22 ERROR: Failed to dial mgo. Error: error parsing uri: scheme must be "mongodb" or "mongodb+srv"
* CSS: 2024/06/18 21:10:22 ERROR: Retrying to connect to mongo
* CSS: 2024/06/18 21:10:22 INFO: connect to mongo...
* CSS: 2024/06/18 21:10:22 ERROR: Error connecting to mongo. Error was: error parsing uri: scheme must be "mongodb" or "mongodb+srv"
* CSS: 2024/06/18 21:10:22 INFO: connect to mongo...
* CSS: 2024/06/18 21:10:22 ERROR: Error connecting to mongo. Error was: error parsing uri: scheme must be "mongodb" or "mongodb+srv"
* CSS: 2024/06/18 21:10:22 INFO: connect to mongo...
* CSS: 2024/06/18 21:10:22 ERROR: Error connecting to mongo. Error was: error parsing uri: scheme must be "mongodb" or "mongodb+srv"
* CSS: 2024/06/18 21:10:22 INFO: connect to mongo...
* CSS: 2024/06/18 21:10:22 ERROR: Error connecting to mongo. Error was: error parsing uri: scheme must be "mongodb" or "mongodb+srv"
* CSS: 2024/06/18 21:10:22 INFO: connect to mongo...
* CSS: 2024/06/18 21:10:22 ERROR: Error connecting to mongo. Error was: error parsing uri: scheme must be "mongodb" or "mongodb+srv"
* CSS: 2024/06/18 21:10:22 INFO: connect to mongo...
* CSS: 2024/06/18 21:10:22 ERROR: Error connecting to mongo. Error was: error parsing uri: scheme must be "mongodb" or "mongodb+srv"
* CSS: 2024/06/18 21:10:22 INFO: connect to mongo...
* CSS: 2024/06/18 21:10:22 ERROR: Error connecting to mongo. Error was: error parsing uri: scheme must be "mongodb" or "mongodb+srv"
* CSS: 2024/06/18 21:10:22 INFO: connect to mongo...
* CSS: 2024/06/18 21:10:22 ERROR: Error connecting to mongo. Error was: error parsing uri: scheme must be "mongodb" or "mongodb+srv"
* CSS: 2024/06/18 21:10:22 INFO: connect to mongo...
* CSS: 2024/06/18 21:10:22 ERROR: Error connecting to mongo. Error was: error parsing uri: scheme must be "mongodb" or "mongodb+srv"
* CSS: 2024/06/18 21:10:22 INFO: connect to mongo...
* CSS: 2024/06/18 21:10:22 ERROR: Error connecting to mongo. Error was: error parsing uri: scheme must be "mongodb" or "mongodb+srv"
* CSS: 2024/06/18 21:10:22 INFO: connect to mongo...
* CSS: 2024/06/18 21:10:22 ERROR: Error connecting to mongo. Error was: error parsing uri: scheme must be "mongodb" or "mongodb+srv"
* CSS: 2024/06/18 21:10:22 INFO: connect to mongo...
* CSS: 2024/06/18 21:10:22 ERROR: Error connecting to mongo. Error was: error parsing uri: scheme must be "mongodb" or "mongodb+srv"
* CSS: 2024/06/18 21:10:22 INFO: connect to mongo...
* CSS: 2024/06/18 21:10:22 ERROR: Error connecting to mongo. Error was: error parsing uri: scheme must be "mongodb" or "mongodb+srv"
* CSS: 2024/06/18 21:10:22 INFO: connect to mongo...
* CSS: 2024/06/18 21:10:22 ERROR: Error connecting to mongo. Error was: error parsing uri: scheme must be "mongodb" or "mongodb+srv"
* CSS: 2024/06/18 21:10:22 INFO: connect to mongo...
* CSS: 2024/06/18 21:10:22 ERROR: Error connecting to mongo. Error was: error parsing uri: scheme must be "mongodb" or "mongodb+srv"
* CSS: 2024/06/18 21:10:22 INFO: connect to mongo...
* CSS: 2024/06/18 21:10:22 ERROR: Error connecting to mongo. Error was: error parsing uri: scheme must be "mongodb" or "mongodb+srv"
* CSS: 2024/06/18 21:10:22 INFO: connect to mongo...
* CSS: 2024/06/18 21:10:22 ERROR: Error connecting to mongo. Error was: error parsing uri: scheme must be "mongodb" or "mongodb+srv"
* CSS: 2024/06/18 21:10:22 INFO: connect to mongo...
* CSS: 2024/06/18 21:10:22 ERROR: Error connecting to mongo. Error was: error parsing uri: scheme must be "mongodb" or "mongodb+srv"
* CSS: 2024/06/18 21:10:22 INFO: connect to mongo...
* CSS: 2024/06/18 21:10:22 ERROR: Error connecting to mongo. Error was: error parsing uri: scheme must be "mongodb" or "mongodb+srv"
* CSS: 2024/06/18 21:10:22 INFO: connect to mongo...
* CSS: 2024/06/18 21:10:22 ERROR: Error connecting to mongo. Error was: error parsing uri: scheme must be "mongodb" or "mongodb+srv"
* CSS: 2024/06/18 21:10:22 INFO: connect to mongo...
* CSS: 2024/06/18 21:10:22 ERROR: Error connecting to mongo. Error was: error parsing uri: scheme must be "mongodb" or "mongodb+srv"
* CSS: 2024/06/18 21:10:22 INFO: connect to mongo...
* CSS: 2024/06/18 21:10:22 ERROR: Error connecting to mongo. Error was: error parsing uri: scheme must be "mongodb" or "mongodb+srv"
* CSS: 2024/06/18 21:10:22 INFO: connect to mongo...
* CSS: 2024/06/18 21:10:22 ERROR: Error connecting to mongo. Error was: error parsing uri: scheme must be "mongodb" or "mongodb+srv"
* CSS: 2024/06/18 21:10:22 INFO: connect to mongo...
* CSS: 2024/06/18 21:10:22 ERROR: Error connecting to mongo. Error was: error parsing uri: scheme must be "mongodb" or "mongodb+srv"
* CSS: 2024/06/18 21:10:22 INFO: connect to mongo...
* CSS: 2024/06/18 21:10:22 ERROR: Error connecting to mongo. Error was: error parsing uri: scheme must be "mongodb" or "mongodb+srv"
* CSS: 2024/06/18 21:10:22 INFO: connect to mongo...
* CSS: 2024/06/18 21:10:22 ERROR: Error connecting to mongo. Error was: error parsing uri: scheme must be "mongodb" or "mongodb+srv"
* CSS: 2024/06/18 21:10:22 INFO: connect to mongo...
* CSS: 2024/06/18 21:10:22 ERROR: Error connecting to mongo. Error was: error parsing uri: scheme must be "mongodb" or "mongodb+srv"
* CSS: 2024/06/18 21:10:22 INFO: connect to mongo...
* CSS: 2024/06/18 21:10:22 ERROR: Error connecting to mongo. Error was: error parsing uri: scheme must be "mongodb" or "mongodb+srv"
* CSS: 2024/06/18 21:10:22 INFO: connect to mongo...
* CSS: 2024/06/18 21:10:22 ERROR: Error connecting to mongo. Error was: error parsing uri: scheme must be "mongodb" or "mongodb+srv"
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x38 pc=0x1428f0a]

goroutine 1 [running]:
go.mongodb.org/mongo-driver/mongo.(*Client).Ping(0x26faec0?, {0x1b453c0?, 0xc00021e070?}, 0x62?)
        /tmp/anax-gopath/pkg/mod/go.mongodb.org/mongo-driver@v1.15.0/mongo/client.go:361 +0x4a
github.com/open-horizon/edge-sync-service/core/storage.(*MongoStorage).Init(0xc0000aa000)
        /tmp/anax-gopath/pkg/mod/github.com/open-horizon/edge-sync-service@v1.11.0/core/storage/mongoStorage.go:204 +0x725
github.com/open-horizon/edge-sync-service/core/base.Start({0x0, 0x0}, 0xe5?)
        /tmp/anax-gopath/pkg/mod/github.com/open-horizon/edge-sync-service@v1.11.0/core/base/base.go:115 +0x673
github.com/open-horizon/edge-sync-service/core/base.StandaloneSyncService({0x1b38890?, 0xc000447f98})
        /tmp/anax-gopath/pkg/mod/github.com/open-horizon/edge-sync-service@v1.11.0/core/base/standalone.go:118 +0x5bc
main.main()
        /tmp/anax-gopath/src/github.com/open-horizon/anax/css/cmd/cloud-sync-service/main.go:28 +0x2e
scwhaley commented 3 months ago

I see that PR #171 was approved and merged into master branch with the fix, but it doesn't actually show up in the master branch... Confused why that is the case @LiilyZhang @bencourliss

My PR #175 would be redundant if the changes from #171 were visible.

joewxboy commented 2 months ago

The LF Edge lab is currently attempting to deploy the latest branch to VM3 and encountering this same error.