search

Orange-OpenSource / towards5gs-helm

Helm charts for deploying 5G network services on Kubernetes
Other
166 stars 130 forks source link

Free5gc NRF is waiting for MongoDB indefinitely #81

Closed Zhenye-Na closed 1 year ago

Zhenye-Na commented 1 year ago

Also raised a similar issue here https://github.com/free5gc/free5gc/issues/436

I am using helm to install free5gc-nrf chart. The MongoDB dependency could be instantiated but nrf is waiting for MongoDB forever. But I verified that mongodb is up. So not sure how does this happen.

kubectl get pods

NAME                                   READY   STATUS     RESTARTS   AGE
mongodb-0                              1/1     Running    0          9m14s
nrf-free5gc-nrf-nrf-79d7d9b788-cmghx   0/1     Init:0/1   0          9m14s

If I checked the log of wait-mongo container of nrf-free5gc-nrf-nrf-79d7d9b788-cmghx

kubectl logs nrf-free5gc-nrf-nrf-79d7d9b788-cmghx -c wait-mongo
nc: bad address 'mongodb:27017'
waiting for the MongoDB
nc: bad address 'mongodb:27017'
waiting for the MongoDB
....

Meanwhile I checked the log of MongoDB-0 pod

kubectl logs mongodb-0
mongodb 04:34:33.79
mongodb 04:34:33.80 Welcome to the Bitnami mongodb container
mongodb 04:34:33.80 Subscribe to project updates by watching https://github.com/bitnami/bitnami-docker-mongodb
mongodb 04:34:33.80 Submit issues and feature requests at https://github.com/bitnami/bitnami-docker-mongodb/issues
mongodb 04:34:33.80
mongodb 04:34:33.80 INFO  ==> ** Starting MongoDB setup **
mongodb 04:34:33.82 INFO  ==> Validating settings in MONGODB_* env vars...
mongodb 04:34:33.82 WARN  ==> You set the environment variable ALLOW_EMPTY_PASSWORD=yes. For safety reasons, do not use this flag in a production environment.
mongodb 04:34:33.84 INFO  ==> Initializing MongoDB...
mongodb 04:34:33.86 INFO  ==> Deploying MongoDB with persisted data...
mongodb 04:34:33.87 INFO  ==> ** MongoDB setup finished! **

mongodb 04:34:33.89 INFO  ==> ** Starting MongoDB **

{"t":{"$date":"2023-03-18T04:34:33.934+00:00"},"s":"I",  "c":"CONTROL",  "id":20698,   "ctx":"main","msg":"***** SERVER RESTARTED *****"}
{"t":{"$date":"2023-03-18T04:34:33.939+00:00"},"s":"I",  "c":"CONTROL",  "id":23285,   "ctx":"main","msg":"Automatically disabling TLS 1.0, to force-enable TLS 1.0 specify --sslDisabledProtocols 'none'"}
{"t":{"$date":"2023-03-18T04:34:33.944+00:00"},"s":"W",  "c":"ASIO",     "id":22601,   "ctx":"main","msg":"No TransportLayer configured during NetworkInterface startup"}
{"t":{"$date":"2023-03-18T04:34:33.946+00:00"},"s":"I",  "c":"NETWORK",  "id":4648601, "ctx":"main","msg":"Implicit TCP FastOpen unavailable. If TCP FastOpen is required, set tcpFastOpenServer, tcpFastOpenClient, and tcpFastOpenQueueSize."}
{"t":{"$date":"2023-03-18T04:34:33.947+00:00"},"s":"I",  "c":"STORAGE",  "id":4615611, "ctx":"initandlisten","msg":"MongoDB starting","attr":{"pid":1,"port":27017,"dbPath":"/bitnami/mongodb/data/db","architecture":"64-bit","host":"mongodb-0"}}
{"t":{"$date":"2023-03-18T04:34:33.947+00:00"},"s":"I",  "c":"CONTROL",  "id":23403,   "ctx":"initandlisten","msg":"Build Info","attr":{"buildInfo":{"version":"4.4.4","gitVersion":"8db30a63db1a9d84bdcad0c83369623f708e0397","openSSLVersion":"OpenSSL 1.1.1d  10 Sep 2019","modules":[],"allocator":"tcmalloc","environment":{"distmod":"debian10","distarch":"x86_64","target_arch":"x86_64"}}}}
{"t":{"$date":"2023-03-18T04:34:33.947+00:00"},"s":"I",  "c":"CONTROL",  "id":51765,   "ctx":"initandlisten","msg":"Operating System","attr":{"os":{"name":"PRETTY_NAME=\"Debian GNU/Linux 10 (buster)\"","version":"Kernel 5.4.226-129.415.amzn2.x86_64"}}}
{"t":{"$date":"2023-03-18T04:34:33.947+00:00"},"s":"I",  "c":"CONTROL",  "id":21951,   "ctx":"initandlisten","msg":"Options set by command line","attr":{"options":{"config":"/opt/bitnami/mongodb/conf/mongodb.conf","net":{"bindIp":"*","ipv6":false,"port":27017,"unixDomainSocket":{"enabled":true,"pathPrefix":"/opt/bitnami/mongodb/tmp"}},"processManagement":{"fork":false,"pidFilePath":"/opt/bitnami/mongodb/tmp/mongodb.pid"},"security":{"authorization":"disabled"},"setParameter":{"enableLocalhostAuthBypass":"true"},"storage":{"dbPath":"/bitnami/mongodb/data/db","directoryPerDB":false,"journal":{"enabled":true}},"systemLog":{"destination":"file","logAppend":true,"logRotate":"reopen","path":"/opt/bitnami/mongodb/logs/mongodb.log","quiet":false,"verbosity":0}}}}
{"t":{"$date":"2023-03-18T04:34:33.951+00:00"},"s":"I",  "c":"STORAGE",  "id":22270,   "ctx":"initandlisten","msg":"Storage engine to use detected by data files","attr":{"dbpath":"/bitnami/mongodb/data/db","storageEngine":"wiredTiger"}}
{"t":{"$date":"2023-03-18T04:34:33.951+00:00"},"s":"I",  "c":"STORAGE",  "id":22297,   "ctx":"initandlisten","msg":"Using the XFS filesystem is strongly recommended with the WiredTiger storage engine. See http://dochub.mongodb.org/core/prodnotes-filesystem","tags":["startupWarnings"]}
{"t":{"$date":"2023-03-18T04:34:33.951+00:00"},"s":"I",  "c":"STORAGE",  "id":22315,   "ctx":"initandlisten","msg":"Opening WiredTiger","attr":{"config":"create,cache_size=7399M,session_max=33000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000,close_scan_interval=10,close_handle_minimum=250),statistics_log=(wait=0),verbose=[recovery_progress,checkpoint_progress,compact_progress],"}}
{"t":{"$date":"2023-03-18T04:34:34.563+00:00"},"s":"I",  "c":"STORAGE",  "id":22430,   "ctx":"initandlisten","msg":"WiredTiger message","attr":{"message":"[1679114074:563496][1:0x7f24214f2140], txn-recover: [WT_VERB_RECOVERY_PROGRESS] Recovering log 1 through 2"}}
{"t":{"$date":"2023-03-18T04:34:34.646+00:00"},"s":"I",  "c":"STORAGE",  "id":22430,   "ctx":"initandlisten","msg":"WiredTiger message","attr":{"message":"[1679114074:645988][1:0x7f24214f2140], txn-recover: [WT_VERB_RECOVERY_PROGRESS] Recovering log 2 through 2"}}
{"t":{"$date":"2023-03-18T04:34:34.722+00:00"},"s":"I",  "c":"STORAGE",  "id":22430,   "ctx":"initandlisten","msg":"WiredTiger message","attr":{"message":"[1679114074:722877][1:0x7f24214f2140], txn-recover: [WT_VERB_RECOVERY | WT_VERB_RECOVERY_PROGRESS] Main recovery loop: starting at 1/26624 to 2/256"}}
{"t":{"$date":"2023-03-18T04:34:34.816+00:00"},"s":"I",  "c":"STORAGE",  "id":22430,   "ctx":"initandlisten","msg":"WiredTiger message","attr":{"message":"[1679114074:816017][1:0x7f24214f2140], txn-recover: [WT_VERB_RECOVERY_PROGRESS] Recovering log 1 through 2"}}
{"t":{"$date":"2023-03-18T04:34:34.871+00:00"},"s":"I",  "c":"STORAGE",  "id":22430,   "ctx":"initandlisten","msg":"WiredTiger message","attr":{"message":"[1679114074:871564][1:0x7f24214f2140], txn-recover: [WT_VERB_RECOVERY_PROGRESS] Recovering log 2 through 2"}}
{"t":{"$date":"2023-03-18T04:34:34.918+00:00"},"s":"I",  "c":"STORAGE",  "id":22430,   "ctx":"initandlisten","msg":"WiredTiger message","attr":{"message":"[1679114074:918108][1:0x7f24214f2140], txn-recover: [WT_VERB_RECOVERY | WT_VERB_RECOVERY_PROGRESS] Set global recovery timestamp: (0, 0)"}}
{"t":{"$date":"2023-03-18T04:34:34.918+00:00"},"s":"I",  "c":"STORAGE",  "id":22430,   "ctx":"initandlisten","msg":"WiredTiger message","attr":{"message":"[1679114074:918183][1:0x7f24214f2140], txn-recover: [WT_VERB_RECOVERY | WT_VERB_RECOVERY_PROGRESS] Set global oldest timestamp: (0, 0)"}}
{"t":{"$date":"2023-03-18T04:34:34.942+00:00"},"s":"I",  "c":"STORAGE",  "id":4795906, "ctx":"initandlisten","msg":"WiredTiger opened","attr":{"durationMillis":991}}
{"t":{"$date":"2023-03-18T04:34:34.942+00:00"},"s":"I",  "c":"RECOVERY", "id":23987,   "ctx":"initandlisten","msg":"WiredTiger recoveryTimestamp","attr":{"recoveryTimestamp":{"$timestamp":{"t":0,"i":0}}}}
{"t":{"$date":"2023-03-18T04:34:34.943+00:00"},"s":"I",  "c":"STORAGE",  "id":4366408, "ctx":"initandlisten","msg":"No table logging settings modifications are required for existing WiredTiger tables","attr":{"loggingEnabled":true}}
{"t":{"$date":"2023-03-18T04:34:34.943+00:00"},"s":"I",  "c":"STORAGE",  "id":22262,   "ctx":"initandlisten","msg":"Timestamp monitor starting"}
{"t":{"$date":"2023-03-18T04:34:34.947+00:00"},"s":"I",  "c":"STORAGE",  "id":20536,   "ctx":"initandlisten","msg":"Flow Control is enabled on this deployment"}
{"t":{"$date":"2023-03-18T04:34:34.947+00:00"},"s":"I",  "c":"FTDC",     "id":20625,   "ctx":"initandlisten","msg":"Initializing full-time diagnostic data capture","attr":{"dataDirectory":"/bitnami/mongodb/data/db/diagnostic.data"}}
{"t":{"$date":"2023-03-18T04:34:34.949+00:00"},"s":"I",  "c":"NETWORK",  "id":23015,   "ctx":"listener","msg":"Listening on","attr":{"address":"/opt/bitnami/mongodb/tmp/mongodb-27017.sock"}}
{"t":{"$date":"2023-03-18T04:34:34.949+00:00"},"s":"I",  "c":"NETWORK",  "id":23015,   "ctx":"listener","msg":"Listening on","attr":{"address":"0.0.0.0"}}
{"t":{"$date":"2023-03-18T04:34:34.949+00:00"},"s":"I",  "c":"NETWORK",  "id":23016,   "ctx":"listener","msg":"Waiting for connections","attr":{"port":27017,"ssl":"off"}}
{"t":{"$date":"2023-03-18T04:34:42.980+00:00"},"s":"I",  "c":"NETWORK",  "id":22943,   "ctx":"listener","msg":"Connection accepted","attr":{"remote":"127.0.0.1:38498","connectionId":1,"connectionCount":1}}
{"t":{"$date":"2023-03-18T04:34:42.980+00:00"},"s":"I",  "c":"NETWORK",  "id":51800,   "ctx":"conn1","msg":"client metadata","attr":{"remote":"127.0.0.1:38498","client":"conn1","doc":{"application":{"name":"MongoDB Shell"},"driver":{"name":"MongoDB Internal Client","version":"4.4.4"},"os":{"type":"Linux","name":"PRETTY_NAME=\"Debian GNU/Linux 10 (buster)\"","architecture":"x86_64","version":"Kernel 5.4.226-129.415.amzn2.x86_64"}}}}
{"t":{"$date":"2023-03-18T04:34:42.984+00:00"},"s":"I",  "c":"NETWORK",  "id":22944,   "ctx":"conn1","msg":"Connection ended","attr":{"remote":"127.0.0.1:38498","connectionId":1,"connectionCount":0}}
{"t":{"$date":"2023-03-18T04:34:52.979+00:00"},"s":"I",  "c":"NETWORK",  "id":22943,   "ctx":"listener","msg":"Connection accepted","attr":{"remote":"127.0.0.1:37554","connectionId":2,"connectionCount":1}}
{"t":{"$date":"2023-03-18T04:34:52.980+00:00"},"s":"I",  "c":"NETWORK",  "id":51800,   "ctx":"conn2","msg":"client metadata","attr":{"remote":"127.0.0.1:37554","client":"conn2","doc":{"application":{"name":"MongoDB Shell"},"driver":{"name":"MongoDB Internal Client","version":"4.4.4"},"os":{"type":"Linux","name":"PRETTY_NAME=\"Debian GNU/Linux 10 (buster)\"","architecture":"x86_64","version":"Kernel 5.4.226-129.415.amzn2.x86_64"}}}}
{"t":{"$date":"2023-03-18T04:34:52.984+00:00"},"s":"I",  "c":"NETWORK",  "id":22944,   "ctx":"conn2","msg":"Connection ended","attr":{"remote":"127.0.0.1:37554","connectionId":2,"connectionCount":0}}
{"t":{"$date":"2023-03-18T04:35:03.010+00:00"},"s":"I",  "c":"NETWORK",  "id":22943,   "ctx":"listener","msg":"Connection accepted","attr":{"remote":"127.0.0.1:36944","connectionId":3,"connectionCount":1}}
{"t":{"$date":"2023-03-18T04:35:03.011+00:00"},"s":"I",  "c":"NETWORK",  "id":51800,   "ctx":"conn3","msg":"client metadata","attr":{"remote":"127.0.0.1:36944","client":"conn3","doc":{"application":{"name":"MongoDB Shell"},"driver":{"name":"MongoDB Internal Client","version":"4.4.4"},"os":{"type":"Linux","name":"PRETTY_NAME=\"Debian GNU/Linux 10 (buster)\"","architecture":"x86_64","version":"Kernel 5.4.226-129.415.amzn2.x86_64"}}}}
{"t":{"$date":"2023-03-18T04:35:03.014+00:00"},"s":"I",  "c":"NETWORK",  "id":22944,   "ctx":"conn3","msg":"Connection ended","attr":{"remote":"127.0.0.1:36944","connectionId":3,"connectionCount":0}}
{"t":{"$date":"2023-03-18T04:35:13.023+00:00"},"s":"I",  "c":"NETWORK",  "id":22943,   "ctx":"listener","msg":"Connection accepted","attr":{"remote":"127.0.0.1:59666","connectionId":4,"connectionCount":1}}
{"t":{"$date":"2023-03-18T04:35:13.023+00:00"},"s":"I",  "c":"NETWORK",  "id":51800,   "ctx":"conn4","msg":"client metadata","attr":{"remote":"127.0.0.1:59666","client":"conn4","doc":{"application":{"name":"MongoDB Shell"},"driver":{"name":"MongoDB Internal Client","version":"4.4.4"},"os":{"type":"Linux","name":"PRETTY_NAME=\"Debian GNU/Linux 10 (buster)\"","architecture":"x86_64","version":"Kernel 5.4.226-129.415.amzn2.x86_64"}}}}
Zhenye-Na commented 1 year ago

Added more findings:

It seems like the NRF chart works when:

  1. AWS EKS Cluster has only instances coming from EKS Managed Node Group
  2. AWS EKS Cluster has only instances coming from EKS Self-managed Node Group

If there are multiple instances, and the source of them is mix-matching. The NRF helm chart is installed and MongoDB is running, the wait-mongo container of NRF could not find the the Mongo pods, with same logs found above

kubectl logs nrf-free5gc-nrf-nrf-79d7d9b788-cmghx -c wait-mongo
nc: bad address 'mongodb:27017'
waiting for the MongoDB
nc: bad address 'mongodb:27017'
waiting for the MongoDB

In AWS EKS Console, I am able to see that mongodb-0 and nrf-free5gc-nrf-nrf-79d7d9b788-cmghx are installed in the same EC2 instances, resources such as memory, total number of pods or IPs are not running up.

So I would really appreciate it if any tips or findings can be provided.

Thanks!!

Zhenye-Na commented 1 year ago

Closed this issue since it is not related to specific helm chart implementation.

For anyone who is curious, this issue is related to DNS lookup issue that is similar to https://github.com/aws/aws-cdk/issues/10884