shellhub-io / shellhub

:computer: Get seamless remote access to any Linux device. Centralized SSH for the edge and cloud computing
https://shellhub.io
Apache License 2.0
1.38k stars 137 forks source link

Issue Updating from 0.5.0 to 0.6.0 on self hosted server #812

Closed suegy closed 3 years ago

suegy commented 3 years ago

Configuration:

Steps: 1.) Updating the repo using git checkout v0.6.0 on previously working 0.5.0 2.) Update mongo-db 3.) Run make start

Output: `Starting ShellHub in production mode...

WARNING: The SHELLHUB_CLOUD variable is not set. Defaulting to a blank string. Removing shellhub_gateway_1 shellhub_ssh_1 is up-to-date shellhub_mongo_1 is up-to-date shellhub_api_1 is up-to-date shellhub_ui_1 is up-to-date Recreating d02aac3da26b_shellhub_gateway_1 ... error

ERROR: for d02aac3da26b_shellhub_gateway_1 Cannot start service gateway: OCI runtime create failed: container_linux.go:367: starting container process caused: process_linux.go:352: getting the final child's pid from pipe caused: EOF: unknown

ERROR: for gateway Cannot start service gateway: OCI runtime create failed: container_linux.go:367: starting container process caused: process_linux.go:352: getting the final child's pid from pipe caused: EOF: unknown ERROR: Encountered errors while bringing up the project. Makefile:41: recipe for target 'start' failed make: *** [start] Error 1 `

otavio commented 3 years ago

Can you gather logs from the service? it might help to find out what is failing.

suegy commented 3 years ago

where do I find the logs as the docker did not start the gateway so the docker logs do not have any info on why the gateway is not building.

otavio commented 3 years ago

See if https://github.com/shellhub-io/shellhub/commit/fc34f1efe6a7722340b4f6dae1ebf5154328494a helps, please.

suegy commented 3 years ago

Same message and output. Setting the cloud to false only removed the warning.

otavio commented 3 years ago

Take a look here: https://github.com/moby/moby/issues/40835

Looks to be on your side. Might be a maximum number of namespace, pids or other constrains.

suegy commented 3 years ago

None of their recommendations work and I can still build and run 0.5.0 whereas for most of their issues, they cannot run a docker container at all. The issue is that all other containers except gateway build and run fine.

otavio commented 3 years ago

I'll let @gustavosbarreto take a look at this. I am not good at Docker, so it is better to wait for his feedback.

suegy commented 3 years ago

When I check the docker logs for the running containers I get the following error in the shellhub_api: panic: server selection error: server selection timeout, current topology: { Type: Unknown, Servers: [{ Addr: mongo:27017, Type: Unknown, Average RTT: 0, Last error: connection() error occured during connection handshake: dial tcp: lookup mongo on 127.0.0.11:53: no such host }, ] }

I used the make file to upgrade the mongodb, could this be related to that?

gustavosbarreto commented 3 years ago

@suegy @otavio Unfortunately I cannot reproduce this here. Are there any more information?

Below is the steps that i did trying to reproduce the issue:

$ git clone -b v0.5.0 https://github.com/shellhub-io/shellhub.git
$ cd shellhub
$ ./bin/keygen
$ ./bin/docker-compose up
$ ./bin/add-user user passwd user@mail.com
$ ./bin/add-namespace testing user
$ ./bin/docker-compose stop
$ git checkout v0.6.0
$ make upgrade_mongodb
$ make start

@suegy I used the make file to upgrade the mongodb, could this be related to that?

It can't be related but it would be nice to make sure MongoDB is up and running after upgrade process.

Please paste the output of following commands:

suegy commented 3 years ago

for config:

  shellhub:
    name: shellhub_network
secrets:
  api_private_key:
    file: <path>/shellhub/api_private_key
  api_public_key:
    file: <path>/shellhub/api_public_key
  ssh_private_key:
    file: <path>/shellhub/ssh_private_key
services:
  api:
    depends_on:
    - mongo
    environment:
      PRIVATE_KEY: /run/secrets/api_private_key
      PUBLIC_KEY: /run/secrets/api_public_key
      SHELLHUB_ENTERPRISE: "false"
    image: shellhubio/api:v0.6.0
    links:
    - mongo
    networks:
      shellhub: {}
    restart: unless-stopped
    secrets:
    - source: api_private_key
    - source: api_public_key
  gateway:
    depends_on:
    - api
    - ui
    environment:
      SHELLHUB_ENTERPRISE: "false"
      SHELLHUB_PROXY: "false"
      SHELLHUB_SSH_PORT: '2222'
      SHELLHUB_VERSION: v0.6.0
    image: shellhubio/gateway:v0.6.0
    networks:
      shellhub: {}
    ports:
    - published: 8081
      target: 80
    restart: unless-stopped
  mongo:
    image: mongo:4.4.4
    networks:
      shellhub: {}
    restart: unless-stopped
    volumes:
    - /usr/local/mongo/data:/data/db:rw
  ssh:
    environment:
      PRIVATE_KEY: /run/secrets/ssh_private_key
      RECORD_URL: api:8080
      SHELLHUB_ENTERPRISE: "false"
      WEBHOOK_PORT: ''
      WEBHOOK_SCHEME: ''
      WEBHOOK_URL: ''
    image: shellhubio/ssh:v0.6.0
    networks:
      shellhub: {}
    ports:
    - published: 2222
      target: 2222
    restart: unless-stopped
    secrets:
    - source: ssh_private_key
  ui:
    depends_on:
    - api
    environment:
      SHELLHUB_CLOUD: "false"
      SHELLHUB_ENTERPRISE: "false"
    image: shellhubio/ui:v0.6.0
    networks:
      shellhub: {}
    restart: unless-stopped
version: '3.7'

for ps:

       Name                   Command           State             Ports
--------------------------------------------------------------------------------
00fc91e249ec_shellhub   /entrypoint.sh          Exit 128
_gateway_1              /usr/local/ ...
shellhub_api_1          /bin/sh -c /api         Up
shellhub_mongo_1        docker-entrypoint.sh    Up         27017/tcp
                        mongod
shellhub_ssh_1          /bin/sh -c /ssh         Up         0.0.0.0:2222->2222/tc
                                                           p
shellhub_ui_1           /scripts/entrypoint.s   Up         80/tcp
                        h

for logs mongo:

mongo_1    | {"t":{"$date":"2021-03-20T22:31:55.404+00:00"},"s":"I",  "c":"CONTR                                                   OL",  "id":23285,   "ctx":"main","msg":"Automatically disabling TLS 1.0, to forc                                                   e-enable TLS 1.0 specify --sslDisabledProtocols 'none'"}
mongo_1    | {"t":{"$date":"2021-03-20T22:31:55.426+00:00"},"s":"W",  "c":"ASIO"                                                   ,     "id":22601,   "ctx":"main","msg":"No TransportLayer configured during Netw                                                   orkInterface startup"}
mongo_1    | {"t":{"$date":"2021-03-20T22:31:55.426+00:00"},"s":"I",  "c":"NETWO                                                   RK",  "id":4648601, "ctx":"main","msg":"Implicit TCP FastOpen unavailable. If TC                                                   P FastOpen is required, set tcpFastOpenServer, tcpFastOpenClient, and tcpFastOpe                                                   nQueueSize."}
mongo_1    | {"t":{"$date":"2021-03-20T22:31:55.442+00:00"},"s":"I",  "c":"STORA                                                   GE",  "id":4615611, "ctx":"initandlisten","msg":"MongoDB starting","attr":{"pid"                                                   :1,"port":27017,"dbPath":"/data/db","architecture":"64-bit","host":"1bd705aa4d16                                                   "}}
mongo_1    | {"t":{"$date":"2021-03-20T22:31:55.442+00:00"},"s":"I",  "c":"CONTR                                                   OL",  "id":23403,   "ctx":"initandlisten","msg":"Build Info","attr":{"buildInfo"                                                   :{"version":"4.4.4","gitVersion":"8db30a63db1a9d84bdcad0c83369623f708e0397","ope                                                   nSSLVersion":"OpenSSL 1.1.1  11 Sep 2018","modules":[],"allocator":"tcmalloc","e                                                   nvironment":{"distmod":"ubuntu1804","distarch":"x86_64","target_arch":"x86_64"}}                                                   }}
mongo_1    | {"t":{"$date":"2021-03-20T22:31:55.442+00:00"},"s":"I",  "c":"CONTR                                                   OL",  "id":51765,   "ctx":"initandlisten","msg":"Operating System","attr":{"os":                                                   {"name":"Ubuntu","version":"18.04"}}}
mongo_1    | {"t":{"$date":"2021-03-20T22:31:55.442+00:00"},"s":"I",  "c":"CONTR                                                   OL",  "id":21951,   "ctx":"initandlisten","msg":"Options set by command line","a                                                   ttr":{"options":{"net":{"bindIp":"*"}}}}
mongo_1    | {"t":{"$date":"2021-03-20T22:31:55.496+00:00"},"s":"I",  "c":"STORA                                                   GE",  "id":22270,   "ctx":"initandlisten","msg":"Storage engine to use detected                                                    by data files","attr":{"dbpath":"/data/db","storageEngine":"wiredTiger"}}
mongo_1    | {"t":{"$date":"2021-03-20T22:31:55.496+00:00"},"s":"I",  "c":"STORA                                                   GE",  "id":22297,   "ctx":"initandlisten","msg":"Using the XFS filesystem is str                                                   ongly recommended with the WiredTiger storage engine. See http://dochub.mongodb.                                                   org/core/prodnotes-filesystem","tags":["startupWarnings"]}
mongo_1    | {"t":{"$date":"2021-03-20T22:31:55.496+00:00"},"s":"I",  "c":"STORA                                                   GE",  "id":22315,   "ctx":"initandlisten","msg":"Opening WiredTiger","attr":{"co                                                   nfig":"create,cache_size=512M,session_max=33000,eviction=(threads_min=4,threads_                                                   max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=j                                                   ournal,compressor=snappy),file_manager=(close_idle_time=100000,close_scan_interv                                                   al=10,close_handle_minimum=250),statistics_log=(wait=0),verbose=[recovery_progre                                                   ss,checkpoint_progress,compact_progress],"}}
mongo_1    | {"t":{"$date":"2021-03-20T22:31:57.096+00:00"},"s":"I",  "c":"STORA                                                   GE",  "id":22430,   "ctx":"initandlisten","msg":"WiredTiger message","attr":{"me                                                   ssage":"[1616279517:96000][1:0x7f8e0875eac0], txn-recover: [WT_VERB_RECOVERY_PRO                                                   GRESS] Recovering log 4 through 5"}}
mongo_1    | {"t":{"$date":"2021-03-20T22:31:58.099+00:00"},"s":"I",  "c":"STORA                                                   GE",  "id":22430,   "ctx":"initandlisten","msg":"WiredTiger message","attr":{"me                                                   ssage":"[1616279518:99380][1:0x7f8e0875eac0], txn-recover: [WT_VERB_RECOVERY_PRO                                                   GRESS] Recovering log 5 through 5"}}
mongo_1    | {"t":{"$date":"2021-03-20T22:32:01.447+00:00"},"s":"I",  "c":"STORA                                                   GE",  "id":22430,   "ctx":"initandlisten","msg":"WiredTiger message","attr":{"me                                                   ssage":"[1616279521:447421][1:0x7f8e0875eac0], txn-recover: [WT_VERB_RECOVERY |                                                    WT_VERB_RECOVERY_PROGRESS] Main recovery loop: starting at 4/18688 to 5/256"}}
mongo_1    | {"t":{"$date":"2021-03-20T22:32:01.701+00:00"},"s":"I",  "c":"STORA                                                   GE",  "id":22430,   "ctx":"initandlisten","msg":"WiredTiger message","attr":{"me                                                   ssage":"[1616279521:701444][1:0x7f8e0875eac0], txn-recover: [WT_VERB_RECOVERY_PR                                                   OGRESS] Recovering log 4 through 5"}}
mongo_1    | {"t":{"$date":"2021-03-20T22:32:01.922+00:00"},"s":"I",  "c":"STORA                                                   GE",  "id":22430,   "ctx":"initandlisten","msg":"WiredTiger message","attr":{"me                                                   ssage":"[1616279521:922405][1:0x7f8e0875eac0], txn-recover: [WT_VERB_RECOVERY_PR                                                   OGRESS] Recovering log 5 through 5"}}
mongo_1    | {"t":{"$date":"2021-03-20T22:32:02.076+00:00"},"s":"I",  "c":"STORA                                                   GE",  "id":22430,   "ctx":"initandlisten","msg":"WiredTiger message","attr":{"me                                                   ssage":"[1616279522:76668][1:0x7f8e0875eac0], txn-recover: [WT_VERB_RECOVERY | W                                                   T_VERB_RECOVERY_PROGRESS] Set global recovery timestamp: (0, 0)"}}
mongo_1    | {"t":{"$date":"2021-03-20T22:32:02.079+00:00"},"s":"I",  "c":"STORA                                                   GE",  "id":22430,   "ctx":"initandlisten","msg":"WiredTiger message","attr":{"me                                                   ssage":"[1616279522:79384][1:0x7f8e0875eac0], txn-recover: [WT_VERB_RECOVERY | W                                                   T_VERB_RECOVERY_PROGRESS] Set global oldest timestamp: (0, 0)"}}
mongo_1    | {"t":{"$date":"2021-03-20T22:32:02.357+00:00"},"s":"I",  "c":"STORA                                                   GE",  "id":4795906, "ctx":"initandlisten","msg":"WiredTiger opened","attr":{"dur                                                   ationMillis":6861}}
mongo_1    | {"t":{"$date":"2021-03-20T22:32:02.358+00:00"},"s":"I",  "c":"RECOV                                                   ERY", "id":23987,   "ctx":"initandlisten","msg":"WiredTiger recoveryTimestamp","                                                   attr":{"recoveryTimestamp":{"$timestamp":{"t":0,"i":0}}}}
mongo_1    | {"t":{"$date":"2021-03-20T22:32:02.361+00:00"},"s":"I",  "c":"STORA                                                   GE",  "id":4366408, "ctx":"initandlisten","msg":"No table logging settings modif                                                   ications are required for existing WiredTiger tables","attr":{"loggingEnabled":t                                                   rue}}
mongo_1    | {"t":{"$date":"2021-03-20T22:32:02.367+00:00"},"s":"I",  "c":"STORA                                                   GE",  "id":22262,   "ctx":"initandlisten","msg":"Timestamp monitor starting"}
mongo_1    | {"t":{"$date":"2021-03-20T22:32:02.368+00:00"},"s":"W",  "c":"CONTR                                                   OL",  "id":22120,   "ctx":"initandlisten","msg":"Access control is not enabled f                                                   or the database. Read and write access to data and configuration is unrestricted                                                   ","tags":["startupWarnings"]}
mongo_1    | {"t":{"$date":"2021-03-20T22:32:02.369+00:00"},"s":"I",  "c":"CONTR                                                   OL",  "id":22161,   "ctx":"initandlisten","msg":"You are running in OpenVZ which                                                    can cause issues on versions of RHEL older than RHEL6","tags":["startupWarnings                                                   "]}
mongo_1    | {"t":{"$date":"2021-03-20T22:32:02.446+00:00"},"s":"I",  "c":"STORA                                                   GE",  "id":20536,   "ctx":"initandlisten","msg":"Flow Control is enabled on this                                                    deployment"}
mongo_1    | {"t":{"$date":"2021-03-20T22:32:02.500+00:00"},"s":"I",  "c":"FTDC"                                                   ,     "id":20625,   "ctx":"initandlisten","msg":"Initializing full-time diagnost                                                   ic data capture","attr":{"dataDirectory":"/data/db/diagnostic.data"}}
mongo_1    | {"t":{"$date":"2021-03-20T22:32:02.506+00:00"},"s":"I",  "c":"NETWO                                                   RK",  "id":23015,   "ctx":"listener","msg":"Listening on","attr":{"address":"/tm                                                   p/mongodb-27017.sock"}}
mongo_1    | {"t":{"$date":"2021-03-20T22:32:02.506+00:00"},"s":"I",  "c":"NETWO                                                   RK",  "id":23015,   "ctx":"listener","msg":"Listening on","attr":{"address":"0.0                                                   .0.0"}}
mongo_1    | {"t":{"$date":"2021-03-20T22:32:02.506+00:00"},"s":"I",  "c":"NETWO                                                   RK",  "id":23016,   "ctx":"listener","msg":"Waiting for connections","attr":{"po                                                   rt":27017,"ssl":"off"}}
mongo_1    | {"t":{"$date":"2021-03-20T22:32:02.603+00:00"},"s":"I",  "c":"NETWO                                                   RK",  "id":22943,   "ctx":"listener","msg":"Connection accepted","attr":{"remote                                                   ":"172.18.0.4:52268","connectionId":1,"connectionCount":1}}
mongo_1    | {"t":{"$date":"2021-03-20T22:32:02.801+00:00"},"s":"I",  "c":"NETWO                                                   RK",  "id":51800,   "ctx":"conn1","msg":"client metadata","attr":{"remote":"172.                                                   18.0.4:52268","client":"conn1","doc":{"driver":{"name":"mongo-go-driver","versio                                                   n":"v1.5.0"},"os":{"type":"linux","architecture":"amd64"},"platform":"go1.13.15"                                                   }}}
mongo_1    | {"t":{"$date":"2021-03-20T22:32:02.814+00:00"},"s":"I",  "c":"NETWO                                                   RK",  "id":22943,   "ctx":"listener","msg":"Connection accepted","attr":{"remote                                                   ":"172.18.0.4:52278","connectionId":2,"connectionCount":2}}
mongo_1    | {"t":{"$date":"2021-03-20T22:32:02.815+00:00"},"s":"I",  "c":"NETWO                                                   RK",  "id":51800,   "ctx":"conn2","msg":"client metadata","attr":{"remote":"172.                                                   18.0.4:52278","client":"conn2","doc":{"driver":{"name":"mongo-go-driver","versio                                                   n":"v1.5.0"},"os":{"type":"linux","architecture":"amd64"},"platform":"go1.13.15"                                                   }}}
mongo_1    | {"t":{"$date":"2021-03-20T22:32:06.554+00:00"},"s":"I",  "c":"NETWO                                                   RK",  "id":22943,   "ctx":"listener","msg":"Connection accepted","attr":{"remote                                                   ":"172.18.0.4:52480","connectionId":3,"connectionCount":3}}
mongo_1    | {"t":{"$date":"2021-03-20T22:32:06.554+00:00"},"s":"I",  "c":"NETWO                                                   RK",  "id":51800,   "ctx":"conn3","msg":"client metadata","attr":{"remote":"172.                                                   18.0.4:52480","client":"conn3","doc":{"driver":{"name":"mongo-go-driver","versio                                                   n":"v1.5.0"},"os":{"type":"linux","architecture":"amd64"},"platform":"go1.13.15"                                                   }}}
mongo_1    | {"t":{"$date":"2021-03-20T22:32:19.471+00:00"},"s":"I",  "c":"-",                                                           "id":20883,   "ctx":"conn1","msg":"Interrupted operation as its client dis                                                   connected","attr":{"opId":151}}
mongo_1    | {"t":{"$date":"2021-03-20T22:32:19.471+00:00"},"s":"I",  "c":"NETWO                                                   RK",  "id":22944,   "ctx":"conn3","msg":"Connection ended","attr":{"remote":"172                                                   .18.0.4:52480","connectionId":3,"connectionCount":2}}
mongo_1    | {"t":{"$date":"2021-03-20T22:32:19.471+00:00"},"s":"I",  "c":"NETWO                                                   RK",  "id":22944,   "ctx":"conn2","msg":"Connection ended","attr":{"remote":"172                                                   .18.0.4:52278","connectionId":2,"connectionCount":1}}
mongo_1    | {"t":{"$date":"2021-03-20T22:32:19.492+00:00"},"s":"I",  "c":"NETWO                                                   RK",  "id":22944,   "ctx":"conn1","msg":"Connection ended","attr":{"remote":"172                                                   .18.0.4:52268","connectionId":1,"connectionCount":0}}
mongo_1    | {"t":{"$date":"2021-03-20T22:32:20.053+00:00"},"s":"I",  "c":"CONTR                                                   OL",  "id":23377,   "ctx":"SignalHandler","msg":"Received signal","attr":{"signa                                                   l":15,"error":"Terminated"}}
mongo_1    | {"t":{"$date":"2021-03-20T22:32:20.053+00:00"},"s":"I",  "c":"CONTR                                                   OL",  "id":23378,   "ctx":"SignalHandler","msg":"Signal was sent by kill(2)","at                                                   tr":{"pid":0,"uid":0}}
mongo_1    | {"t":{"$date":"2021-03-20T22:32:20.056+00:00"},"s":"I",  "c":"CONTR                                                   OL",  "id":23381,   "ctx":"SignalHandler","msg":"will terminate after current cm                                                   d ends"}
mongo_1    | {"t":{"$date":"2021-03-20T22:32:20.057+00:00"},"s":"I",  "c":"REPL"                                                   ,     "id":4784900, "ctx":"SignalHandler","msg":"Stepping down the ReplicationCo                                                   ordinator for shutdown","attr":{"waitTimeMillis":10000}}
mongo_1    | {"t":{"$date":"2021-03-20T22:32:20.058+00:00"},"s":"I",  "c":"COMMA                                                   ND",  "id":4784901, "ctx":"SignalHandler","msg":"Shutting down the MirrorMaestro                                                   "}
mongo_1    | {"t":{"$date":"2021-03-20T22:32:20.058+00:00"},"s":"I",  "c":"SHARD                                                   ING", "id":4784902, "ctx":"SignalHandler","msg":"Shutting down the WaitForMajori                                                   tyService"}
mongo_1    | {"t":{"$date":"2021-03-20T22:32:20.065+00:00"},"s":"I",  "c":"CONTR                                                   OL",  "id":4784903, "ctx":"SignalHandler","msg":"Shutting down the LogicalSessio                                                   nCache"}
mongo_1    | {"t":{"$date":"2021-03-20T22:32:20.065+00:00"},"s":"I",  "c":"NETWO                                                   RK",  "id":20562,   "ctx":"SignalHandler","msg":"Shutdown: going to close listen                                                   ing sockets"}
mongo_1    | {"t":{"$date":"2021-03-20T22:32:20.072+00:00"},"s":"I",  "c":"NETWO                                                   RK",  "id":23017,   "ctx":"listener","msg":"removing socket file","attr":{"path"                                                   :"/tmp/mongodb-27017.sock"}}
mongo_1    | {"t":{"$date":"2021-03-20T22:32:20.072+00:00"},"s":"I",  "c":"NETWO                                                   RK",  "id":4784905, "ctx":"SignalHandler","msg":"Shutting down the global connec                                                   tion pool"}
mongo_1    | {"t":{"$date":"2021-03-20T22:32:20.072+00:00"},"s":"I",  "c":"STORA                                                   GE",  "id":4784906, "ctx":"SignalHandler","msg":"Shutting down the FlowControlTi                                                   cketholder"}
mongo_1    | {"t":{"$date":"2021-03-20T22:32:20.072+00:00"},"s":"I",  "c":"-",                                                           "id":20520,   "ctx":"SignalHandler","msg":"Stopping further Flow Control t                                                   icket acquisitions."}
mongo_1    | {"t":{"$date":"2021-03-20T22:32:20.072+00:00"},"s":"I",  "c":"STORA                                                   GE",  "id":4784908, "ctx":"SignalHandler","msg":"Shutting down the PeriodicThrea                                                   dToAbortExpiredTransactions"}
mongo_1    | {"t":{"$date":"2021-03-20T22:32:20.072+00:00"},"s":"I",  "c":"STORA                                                   GE",  "id":4784934, "ctx":"SignalHandler","msg":"Shutting down the PeriodicThrea                                                   dToDecreaseSnapshotHistoryCachePressure"}
mongo_1    | {"t":{"$date":"2021-03-20T22:32:20.072+00:00"},"s":"I",  "c":"REPL"                                                   ,     "id":4784909, "ctx":"SignalHandler","msg":"Shutting down the ReplicationCo                                                   ordinator"}
mongo_1    | {"t":{"$date":"2021-03-20T22:32:20.072+00:00"},"s":"I",  "c":"SHARD                                                   ING", "id":4784910, "ctx":"SignalHandler","msg":"Shutting down the ShardingIniti                                                   alizationMongoD"}
mongo_1    | {"t":{"$date":"2021-03-20T22:32:20.072+00:00"},"s":"I",  "c":"REPL"                                                   ,     "id":4784911, "ctx":"SignalHandler","msg":"Enqueuing the ReplicationStateT                                                   ransitionLock for shutdown"}
mongo_1    | {"t":{"$date":"2021-03-20T22:32:20.072+00:00"},"s":"I",  "c":"-",                                                           "id":4784912, "ctx":"SignalHandler","msg":"Killing all operations for shut                                                   down"}
mongo_1    | {"t":{"$date":"2021-03-20T22:32:20.072+00:00"},"s":"I",  "c":"-",                                                           "id":4695300, "ctx":"SignalHandler","msg":"Interrupted all currently runni                                                   ng operations","attr":{"opsKilled":3}}
mongo_1    | {"t":{"$date":"2021-03-20T22:32:20.072+00:00"},"s":"I",  "c":"COMMA                                                   ND",  "id":4784913, "ctx":"SignalHandler","msg":"Shutting down all open transact                                                   ions"}
mongo_1    | {"t":{"$date":"2021-03-20T22:32:20.072+00:00"},"s":"I",  "c":"REPL"                                                   ,     "id":4784914, "ctx":"SignalHandler","msg":"Acquiring the ReplicationStateT                                                   ransitionLock for shutdown"}
mongo_1    | {"t":{"$date":"2021-03-20T22:32:20.072+00:00"},"s":"I",  "c":"INDEX                                                   ",    "id":4784915, "ctx":"SignalHandler","msg":"Shutting down the IndexBuildsCo                                                   ordinator"}
mongo_1    | {"t":{"$date":"2021-03-20T22:32:20.072+00:00"},"s":"I",  "c":"REPL"                                                   ,     "id":4784916, "ctx":"SignalHandler","msg":"Reacquiring the ReplicationStat                                                   eTransitionLock for shutdown"}
mongo_1    | {"t":{"$date":"2021-03-20T22:32:20.072+00:00"},"s":"I",  "c":"REPL"                                                   ,     "id":4784917, "ctx":"SignalHandler","msg":"Attempting to mark clean shutdo                                                   wn"}
mongo_1    | {"t":{"$date":"2021-03-20T22:32:20.072+00:00"},"s":"I",  "c":"NETWO                                                   RK",  "id":4784918, "ctx":"SignalHandler","msg":"Shutting down the ReplicaSetMon                                                   itor"}
mongo_1    | {"t":{"$date":"2021-03-20T22:32:20.072+00:00"},"s":"I",  "c":"SHARD                                                   ING", "id":4784921, "ctx":"SignalHandler","msg":"Shutting down the MigrationUtil                                                   Executor"}
mongo_1    | {"t":{"$date":"2021-03-20T22:32:20.073+00:00"},"s":"I",  "c":"CONTR                                                   OL",  "id":4784925, "ctx":"SignalHandler","msg":"Shutting down free monitoring"}
mongo_1    | {"t":{"$date":"2021-03-20T22:32:20.073+00:00"},"s":"I",  "c":"CONTR                                                   OL",  "id":20609,   "ctx":"SignalHandler","msg":"Shutting down free monitoring"}
mongo_1    | {"t":{"$date":"2021-03-20T22:32:20.073+00:00"},"s":"I",  "c":"STORA                                                   GE",  "id":4784927, "ctx":"SignalHandler","msg":"Shutting down the HealthLog"}
mongo_1    | {"t":{"$date":"2021-03-20T22:32:20.073+00:00"},"s":"I",  "c":"STORA                                                   GE",  "id":4784929, "ctx":"SignalHandler","msg":"Acquiring the global lock for s                                                   hutdown"}
mongo_1    | {"t":{"$date":"2021-03-20T22:32:20.073+00:00"},"s":"I",  "c":"STORA                                                   GE",  "id":4784930, "ctx":"SignalHandler","msg":"Shutting down the storage engin                                                   e"}
mongo_1    | {"t":{"$date":"2021-03-20T22:32:20.073+00:00"},"s":"I",  "c":"STORA                                                   GE",  "id":20282,   "ctx":"SignalHandler","msg":"Deregistering all the collectio                                                   ns"}
mongo_1    | {"t":{"$date":"2021-03-20T22:32:20.073+00:00"},"s":"I",  "c":"STORA                                                   GE",  "id":22261,   "ctx":"SignalHandler","msg":"Timestamp monitor shutting down                                                   "}
mongo_1    | {"t":{"$date":"2021-03-20T22:32:20.073+00:00"},"s":"I",  "c":"STORA                                                   GE",  "id":22317,   "ctx":"SignalHandler","msg":"WiredTigerKVEngine shutting dow                                                   n"}
mongo_1    | {"t":{"$date":"2021-03-20T22:32:20.073+00:00"},"s":"I",  "c":"STORA                                                   GE",  "id":22318,   "ctx":"SignalHandler","msg":"Shutting down session sweeper t                                                   hread"}
mongo_1    | {"t":{"$date":"2021-03-20T22:32:20.073+00:00"},"s":"I",  "c":"STORA                                                   GE",  "id":22319,   "ctx":"SignalHandler","msg":"Finished shutting down session                                                    sweeper thread"}
mongo_1    | {"t":{"$date":"2021-03-20T22:32:20.073+00:00"},"s":"I",  "c":"STORA                                                   GE",  "id":22320,   "ctx":"SignalHandler","msg":"Shutting down journal flusher t                                                   hread"}
mongo_1    | {"t":{"$date":"2021-03-20T22:32:20.073+00:00"},"s":"I",  "c":"STORA                                                   GE",  "id":22321,   "ctx":"SignalHandler","msg":"Finished shutting down journal                                                    flusher thread"}
mongo_1    | {"t":{"$date":"2021-03-20T22:32:20.073+00:00"},"s":"I",  "c":"STORA                                                   GE",  "id":22322,   "ctx":"SignalHandler","msg":"Shutting down checkpoint thread                                                   "}
mongo_1    | {"t":{"$date":"2021-03-20T22:32:20.074+00:00"},"s":"I",  "c":"STORA                                                   GE",  "id":22323,   "ctx":"SignalHandler","msg":"Finished shutting down checkpoi                                                   nt thread"}
mongo_1    | {"t":{"$date":"2021-03-20T22:32:20.074+00:00"},"s":"I",  "c":"STORA                                                   GE",  "id":4795902, "ctx":"SignalHandler","msg":"Closing WiredTiger","attr":{"cl                                                   oseConfig":"leak_memory=true,"}}
mongo_1    | {"t":{"$date":"2021-03-20T22:32:20.124+00:00"},"s":"I",  "c":"STORA                                                   GE",  "id":4795901, "ctx":"SignalHandler","msg":"WiredTiger closed","attr":{"dur                                                   ationMillis":50}}
mongo_1    | {"t":{"$date":"2021-03-20T22:32:20.124+00:00"},"s":"I",  "c":"STORA                                                   GE",  "id":22279,   "ctx":"SignalHandler","msg":"shutdown: removing fs lock..."}
mongo_1    | {"t":{"$date":"2021-03-20T22:32:20.125+00:00"},"s":"I",  "c":"-",                                                           "id":4784931, "ctx":"SignalHandler","msg":"Dropping the scope cache for sh                                                   utdown"}
mongo_1    | {"t":{"$date":"2021-03-20T22:32:20.125+00:00"},"s":"I",  "c":"FTDC"                                                   ,     "id":4784926, "ctx":"SignalHandler","msg":"Shutting down full-time data ca                                                   pture"}
mongo_1    | {"t":{"$date":"2021-03-20T22:32:20.125+00:00"},"s":"I",  "c":"FTDC"                                                   ,     "id":20626,   "ctx":"SignalHandler","msg":"Shutting down full-time diagnos                                                   tic data capture"}
mongo_1    | {"t":{"$date":"2021-03-20T22:32:20.127+00:00"},"s":"I",  "c":"CONTR                                                   OL",  "id":20565,   "ctx":"SignalHandler","msg":"Now exiting"}
mongo_1    | {"t":{"$date":"2021-03-20T22:32:20.128+00:00"},"s":"I",  "c":"CONTR                                                   OL",  "id":23138,   "ctx":"SignalHandler","msg":"Shutting down","attr":{"exitCod                                                   e":0}}
mongo_1    | {"t":{"$date":"2021-03-20T22:33:32.916+00:00"},"s":"I",  "c":"CONTR                                                   OL",  "id":23285,   "ctx":"main","msg":"Automatically disabling TLS 1.0, to forc                                                   e-enable TLS 1.0 specify --sslDisabledProtocols 'none'"}
mongo_1    | {"t":{"$date":"2021-03-20T22:33:32.949+00:00"},"s":"W",  "c":"ASIO"                                                   ,     "id":22601,   "ctx":"main","msg":"No TransportLayer configured during Netw                                                   orkInterface startup"}
mongo_1    | {"t":{"$date":"2021-03-20T22:33:32.950+00:00"},"s":"I",  "c":"NETWO                                                   RK",  "id":4648601, "ctx":"main","msg":"Implicit TCP FastOpen unavailable. If TC                                                   P FastOpen is required, set tcpFastOpenServer, tcpFastOpenClient, and tcpFastOpe                                                   nQueueSize."}
mongo_1    | {"t":{"$date":"2021-03-20T22:33:32.957+00:00"},"s":"I",  "c":"STORA                                                   GE",  "id":4615611, "ctx":"initandlisten","msg":"MongoDB starting","attr":{"pid"                                                   :1,"port":27017,"dbPath":"/data/db","architecture":"64-bit","host":"1bd705aa4d16                                                   "}}
mongo_1    | {"t":{"$date":"2021-03-20T22:33:32.957+00:00"},"s":"I",  "c":"CONTR                                                   OL",  "id":23403,   "ctx":"initandlisten","msg":"Build Info","attr":{"buildInfo"                                                   :{"version":"4.4.4","gitVersion":"8db30a63db1a9d84bdcad0c83369623f708e0397","ope                                                   nSSLVersion":"OpenSSL 1.1.1  11 Sep 2018","modules":[],"allocator":"tcmalloc","e                                                   nvironment":{"distmod":"ubuntu1804","distarch":"x86_64","target_arch":"x86_64"}}                                                   }}
mongo_1    | {"t":{"$date":"2021-03-20T22:33:32.957+00:00"},"s":"I",  "c":"CONTR                                                   OL",  "id":51765,   "ctx":"initandlisten","msg":"Operating System","attr":{"os":                                                   {"name":"Ubuntu","version":"18.04"}}}
mongo_1    | {"t":{"$date":"2021-03-20T22:33:32.957+00:00"},"s":"I",  "c":"CONTR                                                   OL",  "id":21951,   "ctx":"initandlisten","msg":"Options set by command line","a                                                   ttr":{"options":{"net":{"bindIp":"*"}}}}
mongo_1    | {"t":{"$date":"2021-03-20T22:33:32.988+00:00"},"s":"I",  "c":"STORA                                                   GE",  "id":22270,   "ctx":"initandlisten","msg":"Storage engine to use detected                                                    by data files","attr":{"dbpath":"/data/db","storageEngine":"wiredTiger"}}
mongo_1    | {"t":{"$date":"2021-03-20T22:33:32.988+00:00"},"s":"I",  "c":"STORA                                                   GE",  "id":22297,   "ctx":"initandlisten","msg":"Using the XFS filesystem is str                                                   ongly recommended with the WiredTiger storage engine. See http://dochub.mongodb.                                                   org/core/prodnotes-filesystem","tags":["startupWarnings"]}
mongo_1    | {"t":{"$date":"2021-03-20T22:33:32.988+00:00"},"s":"I",  "c":"STORA                                                   GE",  "id":22315,   "ctx":"initandlisten","msg":"Opening WiredTiger","attr":{"co                                                   nfig":"create,cache_size=512M,session_max=33000,eviction=(threads_min=4,threads_                                                   max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=j                                                   ournal,compressor=snappy),file_manager=(close_idle_time=100000,close_scan_interv                                                   al=10,close_handle_minimum=250),statistics_log=(wait=0),verbose=[recovery_progre                                                   ss,checkpoint_progress,compact_progress],"}}
mongo_1    | {"t":{"$date":"2021-03-20T22:33:35.325+00:00"},"s":"I",  "c":"STORA                                                   GE",  "id":22430,   "ctx":"initandlisten","msg":"WiredTiger message","attr":{"me                                                   ssage":"[1616279615:325309][1:0x7feb93b03ac0], txn-recover: [WT_VERB_RECOVERY_PR                                                   OGRESS] Recovering log 5 through 6"}}
mongo_1    | {"t":{"$date":"2021-03-20T22:33:35.529+00:00"},"s":"I",  "c":"STORA                                                   GE",  "id":22430,   "ctx":"initandlisten","msg":"WiredTiger message","attr":{"me                                                   ssage":"[1616279615:529383][1:0x7feb93b03ac0], txn-recover: [WT_VERB_RECOVERY_PR                                                   OGRESS] Recovering log 6 through 6"}}
mongo_1    | {"t":{"$date":"2021-03-20T22:33:35.658+00:00"},"s":"I",  "c":"STORA                                                   GE",  "id":22430,   "ctx":"initandlisten","msg":"WiredTiger message","attr":{"me                                                   ssage":"[1616279615:658521][1:0x7feb93b03ac0], txn-recover: [WT_VERB_RECOVERY |                                                    WT_VERB_RECOVERY_PROGRESS] Main recovery loop: starting at 5/5120 to 6/256"}}
mongo_1    | {"t":{"$date":"2021-03-20T22:33:35.812+00:00"},"s":"I",  "c":"STORA                                                   GE",  "id":22430,   "ctx":"initandlisten","msg":"WiredTiger message","attr":{"me                                                   ssage":"[1616279615:812836][1:0x7feb93b03ac0], txn-recover: [WT_VERB_RECOVERY_PR                                                   OGRESS] Recovering log 5 through 6"}}
mongo_1    | {"t":{"$date":"2021-03-20T22:33:35.915+00:00"},"s":"I",  "c":"STORA                                                   GE",  "id":22430,   "ctx":"initandlisten","msg":"WiredTiger message","attr":{"me                                                   ssage":"[1616279615:915462][1:0x7feb93b03ac0], txn-recover: [WT_VERB_RECOVERY_PR                                                   OGRESS] Recovering log 6 through 6"}}
mongo_1    | {"t":{"$date":"2021-03-20T22:33:35.997+00:00"},"s":"I",  "c":"STORA                                                   GE",  "id":22430,   "ctx":"initandlisten","msg":"WiredTiger message","attr":{"me                                                   ssage":"[1616279615:997685][1:0x7feb93b03ac0], txn-recover: [WT_VERB_RECOVERY |                                                    WT_VERB_RECOVERY_PROGRESS] Set global recovery timestamp: (0, 0)"}}
mongo_1    | {"t":{"$date":"2021-03-20T22:33:35.997+00:00"},"s":"I",  "c":"STORA                                                   GE",  "id":22430,   "ctx":"initandlisten","msg":"WiredTiger message","attr":{"me                                                   ssage":"[1616279615:997910][1:0x7feb93b03ac0], txn-recover: [WT_VERB_RECOVERY |                                                    WT_VERB_RECOVERY_PROGRESS] Set global oldest timestamp: (0, 0)"}}
mongo_1    | {"t":{"$date":"2021-03-20T22:33:36.087+00:00"},"s":"I",  "c":"STORA                                                   GE",  "id":4795906, "ctx":"initandlisten","msg":"WiredTiger opened","attr":{"dur                                                   ationMillis":3099}}
mongo_1    | {"t":{"$date":"2021-03-20T22:33:36.087+00:00"},"s":"I",  "c":"RECOV                                                   ERY", "id":23987,   "ctx":"initandlisten","msg":"WiredTiger recoveryTimestamp","                                                   attr":{"recoveryTimestamp":{"$timestamp":{"t":0,"i":0}}}}
mongo_1    | {"t":{"$date":"2021-03-20T22:33:36.089+00:00"},"s":"I",  "c":"STORA                                                   GE",  "id":4366408, "ctx":"initandlisten","msg":"No table logging settings modif                                                   ications are required for existing WiredTiger tables","attr":{"loggingEnabled":t                                                   rue}}
mongo_1    | {"t":{"$date":"2021-03-20T22:33:36.092+00:00"},"s":"I",  "c":"STORA                                                   GE",  "id":22262,   "ctx":"initandlisten","msg":"Timestamp monitor starting"}
mongo_1    | {"t":{"$date":"2021-03-20T22:33:36.093+00:00"},"s":"W",  "c":"CONTR                                                   OL",  "id":22120,   "ctx":"initandlisten","msg":"Access control is not enabled f                                                   or the database. Read and write access to data and configuration is unrestricted                                                   ","tags":["startupWarnings"]}
mongo_1    | {"t":{"$date":"2021-03-20T22:33:36.093+00:00"},"s":"I",  "c":"CONTR                                                   OL",  "id":22161,   "ctx":"initandlisten","msg":"You are running in OpenVZ which                                                    can cause issues on versions of RHEL older than RHEL6","tags":["startupWarnings                                                   "]}
mongo_1    | {"t":{"$date":"2021-03-20T22:33:36.107+00:00"},"s":"I",  "c":"STORA                                                   GE",  "id":20536,   "ctx":"initandlisten","msg":"Flow Control is enabled on this                                                    deployment"}
mongo_1    | {"t":{"$date":"2021-03-20T22:33:36.114+00:00"},"s":"I",  "c":"FTDC"                                                   ,     "id":20625,   "ctx":"initandlisten","msg":"Initializing full-time diagnost                                                   ic data capture","attr":{"dataDirectory":"/data/db/diagnostic.data"}}
mongo_1    | {"t":{"$date":"2021-03-20T22:33:36.119+00:00"},"s":"I",  "c":"NETWO                                                   RK",  "id":23015,   "ctx":"listener","msg":"Listening on","attr":{"address":"/tm                                                   p/mongodb-27017.sock"}}
mongo_1    | {"t":{"$date":"2021-03-20T22:33:36.119+00:00"},"s":"I",  "c":"NETWO                                                   RK",  "id":23015,   "ctx":"listener","msg":"Listening on","attr":{"address":"0.0                                                   .0.0"}}
mongo_1    | {"t":{"$date":"2021-03-20T22:33:36.119+00:00"},"s":"I",  "c":"NETWO                                                   RK",  "id":23016,   "ctx":"listener","msg":"Waiting for connections","attr":{"po                                                   rt":27017,"ssl":"off"}}
mongo_1    | {"t":{"$date":"2021-03-20T22:33:36.255+00:00"},"s":"I",  "c":"NETWO                                                   RK",  "id":22943,   "ctx":"listener","msg":"Connection accepted","attr":{"remote                                                   ":"172.18.0.4:56834","connectionId":1,"connectionCount":1}}
mongo_1    | {"t":{"$date":"2021-03-20T22:33:36.261+00:00"},"s":"I",  "c":"NETWO                                                   RK",  "id":51800,   "ctx":"conn1","msg":"client metadata","attr":{"remote":"172.                                                   18.0.4:56834","client":"conn1","doc":{"driver":{"name":"mongo-go-driver","versio                                                   n":"v1.5.0"},"os":{"type":"linux","architecture":"amd64"},"platform":"go1.13.15"                                                   }}}
mongo_1    | {"t":{"$date":"2021-03-20T22:33:36.262+00:00"},"s":"I",  "c":"NETWO                                                   RK",  "id":22943,   "ctx":"listener","msg":"Connection accepted","attr":{"remote                                                   ":"172.18.0.4:56836","connectionId":2,"connectionCount":2}}
mongo_1    | {"t":{"$date":"2021-03-20T22:33:36.262+00:00"},"s":"I",  "c":"NETWO                                                   RK",  "id":51800,   "ctx":"conn2","msg":"client metadata","attr":{"remote":"172.                                                   18.0.4:56836","client":"conn2","doc":{"driver":{"name":"mongo-go-driver","versio                                                   n":"v1.5.0"},"os":{"type":"linux","architecture":"amd64"},"platform":"go1.13.15"                                                   }}}
mongo_1    | {"t":{"$date":"2021-03-20T22:33:43.744+00:00"},"s":"I",  "c":"NETWO                                                   RK",  "id":22943,   "ctx":"listener","msg":"Connection accepted","attr":{"remote                                                   ":"172.18.0.4:57198","connectionId":3,"connectionCount":3}}
mongo_1    | {"t":{"$date":"2021-03-20T22:33:43.745+00:00"},"s":"I",  "c":"NETWO                                                   RK",  "id":51800,   "ctx":"conn3","msg":"client metadata","attr":{"remote":"172.                                                   18.0.4:57198","client":"conn3","doc":{"driver":{"name":"mongo-go-driver","versio                                                   n":"v1.5.0"},"os":{"type":"linux","architecture":"amd64"},"platform":"go1.13.15"                                                   }}}
mongo_1    | {"t":{"$date":"2021-03-20T22:33:51.228+00:00"},"s":"I",  "c":"-",                                                           "id":20883,   "ctx":"conn1","msg":"Interrupted operation as its client dis                                                   connected","attr":{"opId":148}}
mongo_1    | {"t":{"$date":"2021-03-20T22:33:51.228+00:00"},"s":"I",  "c":"NETWO                                                   RK",  "id":22944,   "ctx":"conn2","msg":"Connection ended","attr":{"remote":"172                                                   .18.0.4:56836","connectionId":2,"connectionCount":2}}
mongo_1    | {"t":{"$date":"2021-03-20T22:33:51.228+00:00"},"s":"I",  "c":"NETWO                                                   RK",  "id":22944,   "ctx":"conn3","msg":"Connection ended","attr":{"remote":"172                                                   .18.0.4:57198","connectionId":3,"connectionCount":1}}
mongo_1    | {"t":{"$date":"2021-03-20T22:33:51.240+00:00"},"s":"I",  "c":"NETWO                                                   RK",  "id":22944,   "ctx":"conn1","msg":"Connection ended","attr":{"remote":"172                                                   .18.0.4:56834","connectionId":1,"connectionCount":0}}
mongo_1    | {"t":{"$date":"2021-03-20T22:33:51.596+00:00"},"s":"I",  "c":"CONTR                                                   OL",  "id":23377,   "ctx":"SignalHandler","msg":"Received signal","attr":{"signa                                                   l":15,"error":"Terminated"}}
mongo_1    | {"t":{"$date":"2021-03-20T22:33:51.596+00:00"},"s":"I",  "c":"CONTR                                                   OL",  "id":23378,   "ctx":"SignalHandler","msg":"Signal was sent by kill(2)","at                                                   tr":{"pid":0,"uid":0}}
mongo_1    | {"t":{"$date":"2021-03-20T22:33:51.596+00:00"},"s":"I",  "c":"CONTR                                                   OL",  "id":23381,   "ctx":"SignalHandler","msg":"will terminate after current cm                                                   d ends"}
mongo_1    | {"t":{"$date":"2021-03-20T22:33:51.596+00:00"},"s":"I",  "c":"REPL"                                                   ,     "id":4784900, "ctx":"SignalHandler","msg":"Stepping down the ReplicationCo                                                   ordinator for shutdown","attr":{"waitTimeMillis":10000}}
mongo_1    | {"t":{"$date":"2021-03-20T22:33:51.598+00:00"},"s":"I",  "c":"COMMA                                                   ND",  "id":4784901, "ctx":"SignalHandler","msg":"Shutting down the MirrorMaestro                                                   "}
mongo_1    | {"t":{"$date":"2021-03-20T22:33:51.598+00:00"},"s":"I",  "c":"SHARD                                                   ING", "id":4784902, "ctx":"SignalHandler","msg":"Shutting down the WaitForMajori                                                   tyService"}
mongo_1    | {"t":{"$date":"2021-03-20T22:33:51.602+00:00"},"s":"I",  "c":"CONTR                                                   OL",  "id":4784903, "ctx":"SignalHandler","msg":"Shutting down the LogicalSessio                                                   nCache"}
mongo_1    | {"t":{"$date":"2021-03-20T22:33:51.602+00:00"},"s":"I",  "c":"NETWO                                                   RK",  "id":20562,   "ctx":"SignalHandler","msg":"Shutdown: going to close listen                                                   ing sockets"}
mongo_1    | {"t":{"$date":"2021-03-20T22:33:51.606+00:00"},"s":"I",  "c":"NETWO                                                   RK",  "id":23017,   "ctx":"listener","msg":"removing socket file","attr":{"path"                                                   :"/tmp/mongodb-27017.sock"}}
mongo_1    | {"t":{"$date":"2021-03-20T22:33:51.606+00:00"},"s":"I",  "c":"NETWO                                                   RK",  "id":4784905, "ctx":"SignalHandler","msg":"Shutting down the global connec                                                   tion pool"}
mongo_1    | {"t":{"$date":"2021-03-20T22:33:51.606+00:00"},"s":"I",  "c":"STORA                                                   GE",  "id":4784906, "ctx":"SignalHandler","msg":"Shutting down the FlowControlTi                                                   cketholder"}
mongo_1    | {"t":{"$date":"2021-03-20T22:33:51.606+00:00"},"s":"I",  "c":"-",                                                           "id":20520,   "ctx":"SignalHandler","msg":"Stopping further Flow Control t                                                   icket acquisitions."}
mongo_1    | {"t":{"$date":"2021-03-20T22:33:51.606+00:00"},"s":"I",  "c":"STORA                                                   GE",  "id":4784908, "ctx":"SignalHandler","msg":"Shutting down the PeriodicThrea                                                   dToAbortExpiredTransactions"}
mongo_1    | {"t":{"$date":"2021-03-20T22:33:51.607+00:00"},"s":"I",  "c":"STORA                                                   GE",  "id":4784934, "ctx":"SignalHandler","msg":"Shutting down the PeriodicThrea                                                   dToDecreaseSnapshotHistoryCachePressure"}
mongo_1    | {"t":{"$date":"2021-03-20T22:33:51.607+00:00"},"s":"I",  "c":"REPL"                                                   ,     "id":4784909, "ctx":"SignalHandler","msg":"Shutting down the ReplicationCo                                                   ordinator"}
mongo_1    | {"t":{"$date":"2021-03-20T22:33:51.607+00:00"},"s":"I",  "c":"SHARD                                                   ING", "id":4784910, "ctx":"SignalHandler","msg":"Shutting down the ShardingIniti                                                   alizationMongoD"}
mongo_1    | {"t":{"$date":"2021-03-20T22:33:51.607+00:00"},"s":"I",  "c":"REPL"                                                   ,     "id":4784911, "ctx":"SignalHandler","msg":"Enqueuing the ReplicationStateT                                                   ransitionLock for shutdown"}
mongo_1    | {"t":{"$date":"2021-03-20T22:33:51.607+00:00"},"s":"I",  "c":"-",                                                           "id":4784912, "ctx":"SignalHandler","msg":"Killing all operations for shut                                                   down"}
mongo_1    | {"t":{"$date":"2021-03-20T22:33:51.607+00:00"},"s":"I",  "c":"-",                                                           "id":4695300, "ctx":"SignalHandler","msg":"Interrupted all currently runni                                                   ng operations","attr":{"opsKilled":3}}
mongo_1    | {"t":{"$date":"2021-03-20T22:33:51.607+00:00"},"s":"I",  "c":"COMMA                                                   ND",  "id":4784913, "ctx":"SignalHandler","msg":"Shutting down all open transact                                                   ions"}
mongo_1    | {"t":{"$date":"2021-03-20T22:33:51.607+00:00"},"s":"I",  "c":"REPL"                                                   ,     "id":4784914, "ctx":"SignalHandler","msg":"Acquiring the ReplicationStateT                                                   ransitionLock for shutdown"}
mongo_1    | {"t":{"$date":"2021-03-20T22:33:51.607+00:00"},"s":"I",  "c":"INDEX                                                   ",    "id":4784915, "ctx":"SignalHandler","msg":"Shutting down the IndexBuildsCo                                                   ordinator"}
mongo_1    | {"t":{"$date":"2021-03-20T22:33:51.607+00:00"},"s":"I",  "c":"REPL"                                                   ,     "id":4784916, "ctx":"SignalHandler","msg":"Reacquiring the ReplicationStat                                                   eTransitionLock for shutdown"}
mongo_1    | {"t":{"$date":"2021-03-20T22:33:51.607+00:00"},"s":"I",  "c":"REPL"                                                   ,     "id":4784917, "ctx":"SignalHandler","msg":"Attempting to mark clean shutdo                                                   wn"}
mongo_1    | {"t":{"$date":"2021-03-20T22:33:51.607+00:00"},"s":"I",  "c":"NETWO                                                   RK",  "id":4784918, "ctx":"SignalHandler","msg":"Shutting down the ReplicaSetMon                                                   itor"}
mongo_1    | {"t":{"$date":"2021-03-20T22:33:51.607+00:00"},"s":"I",  "c":"SHARD                                                   ING", "id":4784921, "ctx":"SignalHandler","msg":"Shutting down the MigrationUtil                                                   Executor"}
mongo_1    | {"t":{"$date":"2021-03-20T22:33:51.607+00:00"},"s":"I",  "c":"CONTR                                                   OL",  "id":4784925, "ctx":"SignalHandler","msg":"Shutting down free monitoring"}
mongo_1    | {"t":{"$date":"2021-03-20T22:33:51.607+00:00"},"s":"I",  "c":"CONTR                                                   OL",  "id":20609,   "ctx":"SignalHandler","msg":"Shutting down free monitoring"}
mongo_1    | {"t":{"$date":"2021-03-20T22:33:51.607+00:00"},"s":"I",  "c":"STORA                                                   GE",  "id":4784927, "ctx":"SignalHandler","msg":"Shutting down the HealthLog"}
mongo_1    | {"t":{"$date":"2021-03-20T22:33:51.607+00:00"},"s":"I",  "c":"STORA                                                   GE",  "id":4784929, "ctx":"SignalHandler","msg":"Acquiring the global lock for s                                                   hutdown"}
mongo_1    | {"t":{"$date":"2021-03-20T22:33:51.607+00:00"},"s":"I",  "c":"STORA                                                   GE",  "id":4784930, "ctx":"SignalHandler","msg":"Shutting down the storage engin                                                   e"}
mongo_1    | {"t":{"$date":"2021-03-20T22:33:51.607+00:00"},"s":"I",  "c":"STORA                                                   GE",  "id":20282,   "ctx":"SignalHandler","msg":"Deregistering all the collectio                                                   ns"}
mongo_1    | {"t":{"$date":"2021-03-20T22:33:51.607+00:00"},"s":"I",  "c":"STORA                                                   GE",  "id":22261,   "ctx":"SignalHandler","msg":"Timestamp monitor shutting down                                                   "}
mongo_1    | {"t":{"$date":"2021-03-20T22:33:51.607+00:00"},"s":"I",  "c":"STORA                                                   GE",  "id":22317,   "ctx":"SignalHandler","msg":"WiredTigerKVEngine shutting dow                                                   n"}
mongo_1    | {"t":{"$date":"2021-03-20T22:33:51.607+00:00"},"s":"I",  "c":"STORA                                                   GE",  "id":22318,   "ctx":"SignalHandler","msg":"Shutting down session sweeper t                                                   hread"}
mongo_1    | {"t":{"$date":"2021-03-20T22:33:51.608+00:00"},"s":"I",  "c":"STORA                                                   GE",  "id":22319,   "ctx":"SignalHandler","msg":"Finished shutting down session                                                    sweeper thread"}
mongo_1    | {"t":{"$date":"2021-03-20T22:33:51.608+00:00"},"s":"I",  "c":"STORA                                                   GE",  "id":22320,   "ctx":"SignalHandler","msg":"Shutting down journal flusher t                                                   hread"}
mongo_1    | {"t":{"$date":"2021-03-20T22:33:51.608+00:00"},"s":"I",  "c":"STORA                                                   GE",  "id":22321,   "ctx":"SignalHandler","msg":"Finished shutting down journal                                                    flusher thread"}
mongo_1    | {"t":{"$date":"2021-03-20T22:33:51.608+00:00"},"s":"I",  "c":"STORA                                                   GE",  "id":22322,   "ctx":"SignalHandler","msg":"Shutting down checkpoint thread                                                   "}
mongo_1    | {"t":{"$date":"2021-03-20T22:33:51.608+00:00"},"s":"I",  "c":"STORA                                                   GE",  "id":22323,   "ctx":"SignalHandler","msg":"Finished shutting down checkpoi                                                   nt thread"}
mongo_1    | {"t":{"$date":"2021-03-20T22:33:51.608+00:00"},"s":"I",  "c":"STORA                                                   GE",  "id":4795902, "ctx":"SignalHandler","msg":"Closing WiredTiger","attr":{"cl                                                   oseConfig":"leak_memory=true,"}}
mongo_1    | {"t":{"$date":"2021-03-20T22:33:51.641+00:00"},"s":"I",  "c":"STORA                                                   GE",  "id":4795901, "ctx":"SignalHandler","msg":"WiredTiger closed","attr":{"dur                                                   ationMillis":33}}
mongo_1    | {"t":{"$date":"2021-03-20T22:33:51.642+00:00"},"s":"I",  "c":"STORA                                                   GE",  "id":22279,   "ctx":"SignalHandler","msg":"shutdown: removing fs lock..."}
mongo_1    | {"t":{"$date":"2021-03-20T22:33:51.642+00:00"},"s":"I",  "c":"-",                                                           "id":4784931, "ctx":"SignalHandler","msg":"Dropping the scope cache for sh                                                   utdown"}
mongo_1    | {"t":{"$date":"2021-03-20T22:33:51.642+00:00"},"s":"I",  "c":"FTDC"                                                   ,     "id":4784926, "ctx":"SignalHandler","msg":"Shutting down full-time data ca                                                   pture"}
mongo_1    | {"t":{"$date":"2021-03-20T22:33:51.642+00:00"},"s":"I",  "c":"FTDC"                                                   ,     "id":20626,   "ctx":"SignalHandler","msg":"Shutting down full-time diagnos                                                   tic data capture"}
mongo_1    | {"t":{"$date":"2021-03-20T22:33:51.644+00:00"},"s":"I",  "c":"CONTR                                                   OL",  "id":20565,   "ctx":"SignalHandler","msg":"Now exiting"}
mongo_1    | {"t":{"$date":"2021-03-20T22:33:51.644+00:00"},"s":"I",  "c":"CONTR                                                   OL",  "id":23138,   "ctx":"SignalHandler","msg":"Shutting down","attr":{"exitCod                                                   e":0}}
mongo_1    | {"t":{"$date":"2021-03-20T22:40:42.387+00:00"},"s":"I",  "c":"CONTR                                                   OL",  "id":23285,   "ctx":"main","msg":"Automatically disabling TLS 1.0, to forc                                                   e-enable TLS 1.0 specify --sslDisabledProtocols 'none'"}
mongo_1    | {"t":{"$date":"2021-03-20T22:40:42.409+00:00"},"s":"W",  "c":"ASIO"                                                   ,     "id":22601,   "ctx":"main","msg":"No TransportLayer configured during Netw                                                   orkInterface startup"}
mongo_1    | {"t":{"$date":"2021-03-20T22:40:42.410+00:00"},"s":"I",  "c":"NETWO                                                   RK",  "id":4648601, "ctx":"main","msg":"Implicit TCP FastOpen unavailable. If TC                                                   P FastOpen is required, set tcpFastOpenServer, tcpFastOpenClient, and tcpFastOpe                                                   nQueueSize."}
mongo_1    | {"t":{"$date":"2021-03-20T22:40:42.411+00:00"},"s":"I",  "c":"STORA                                                   GE",  "id":4615611, "ctx":"initandlisten","msg":"MongoDB starting","attr":{"pid"                                                   :1,"port":27017,"dbPath":"/data/db","architecture":"64-bit","host":"1bd705aa4d16                                                   "}}
mongo_1    | {"t":{"$date":"2021-03-20T22:40:42.411+00:00"},"s":"I",  "c":"CONTR                                                   OL",  "id":23403,   "ctx":"initandlisten","msg":"Build Info","attr":{"buildInfo"                                                   :{"version":"4.4.4","gitVersion":"8db30a63db1a9d84bdcad0c83369623f708e0397","ope                                                   nSSLVersion":"OpenSSL 1.1.1  11 Sep 2018","modules":[],"allocator":"tcmalloc","e                                                   nvironment":{"distmod":"ubuntu1804","distarch":"x86_64","target_arch":"x86_64"}}                                                   }}
mongo_1    | {"t":{"$date":"2021-03-20T22:40:42.411+00:00"},"s":"I",  "c":"CONTR                                                   OL",  "id":51765,   "ctx":"initandlisten","msg":"Operating System","attr":{"os":                                                   {"name":"Ubuntu","version":"18.04"}}}
mongo_1    | {"t":{"$date":"2021-03-20T22:40:42.411+00:00"},"s":"I",  "c":"CONTR                                                   OL",  "id":21951,   "ctx":"initandlisten","msg":"Options set by command line","a                                                   ttr":{"options":{"net":{"bindIp":"*"}}}}
mongo_1    | {"t":{"$date":"2021-03-20T22:40:42.418+00:00"},"s":"I",  "c":"STORA                                                   GE",  "id":22270,   "ctx":"initandlisten","msg":"Storage engine to use detected                                                    by data files","attr":{"dbpath":"/data/db","storageEngine":"wiredTiger"}}
mongo_1    | {"t":{"$date":"2021-03-20T22:40:42.418+00:00"},"s":"I",  "c":"STORA                                                   GE",  "id":22297,   "ctx":"initandlisten","msg":"Using the XFS filesystem is str                                                   ongly recommended with the WiredTiger storage engine. See http://dochub.mongodb.                                                   org/core/prodnotes-filesystem","tags":["startupWarnings"]}
mongo_1    | {"t":{"$date":"2021-03-20T22:40:42.418+00:00"},"s":"I",  "c":"STORA                                                   GE",  "id":22315,   "ctx":"initandlisten","msg":"Opening WiredTiger","attr":{"co                                                   nfig":"create,cache_size=512M,session_max=33000,eviction=(threads_min=4,threads_                                                   max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=j                                                   ournal,compressor=snappy),file_manager=(close_idle_time=100000,close_scan_interv                                                   al=10,close_handle_minimum=250),statistics_log=(wait=0),verbose=[recovery_progre                                                   ss,checkpoint_progress,compact_progress],"}}
mongo_1    | {"t":{"$date":"2021-03-20T22:40:43.734+00:00"},"s":"I",  "c":"STORA                                                   GE",  "id":22430,   "ctx":"initandlisten","msg":"WiredTiger message","attr":{"me                                                   ssage":"[1616280043:734021][1:0x7fce739e4ac0], txn-recover: [WT_VERB_RECOVERY_PR                                                   OGRESS] Recovering log 6 through 7"}}
mongo_1    | {"t":{"$date":"2021-03-20T22:40:44.144+00:00"},"s":"I",  "c":"STORA                                                   GE",  "id":22430,   "ctx":"initandlisten","msg":"WiredTiger message","attr":{"me                                                   ssage":"[1616280044:144428][1:0x7fce739e4ac0], txn-recover: [WT_VERB_RECOVERY_PR                                                   OGRESS] Recovering log 7 through 7"}}
mongo_1    | {"t":{"$date":"2021-03-20T22:40:44.609+00:00"},"s":"I",  "c":"STORA                                                   GE",  "id":22430,   "ctx":"initandlisten","msg":"WiredTiger message","attr":{"me                                                   ssage":"[1616280044:608728][1:0x7fce739e4ac0], txn-recover: [WT_VERB_RECOVERY |                                                    WT_VERB_RECOVERY_PROGRESS] Main recovery loop: starting at 6/5120 to 7/256"}}
mongo_1    | {"t":{"$date":"2021-03-20T22:40:44.748+00:00"},"s":"I",  "c":"STORA                                                   GE",  "id":22430,   "ctx":"initandlisten","msg":"WiredTiger message","attr":{"me                                                   ssage":"[1616280044:748140][1:0x7fce739e4ac0], txn-recover: [WT_VERB_RECOVERY_PR                                                   OGRESS] Recovering log 6 through 7"}}
mongo_1    | {"t":{"$date":"2021-03-20T22:40:44.853+00:00"},"s":"I",  "c":"STORA                                                   GE",  "id":22430,   "ctx":"initandlisten","msg":"WiredTiger message","attr":{"me                                                   ssage":"[1616280044:853920][1:0x7fce739e4ac0], txn-recover: [WT_VERB_RECOVERY_PR                                                   OGRESS] Recovering log 7 through 7"}}
mongo_1    | {"t":{"$date":"2021-03-20T22:40:44.943+00:00"},"s":"I",  "c":"STORA                                                   GE",  "id":22430,   "ctx":"initandlisten","msg":"WiredTiger message","attr":{"me                                                   ssage":"[1616280044:943940][1:0x7fce739e4ac0], txn-recover: [WT_VERB_RECOVERY |                                                    WT_VERB_RECOVERY_PROGRESS] Set global recovery timestamp: (0, 0)"}}
mongo_1    | {"t":{"$date":"2021-03-20T22:40:44.944+00:00"},"s":"I",  "c":"STORA                                                   GE",  "id":22430,   "ctx":"initandlisten","msg":"WiredTiger message","attr":{"me                                                   ssage":"[1616280044:944244][1:0x7fce739e4ac0], txn-recover: [WT_VERB_RECOVERY |                                                    WT_VERB_RECOVERY_PROGRESS] Set global oldest timestamp: (0, 0)"}}
mongo_1    | {"t":{"$date":"2021-03-20T22:40:45.068+00:00"},"s":"I",  "c":"STORA                                                   GE",  "id":4795906, "ctx":"initandlisten","msg":"WiredTiger opened","attr":{"dur                                                   ationMillis":2650}}
mongo_1    | {"t":{"$date":"2021-03-20T22:40:45.070+00:00"},"s":"I",  "c":"RECOV                                                   ERY", "id":23987,   "ctx":"initandlisten","msg":"WiredTiger recoveryTimestamp","                                                   attr":{"recoveryTimestamp":{"$timestamp":{"t":0,"i":0}}}}
mongo_1    | {"t":{"$date":"2021-03-20T22:40:45.095+00:00"},"s":"I",  "c":"STORA                                                   GE",  "id":4366408, "ctx":"initandlisten","msg":"No table logging settings modif                                                   ications are required for existing WiredTiger tables","attr":{"loggingEnabled":t                                                   rue}}
mongo_1    | {"t":{"$date":"2021-03-20T22:40:45.124+00:00"},"s":"I",  "c":"STORA                                                   GE",  "id":22262,   "ctx":"initandlisten","msg":"Timestamp monitor starting"}
mongo_1    | {"t":{"$date":"2021-03-20T22:40:45.129+00:00"},"s":"W",  "c":"CONTR                                                   OL",  "id":22120,   "ctx":"initandlisten","msg":"Access control is not enabled f                                                   or the database. Read and write access to data and configuration is unrestricted                                                   ","tags":["startupWarnings"]}
mongo_1    | {"t":{"$date":"2021-03-20T22:40:45.129+00:00"},"s":"I",  "c":"CONTR                                                   OL",  "id":22161,   "ctx":"initandlisten","msg":"You are running in OpenVZ which                                                    can cause issues on versions of RHEL older than RHEL6","tags":["startupWarnings                                                   "]}
mongo_1    | {"t":{"$date":"2021-03-20T22:40:45.270+00:00"},"s":"I",  "c":"STORA                                                   GE",  "id":20536,   "ctx":"initandlisten","msg":"Flow Control is enabled on this                                                    deployment"}
mongo_1    | {"t":{"$date":"2021-03-20T22:40:45.350+00:00"},"s":"I",  "c":"FTDC"                                                   ,     "id":20625,   "ctx":"initandlisten","msg":"Initializing full-time diagnost                                                   ic data capture","attr":{"dataDirectory":"/data/db/diagnostic.data"}}
mongo_1    | {"t":{"$date":"2021-03-20T22:40:45.358+00:00"},"s":"I",  "c":"NETWO                                                   RK",  "id":23015,   "ctx":"listener","msg":"Listening on","attr":{"address":"/tm                                                   p/mongodb-27017.sock"}}
mongo_1    | {"t":{"$date":"2021-03-20T22:40:45.359+00:00"},"s":"I",  "c":"NETWO                                                   RK",  "id":23015,   "ctx":"listener","msg":"Listening on","attr":{"address":"0.0                                                   .0.0"}}
mongo_1    | {"t":{"$date":"2021-03-20T22:40:45.359+00:00"},"s":"I",  "c":"NETWO                                                   RK",  "id":23016,   "ctx":"listener","msg":"Waiting for connections","attr":{"po                                                   rt":27017,"ssl":"off"}}
mongo_1    | {"t":{"$date":"2021-03-20T22:40:45.867+00:00"},"s":"I",  "c":"NETWO                                                   RK",  "id":22943,   "ctx":"listener","msg":"Connection accepted","attr":{"remote                                                   ":"172.18.0.4:50800","connectionId":1,"connectionCount":1}}
mongo_1    | {"t":{"$date":"2021-03-20T22:40:45.946+00:00"},"s":"I",  "c":"NETWO                                                   RK",  "id":51800,   "ctx":"conn1","msg":"client metadata","attr":{"remote":"172.                                                   18.0.4:50800","client":"conn1","doc":{"driver":{"name":"mongo-go-driver","versio                                                   n":"v1.5.0"},"os":{"type":"linux","architecture":"amd64"},"platform":"go1.13.15"                                                   }}}
mongo_1    | {"t":{"$date":"2021-03-20T22:40:45.948+00:00"},"s":"I",  "c":"NETWO                                                   RK",  "id":22943,   "ctx":"listener","msg":"Connection accepted","attr":{"remote                                                   ":"172.18.0.4:50808","connectionId":2,"connectionCount":2}}
mongo_1    | {"t":{"$date":"2021-03-20T22:40:45.949+00:00"},"s":"I",  "c":"NETWO                                                   RK",  "id":51800,   "ctx":"conn2","msg":"client metadata","attr":{"remote":"172.                                                   18.0.4:50808","client":"conn2","doc":{"driver":{"name":"mongo-go-driver","versio                                                   n":"v1.5.0"},"os":{"type":"linux","architecture":"amd64"},"platform":"go1.13.15"                                                   }}}
mongo_1    | {"t":{"$date":"2021-03-20T22:40:53.345+00:00"},"s":"I",  "c":"NETWO                                                   RK",  "id":22943,   "ctx":"listener","msg":"Connection accepted","attr":{"remote                                                   ":"172.18.0.4:51192","connectionId":3,"connectionCount":3}}
mongo_1    | {"t":{"$date":"2021-03-20T22:40:53.370+00:00"},"s":"I",  "c":"NETWO                                                   RK",  "id":51800,   "ctx":"conn3","msg":"client metadata","attr":{"remote":"172.                                                   18.0.4:51192","client":"conn3","doc":{"driver":{"name":"mongo-go-driver","versio                                                   n":"v1.5.0"},"os":{"type":"linux","architecture":"amd64"},"platform":"go1.13.15"                                                   }}}
mongo_1    | {"t":{"$date":"2021-03-20T22:41:06.752+00:00"},"s":"I",  "c":"-",                                                           "id":20883,   "ctx":"conn1","msg":"Interrupted operation as its client dis                                                   connected","attr":{"opId":282}}
mongo_1    | {"t":{"$date":"2021-03-20T22:41:06.752+00:00"},"s":"I",  "c":"NETWO                                                   RK",  "id":22944,   "ctx":"conn2","msg":"Connection ended","attr":{"remote":"172                                                   .18.0.4:50808","connectionId":2,"connectionCount":2}}
mongo_1    | {"t":{"$date":"2021-03-20T22:41:06.752+00:00"},"s":"I",  "c":"NETWO                                                   RK",  "id":22944,   "ctx":"conn3","msg":"Connection ended","attr":{"remote":"172                                                   .18.0.4:51192","connectionId":3,"connectionCount":1}}
mongo_1    | {"t":{"$date":"2021-03-20T22:41:06.813+00:00"},"s":"I",  "c":"NETWO                                                   RK",  "id":22944,   "ctx":"conn1","msg":"Connection ended","attr":{"remote":"172                                                   .18.0.4:50800","connectionId":1,"connectionCount":0}}
mongo_1    | {"t":{"$date":"2021-03-20T22:41:07.096+00:00"},"s":"I",  "c":"CONTR                                                   OL",  "id":23377,   "ctx":"SignalHandler","msg":"Received signal","attr":{"signa                                                   l":15,"error":"Terminated"}}
gustavosbarreto commented 3 years ago

It looks like MongoDB is up and running after upgrade process. The issue is really with the gateway service.

suegy commented 3 years ago

As mentioned before. I also do get this weird log entry in the api service which might be related?

gustavosbarreto commented 3 years ago

As mentioned before. I also do get this weird log entry in the api service which might be related?

SHELLHUB_CLOUD? It is internal to our cloud service. - don't worry about it.

Take a look here: moby/moby#40835

Looks to be on your side. Might be a maximum number of namespace, pids or other constrains.

@suegy I would suspect the issue to be on your side.

How many containers are you running on docker host? Have you tried to reboot your system or the docker daemon?

suegy commented 3 years ago

I am only running the shellhub containers as before. I did try to reboot the machine which did not change the error.

I could do a fresh install of shellhub and delete the database and all configurations but I thought it might be worth investigating why the upgrade did not work.

otavio commented 3 years ago

@suegy, indeed. Understand what causes this error is indeed valuable; however, we are stuck with the information we have at our disposal.

Please send the information @gustavosbarreto asked but please quote it accordingly. For reference: https://github.github.com/gfm/#fenced-code-blocks

We are indeed not capable of reproduce it here yet, so we must be missing something ...

suegy commented 3 years ago

Updated the information above with the correct code fencing. This contains the output for the three commands. The mongodb logs are a lot longer still but seem to be repeating.

@otavio After going through most of the configs the issue seems to be related to an automatic rule update of the firewall which blocks setting up some of the ports. After resetting the server to a previous state 2 weeks ago before the rules were pushed the upgrade works. I still have not pinpointed the specific rule as this is a standard package.