High CPU and Low throughput

[akhare22sandeep]

Hi All,

We are using Crossbar version 4.0.0.0 in production as a WAMP protocol router. We have developed Python apps that make RPC calls to systems which are also connected to the router. Until now, everything was good, but we have encountered issues since introducing file uploads. The server application makes RPC calls to the system to upload files in 128 KB chunks. We have observed that with only 20 systems uploading files, Crossbar's CPU usage reaches 100% and chokes.

We also noticed a decrease in overall throughput due to the high CPU usage (uploading around total 7 to 8 GB files in 4 hours from each server). Recently, we set validate_utf8 to false, which improved performance by 20 to 30%, but the overall file upload process is still slow due to 100% CPU utilization.

Other RPC are simple JSON structure but file upload RPC uploads binary ,videos and other types of large files.

1. What would be the cause of high CPU , is there any way we can reduce the CPU by changing the configuration
2. How can we run the crossbar on multi CPU to take advantage of 8 cores on the server . Right now its using single core
3. Can we WSS and Compression settings increase the CPU ?

Below are our crossbar settings . Any Help would be appreciated

{
  "version": 2,
  "workers": [
    {
      "type": "router",
       "realms": [
        {
          "name": "realm1",
          "roles": [
            {
              "name": "system",
              "authorizer": "com.company.auth.authorize"
            },
            {
              "name": "user",
              "authorizer": "com.company.auth.authorize"
            },
            {
              "name": "service",
              "authorizer": "com.company.auth.authorize"
            }
          ]
        }
      ],
      "transports": [
        {
          "type": "web",
          "endpoint": {
            "type": "tcp",
            "port": 8080
          },
          "paths": {
            "status": {
              "type": "resource",
              "classname": "abcd_router_inspector.create_resource",
              "extra": {
                "services": [
                  {
                    "name": "abcd-realm-auditor",
                    "url": "http://localhost:9503/api/status"
                  },
                  {
                    "name": "abcd-realm-monitor",
                    "url": "http://localhost:9504/api/status"
                  }
                ],
                "platforms": [
                    {
                      "name": "abcd-heartbeat-consumer",
                      "url": "http://sl2ionapp302:9502/api/status"
                    }
                ],
                "status_healthy": "abcd up",
                "status_unhealthy": "abcd down",
                "db_connection_successfull": "DB connection is up",
                "db_connection_unsuccessfull": "DB connection is down"
              }
            },
            "info": {
              "type": "nodeinfo"
            },
            "ws": {
              "type": "websocket",
              "auth": {
                "anonymous": {
                  "type": "dynamic",
                  "authenticator": "com.company.auth.authenticate"
                },
                "ticket": {
                  "type": "dynamic",
                  "authenticator": "com.company.auth.authenticate"
                }
              },
              "options": {
                "auto_ping_interval": 60000,
                "auto_ping_timeout": 10000,
                "auto_ping_size": 4,
                "compression": {
                  "deflate": {
                    "request_no_context_takeover": false,
                    "request_max_window_bits": 11,
                    "no_context_takeover": false,
                    "max_window_bits": 11,
                    "memory_level": 4
                  }
                }
              }
            }
          }
        }
      ],
      "components": [
        {
          "id": "auth",
          "type": "class",
          "classname": "auth.F10MSSQLAuthSession",
          "realm": "realm1",
          "role": "trusted",
          "extra": {
            "authenticate_uri": "com.company.auth.authenticate",
            "authorize_uri": "com.company.auth.authorize",
            "database": {
              "module": "pymssql",
              "server": "DbServer.dv.local",
              "port": "1517",
              "database": "abcdIONAPP",
              "user": "user",
              "password": "sometoken==",
              "autocommit": true
            },
            "validate_token_url": "http://server.local:9000/validate_token/",
            "enforce_cert_match": false,
            "cert_sub_pattern": "[\\s_-]"
          }
        }
      ]
    }
  ]
}