search

TykTechnologies / tyk

Tyk Open Source API Gateway written in Go, supporting REST, GraphQL, TCP and gRPC protocols
Other
9.62k stars 1.08k forks source link

Hybrid gateway crashes after load #1967

Closed ilijabojanovic closed 5 years ago

ilijabojanovic commented 5 years ago

Do you want to request a feature or report a bug? BUG What is the current behavior? When we send bigger traffic to salved gateway, gateway crashes after traffic What is the expected behavior? Gateway should not crash If the current behavior is a bug, please provide the steps to reproduce and if possible a minimal demo of the problem

  1. Start one or two slaved gateways
  2. Send traffic to those gateways

Gateway configuration:

{
    "listen_port": 8080,
    "secret": "352d20ee67be67f6340b4c0605b044b7",
    "node_secret": "",
    "template_path": "./templates",
    "middleware_path": "./middleware",
    "use_db_app_configs": false,
    "app_path": "./test_apps/",
        "storage": {
        "type": "redis",
        "host": "localhost",
        "port": 6380,
        "hosts": {},
        "username": "",
        "password": "",
        "database": 0,
        "optimisation_max_idle": 3000,
        "optimisation_max_active": 5000,
        "enable_cluster": false
    },
    "enable_analytics": true,
    "analytics_config": {
        "type": "rpc",
        "csv_dir": "/tmp",
        "mongo_url": "localhost",
        "mongo_db_name": "tyk_analytics",
        "mongo_collection": "tyk_analytics",
        "purge_delay": 10,
        "ignored_ips": [],
        "enable_geo_ip": true,
        "geo_ip_db_path": "./GeoLite2-City.mmdb",
        "normalise_urls": {
            "enabled": true,
            "normalise_uuids": true,
            "normalise_numbers": true,
            "custom_patterns": []
        }
    },
    "health_check": {
        "enable_health_checks": false,
        "health_check_value_timeouts": 60
    },
    "optimisations_use_async_session_write": true,
    "allow_master_keys": false,
    "policies": {
      "policy_source": "rpc",
      "policy_record_name": "tyk_policies"
    },
    "hash_keys": true,
    "suppress_redis_signal_reload": false,
    "use_sentry": false,
    "sentry_code": "",
    "enforce_org_data_age": true,
    "http_server_options": {
        "enable_websockets": true
    },
    "monitor": {
        "enable_trigger_monitors": true,
        "configuration": {
        "method": "POST",
            "target_path": "http://cloud.tyk.io/1337/tyk/webhook",
            "template_path": "templates/monitor_template.json",
            "header_map": {"x-tyk-monitor-secret": "sjdkfhjKHKJHkjsdhsufdudfhjHKIHJ1"},
            "event_timeout": 10
        },
        "global_trigger_limit": 80.0,
        "monitor_user_keys": false,
        "monitor_org_keys": true
    },
    "slave_options": {
        "use_rpc": true,
        "rpc_key": "5bd82c299b7d7c0873b203f1",
        "api_key": "3711bd96e32643ec78a6395724889de1",
        "connection_string": "localhost:9090",
        "enable_rpc_cache": true,
        "use_ssl": false,
        "ssl_insecure_skip_verify": false,
        "bind_to_slugs": true,
        "call_timeout": 5
    },
    "local_session_cache": {
        "disable_cached_session_state": false,
        "cached_session_timeout": 5,
        "cached_session_eviction": 10
    },
    "enforce_org_quotas": false,
    "experimental_process_org_off_thread": true,
    "enable_non_transactional_rate_limiter": true,
    "enable_sentinel_rate_limiter": false,
    "local_session_cache": {
        "disable_cached_session_state": false
        },
    "auth_override": {
        "force_auth_provider": true,
        "auth_provider": {
            "name": "",
            "storage_engine": "rpc",
            "meta": {}
        }
    },
    "http_server_options": {
        "skip_url_cleaning": true,
        "override_defaults": true,
        "read_timeout": 0,
        "ssl_insecure_skip_verify": true,
        "write_timeout": 0,
        "use_ssl": false,
        "use_ssl_le": false,
        "enable_websockets": true,
        "certificates": [
        {
            "domain_name": "*.tyk-test.com",
            "cert_file": "./server.crt",
            "key_file": "./server.key"
                }
        ],
        "server_name": "",
        "min_version": 0,
        "flush_interval": 0
    },
    "enable_context_vars": true,
    "hostname": "",
    "enable_api_segregation": false,
    "control_api_hostname": "",
    "enable_custom_domains": true,
    "enable_jsvm": false,
    "coprocess_options": {
        "enable_coprocess": true
    },
    "enable_bundle_downloader": true,
    "bundle_base_url": "http://192.168.1.44:8000/",
    "hide_generator_header": false,
    "event_handlers": {
        "events": {}
    },
    "pid_file_location": "./tyk-gateway.pid",
    "allow_insecure_configs": true,
    "public_key_path": "",
    "log_level": "info",
    "allow_remote_config": false,
    "enable_bundle_downloader": true,
    "service_discovery": {
        "default_cache_timeout": 20
    },
    "close_idle_connections": true,
    "enable_key_logging": false,
    "close_connections": false,
    "max_idle_connections_per_host": 400,
    "disable_dashboard_zeroconf": true,
    "uptime_tests": {
        "disable": false,
        "config": {
            "failure_trigger_sample_size": 1,
            "time_wait": 7,
            "checker_pool_size": 50,
            "enable_uptime_analytics": false
            }
    },
    "proxy_ssl_insecure_skip_verify": true
}

Error:

[Nov  6 14:07:45]  WARN Incorrect key expiry setting detected, correcting
[Nov  6 14:07:45]  WARN Incorrect key expiry setting detected, correcting
panic: reflect: reflect.Value.SetString using unaddressable value

goroutine 165 [running]:
reflect.flag.mustBeAssignable(0x98)
    /usr/local/go/src/reflect/value.go:234 +0x15c
reflect.Value.SetString(0x194f680, 0xc423ba24e0, 0x98, 0xc420015250, 0x3)
    /usr/local/go/src/reflect/value.go:1551 +0x2b
github.com/TykTechnologies/tyk/vendor/gopkg.in/vmihailenco/msgpack%2ev2.decodeStringValue(0xc42378ecc0, 0x194f680, 0xc423ba24e0, 0x98, 0x194f680, 0xc423ba24e0)
    /Users/ilijabojanovic/work/src/github.com/TykTechnologies/tyk/vendor/gopkg.in/vmihailenco/msgpack.v2/decode_string.go:56 +0x88
github.com/TykTechnologies/tyk/vendor/gopkg.in/vmihailenco/msgpack%2ev2.(*field).DecodeValue(0xc4204b9d40, 0xc42378ecc0, 0x1ac8e60, 0xc423ba24e0, 0x99, 0xc4223cbc78, 0xff0f19c5)
    /Users/ilijabojanovic/work/src/github.com/TykTechnologies/tyk/vendor/gopkg.in/vmihailenco/msgpack.v2/types.go:90 +0x82
github.com/TykTechnologies/tyk/vendor/gopkg.in/vmihailenco/msgpack%2ev2.decodeStructValue(0xc42378ecc0, 0x1ac8e60, 0xc423ba24e0, 0x99, 0x1ac8e60, 0xc423ba24e0)
    /Users/ilijabojanovic/work/src/github.com/TykTechnologies/tyk/vendor/gopkg.in/vmihailenco/msgpack.v2/decode_map.go:215 +0x15c
github.com/TykTechnologies/tyk/vendor/gopkg.in/vmihailenco/msgpack%2ev2.(*Decoder).DecodeValue(0xc42378ecc0, 0x1ac8e60, 0xc423ba24e0, 0x99, 0xc423ba24e0, 0x99)
    /Users/ilijabojanovic/work/src/github.com/TykTechnologies/tyk/vendor/gopkg.in/vmihailenco/msgpack.v2/decode.go:188 +0x8c
github.com/TykTechnologies/tyk/vendor/gopkg.in/vmihailenco/msgpack%2ev2.decodeInterfaceValue(0xc42378ecc0, 0x1985120, 0xc423547db0, 0x194, 0x1985120, 0xc423547db0)
    /Users/ilijabojanovic/work/src/github.com/TykTechnologies/tyk/vendor/gopkg.in/vmihailenco/msgpack.v2/decode_value.go:158 +0xe4
github.com/TykTechnologies/tyk/vendor/gopkg.in/vmihailenco/msgpack%2ev2.(*Decoder).DecodeValue(0xc42378ecc0, 0x1985120, 0xc423547db0, 0x194, 0xc423547db0, 0x194)
    /Users/ilijabojanovic/work/src/github.com/TykTechnologies/tyk/vendor/gopkg.in/vmihailenco/msgpack.v2/decode.go:188 +0x8c
github.com/TykTechnologies/tyk/vendor/gopkg.in/vmihailenco/msgpack%2ev2.(*Decoder).decode(0xc42378ecc0, 0x1924460, 0xc423547db0, 0x40, 0xc420343000)
    /Users/ilijabojanovic/work/src/github.com/TykTechnologies/tyk/vendor/gopkg.in/vmihailenco/msgpack.v2/decode.go:183 +0x120
github.com/TykTechnologies/tyk/vendor/gopkg.in/vmihailenco/msgpack%2ev2.(*Decoder).Decode(0xc42378ecc0, 0xc4223cbf28, 0x1, 0x1, 0x0, 0x0)
    /Users/ilijabojanovic/work/src/github.com/TykTechnologies/tyk/vendor/gopkg.in/vmihailenco/msgpack.v2/decode.go:65 +0x6f
github.com/TykTechnologies/tyk/vendor/gopkg.in/vmihailenco/msgpack%2ev2.Unmarshal(0xc422ceb180, 0x24d, 0x24d, 0xc4223cbf28, 0x1, 0x1, 0x0, 0x0)
    /Users/ilijabojanovic/work/src/github.com/TykTechnologies/tyk/vendor/gopkg.in/vmihailenco/msgpack.v2/decode.go:39 +0xcf
github.com/TykTechnologies/tyk/rpc.(*Purger).PurgeCache(0xc4223cbfc0)
    /Users/ilijabojanovic/work/src/github.com/TykTechnologies/tyk/rpc/rpc_analytics_purger.go:72 +0x228
github.com/TykTechnologies/tyk/rpc.Purger.PurgeLoop(0x1cb6740, 0xc420504160, 0x1b7c110, 0xc420158660)
    /Users/ilijabojanovic/work/src/github.com/TykTechnologies/tyk/rpc/rpc_analytics_purger.go:49 +0x42
created by main.setupGlobals.func2
    /Users/ilijabojanovic/work/src/github.com/TykTechnologies/tyk/main.go:169 +0xd2

Which versions of Tyk affected by this issue? Did this work in previous versions of Tyk? Master

buger commented 5 years ago

Should be fixed by https://github.com/TykTechnologies/tyk/pull/1974

ilijabojanovic commented 5 years ago

Verified