TykTechnologies / tyk

Tyk Open Source API Gateway written in Go, supporting REST, GraphQL, TCP and gRPC protocols
Other
9.6k stars 1.08k forks source link

panic: runtime error: invalid memory address or nil pointer dereference #1011

Closed bitsofinfo closed 7 years ago

bitsofinfo commented 7 years ago

Do you want to request a feature or report a bug?

bug

What is the current behavior?

My listen path is /my/api/1.0, strip listen path is checked

I need to whitelist GET's to /my/api/1.0?wsdl

In a GET whitelist I put in ?wsdl and get this stack trace on a request to the gateway

In a GET whitelist I put in /?wsdl throws no error but doesn't work (get forbidden on a request)

An explicit whitelist on /my/api/1.0?wsdl doesn't work either (get forbidden on a request)

2017-08-15T20:28:02.413172594Z 2017/08/15 20:28:02 http: panic serving 10.218.8.28:50319: runtime error: invalid memory address or nil pointer dereference
2017-08-15T20:28:02.413177794Z goroutine 245185 [running]:
2017-08-15T20:28:02.413181694Z net/http.(*conn).serve.func1(0xc4409f3280)
2017-08-15T20:28:02.413185594Z  /usr/local/go/src/net/http/server.go:1491 +0x12a
2017-08-15T20:28:02.413189794Z panic(0xccaae0, 0xc420010070)
2017-08-15T20:28:02.413193793Z  /usr/local/go/src/runtime/panic.go:458 +0x243
2017-08-15T20:28:02.413197893Z regexp.(*Regexp).get(0x0, 0x19)
2017-08-15T20:28:02.413202093Z  /usr/local/go/src/regexp/regexp.go:208 +0x26
2017-08-15T20:28:02.413206193Z regexp.(*Regexp).doExecute(0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xc441521e60, 0x19, 0x0, 0x0, ...)
2017-08-15T20:28:02.413210193Z  /usr/local/go/src/regexp/exec.go:416 +0x40
2017-08-15T20:28:02.413213893Z regexp.(*Regexp).MatchString(0x0, 0xc441521e60, 0x19, 0x19)
2017-08-15T20:28:02.413217693Z  /usr/local/go/src/regexp/regexp.go:416 +0x88
2017-08-15T20:28:02.413221593Z main.(*APISpec).IsURLAllowedAndIgnored(0xc43dd23d60, 0xc440a06ae0, 0x3, 0xc441521d00, 0x19, 0xc441521e20, 0x1, 0xc43fca51f8, 0x556d85, 0x20, ...)
2017-08-15T20:28:02.413225693Z  /src/github.com/TykTechnologies/tyk/api_definition_manager.go:1066 +0x108
2017-08-15T20:28:02.413235493Z main.(*APISpec).IsRequestValid(0xc43dd23d60, 0xc44026d0e0, 0xc441521e0c, 0xdbe1d3, 0x9, 0x15, 0xc441521e00)
2017-08-15T20:28:02.413239593Z  /src/github.com/TykTechnologies/tyk/api_definition_manager.go:1291 +0x17d
2017-08-15T20:28:02.413243593Z main.(*VersionCheck).ProcessRequest(0xc440166710, 0x12ba260, 0xc43fee2340, 0xc44026d0e0, 0x0, 0x0, 0x15, 0x0, 0x14)
2017-08-15T20:28:02.413247393Z  /src/github.com/TykTechnologies/tyk/middleware_version_check.go:51 +0x53
2017-08-15T20:28:02.413251293Z main.CreateMiddleware.func1.1(0x12ba260, 0xc43fee2340, 0xc44026d0e0)
2017-08-15T20:28:02.413255093Z  /src/github.com/TykTechnologies/tyk/middleware.go:76 +0x637
2017-08-15T20:28:02.413258793Z net/http.HandlerFunc.ServeHTTP(0xc440154680, 0x12ba260, 0xc43fee2340, 0xc44026d0e0)
2017-08-15T20:28:02.413262593Z  /usr/local/go/src/net/http/server.go:1726 +0x44
2017-08-15T20:28:02.413266293Z main.CreateMiddleware.func1.1(0x12ba260, 0xc43fee2340, 0xc44026d0e0)
2017-08-15T20:28:02.413270193Z  /src/github.com/TykTechnologies/tyk/middleware.go:100 +0x83c
2017-08-15T20:28:02.413273993Z net/http.HandlerFunc.ServeHTTP(0xc4401546c0, 0x12ba260, 0xc43fee2340, 0xc44026d0e0)
2017-08-15T20:28:02.413298493Z  /usr/local/go/src/net/http/server.go:1726 +0x44
2017-08-15T20:28:02.413302693Z main.CreateMiddleware.func1.1(0x12ba260, 0xc43fee2340, 0xc44026d0e0)
2017-08-15T20:28:02.413308293Z  /src/github.com/TykTechnologies/tyk/middleware.go:100 +0x83c
2017-08-15T20:28:02.413312593Z net/http.HandlerFunc.ServeHTTP(0xc440154700, 0x12ba260, 0xc43fee2340, 0xc44026d0e0)
2017-08-15T20:28:02.413316793Z  /usr/local/go/src/net/http/server.go:1726 +0x44
2017-08-15T20:28:02.413320893Z main.CreateMiddleware.func1.1(0x12ba260, 0xc43fee2340, 0xc44026d0e0)
2017-08-15T20:28:02.413324993Z  /src/github.com/TykTechnologies/tyk/middleware.go:100 +0x83c
2017-08-15T20:28:02.413329193Z net/http.HandlerFunc.ServeHTTP(0xc440154740, 0x12ba260, 0xc43fee2340, 0xc44026d0e0)
2017-08-15T20:28:02.413333193Z  /usr/local/go/src/net/http/server.go:1726 +0x44
2017-08-15T20:28:02.413337293Z github.com/TykTechnologies/tyk/vendor/github.com/gorilla/context.ClearHandler.func1(0x12ba260, 0xc43fee2340, 0xc44026d0e0)
2017-08-15T20:28:02.413341493Z  /src/github.com/TykTechnologies/tyk/vendor/github.com/gorilla/context/context.go:141 +0x8b
2017-08-15T20:28:02.413345593Z net/http.HandlerFunc.ServeHTTP(0xc440169a20, 0x12ba260, 0xc43fee2340, 0xc44026d0e0)
2017-08-15T20:28:02.413349993Z  /usr/local/go/src/net/http/server.go:1726 +0x44
2017-08-15T20:28:02.413353993Z github.com/TykTechnologies/tyk/vendor/github.com/gorilla/mux.(*Router).ServeHTTP(0xc43fa20d20, 0x12ba260, 0xc43fee2340, 0xc44026d0e0)
2017-08-15T20:28:02.413358293Z  /src/github.com/TykTechnologies/tyk/vendor/github.com/gorilla/mux/mux.go:114 +0x10d
2017-08-15T20:28:02.413362293Z net/http.(*ServeMux).ServeHTTP(0xc440262b10, 0x12ba260, 0xc43fee2340, 0xc440fb40f0)
2017-08-15T20:28:02.413369093Z  /usr/local/go/src/net/http/server.go:2022 +0x7f
2017-08-15T20:28:02.413373393Z net/http.serverHandler.ServeHTTP(0xc422dfd300, 0x12ba260, 0xc43fee2340, 0xc440fb40f0)
2017-08-15T20:28:02.413377493Z  /usr/local/go/src/net/http/server.go:2202 +0x7d
2017-08-15T20:28:02.413381493Z net/http.(*conn).serve(0xc4409f3280, 0x12bb2e0, 0xc4409b3ac0)
2017-08-15T20:28:02.413385493Z  /usr/local/go/src/net/http/server.go:1579 +0x4b7
2017-08-15T20:28:02.413389492Z created by net/http.(*Server).Serve
2017-08-15T20:28:02.413393492Z  /usr/local/go/src/net/http/server.go:2293 +0x44d

What is the expected behavior?

Should not throw such an error, if this is an invalid whitelist match at the root of a uri path, should be caught by the gateway api for configuring a whitelist?

Which versions of Tyk affected by this issue? Did this work in previous versions of Tyk?

2.3.7/1.3.7

mvdan commented 7 years ago

Thanks for the report - will look into this straight away.

Could you provide your gateway config and the API definition you're using?

bitsofinfo commented 7 years ago

below (with this conf, a GET to http://localhost:8080/my/endpoint/1.0?wsdl dumps that stack in the gateway logs)

tyk.conf

{
    "listen_port": 8080,
    "secret": "352d20ee67be67f6340b4c0605b044b7",
    "node_secret": "352d20ee67be67f6340b4c0605b044b7",
    "template_path": "/opt/tyk-gateway/templates",
    "tyk_js_path": "/opt/tyk-gateway/js/tyk.js",
    "middleware_path": "/opt/tyk-gateway/middleware",
    "use_db_app_configs": true,
    "enable_custom_domains": true,
    "db_app_conf_options": {
        "connection_string": "http://tyk_dashboard:3000",
        "node_is_segmented": false,
        "tags": ["test2"]
    },
    "app_path": "/opt/tyk-gateway/apps/",
    "storage": {
        "type": "redis",
        "host": "redis",
        "port": 6379,
        "username": "",
        "password": "",
        "database": 0,
        "optimisation_max_idle": 100
    },
    "enable_jsvm": true,
    "enable_analytics": true,
    "analytics_config": {
        "type": "mongo",
        "csv_dir": "/tmp",
        "mongo_url": "",
        "mongo_db_name": "",
        "mongo_collection": "",
        "purge_delay": -1,
        "ignored_ips": []
    },
    "health_check": {
        "enable_health_checks": true,
        "health_check_value_timeouts": 60
    },
    "optimisations_use_async_session_write": true,
    "enable_non_transactional_rate_limiter": true,
    "enable_sentinel_rate_limiter": false,
    "allow_master_keys": false,
    "policies": {
        "policy_source": "service",
        "policy_connection_string": "http://tyk_dashboard:3000",
        "policy_record_name": "tyk_policies"
    },
    "hash_keys": true,
    "close_connections": true,
    "allow_insecure_configs": true,
     "coprocess_options": {
        "enable_coprocess": true,
        "coprocess_grpc_server": ""
    },
    "enable_bundle_downloader": true,
    "bundle_base_url": "http://192.168.3.189:8888/tyk/",
    "global_session_lifetime": 100,
    "force_global_session_lifetime": false,
    "max_idle_connections_per_host": 100,
    "local_session_cache" : {
          "disable_cached_session_state": true
     },
    "proxy_ssl_insecure_skip_verify":true
}

API conf

{
    "id": "5994915c2d9623000150e269",
    "name": "/my/endpoint/1.0",
    "slug": "my/endpoint/10",
    "api_id": "7ebc2adfd4e846656a6df67ed76209ab",
    "org_id": "599490f92d962300014ef549",
    "use_keyless": false,
    "use_oauth2": false,
    "use_openid": false,
    "openid_options": {
        "providers": [],
        "segregate_by_client": false
    },
    "oauth_meta": {
        "allowed_access_types": [],
        "allowed_authorize_types": [],
        "auth_login_redirect": ""
    },
    "auth": {
        "use_param": false,
        "param_name": "",
        "use_cookie": false,
        "cookie_name": "",
        "auth_header_name": "x-internal-authorization"
    },
    "use_basic_auth": false,
    "enable_jwt": false,
    "use_standard_auth": true,
    "enable_coprocess_auth": false,
    "jwt_signing_method": "",
    "jwt_source": "",
    "jwt_identity_base_field": "",
    "jwt_client_base_field": "",
    "jwt_policy_field_name": "",
    "notifications": {
        "shared_secret": "",
        "oauth_on_keychange_url": ""
    },
    "enable_signature_checking": false,
    "hmac_allowed_clock_skew": -1,
    "base_identity_provided_by": "",
    "definition": {
        "location": "header",
        "key": "x-api-version"
    },
    "version_data": {
        "not_versioned": true,
        "versions": {
            "Default": {
                "name": "Default",
                "expires": "",
                "paths": {
                    "ignored": [],
                    "white_list": [],
                    "black_list": []
                },
                "use_extended_paths": true,
                "extended_paths": {
                    "white_list": [
                        {
                            "path": "?wsdl",
                            "method_actions": {
                                "GET": {
                                    "action": "no_action",
                                    "code": 200,
                                    "data": "",
                                    "headers": {}
                                }
                            }
                        },
                        {
                            "path": "/.*",
                            "method_actions": {
                                "POST": {
                                    "action": "no_action",
                                    "code": 200,
                                    "data": "",
                                    "headers": {}
                                }
                            }
                        }
                    ]
                },
                "global_headers": {},
                "global_headers_remove": [],
                "global_size_limit": 0,
                "override_target": ""
            }
        }
    },
    "uptime_tests": {
        "check_list": [],
        "config": {
            "expire_utime_after": 0,
            "service_discovery": {
                "use_discovery_service": false,
                "query_endpoint": "",
                "use_nested_query": false,
                "parent_data_path": "",
                "data_path": "",
                "port_data_path": "",
                "target_path": "",
                "use_target_list": false,
                "cache_timeout": 60,
                "endpoint_returns_list": false
            },
            "recheck_wait": 0
        }
    },
    "proxy": {
        "preserve_host_header": false,
        "listen_path": "/my/endpoint/1.0",
        "target_url": "http://192.168.0.230:9080/my/service",
        "strip_listen_path": true,
        "enable_load_balancing": false,
        "target_list": [],
        "check_host_against_uptime_tests": false,
        "service_discovery": {
            "use_discovery_service": false,
            "query_endpoint": "",
            "use_nested_query": false,
            "parent_data_path": "",
            "data_path": "hostname",
            "port_data_path": "port",
            "target_path": "/api-slug",
            "use_target_list": false,
            "cache_timeout": 60,
            "endpoint_returns_list": false
        }
    },
    "disable_rate_limit": false,
    "disable_quota": false,
    "custom_middleware": {
        "pre": [],
        "post": [],
        "post_key_auth": [],
        "auth_check": {
            "name": "",
            "path": "",
            "require_session": false
        },
        "response": [],
        "driver": "",
        "id_extractor": {
            "extract_from": "",
            "extract_with": "",
            "extractor_config": {}
        }
    },
    "custom_middleware_bundle": "my-bundle.zip",
    "cache_options": {
        "cache_timeout": 60,
        "enable_cache": true,
        "cache_all_safe_requests": false,
        "cache_response_codes": [],
        "enable_upstream_cache_control": false
    },
    "session_lifetime": 0,
    "active": true,
    "auth_provider": {
        "name": "",
        "storage_engine": "",
        "meta": {}
    },
    "session_provider": {
        "name": "",
        "storage_engine": "",
        "meta": null
    },
    "event_handlers": {
        "events": {}
    },
    "enable_batch_request_support": false,
    "enable_ip_whitelisting": false,
    "allowed_ips": [],
    "dont_set_quota_on_create": false,
    "expire_analytics_after": 0,
    "response_processors": [],
    "CORS": {
        "enable": false,
        "allowed_origins": [],
        "allowed_methods": [],
        "allowed_headers": [],
        "exposed_headers": [],
        "allow_credentials": false,
        "max_age": 24,
        "options_passthrough": false,
        "debug": false
    },
    "domain": "",
    "do_not_track": false,
    "tags": [],
    "enable_context_vars": false,
    "config_data": {
    }
}
mvdan commented 7 years ago

Okay, so the issue here is simpler than I thought.

The cause is the regex ?wsdl - it's not a valid regex. If you try this with Tyk built from master, it will error on startup with:

panic: regexp: Compile(`?wsdl`): error parsing regexp: missing argument to repetition operator: `?`

In other words, you probably want \?wsdl.

The reason why 2.3.x panics with a more confusing message is because the stable version sometimes doesn't check for errors in user-provided regexes. The upcoming 2.4 version does, hence the immediate crash with a proper error message.

I'm going to close this, as this is an issue in the API definition and the confusing panic has been addressed in the upcoming 2.4. Thanks!