search

TykTechnologies / tyk

Tyk Open Source API Gateway written in Go, supporting REST, GraphQL, TCP and gRPC protocols
Other
9.65k stars 1.08k forks source link

Service discovery for uptime tests crashes gateway #2715

Closed ilijabojanovic closed 4 years ago

ilijabojanovic commented 4 years ago

Branch/Environment/Version

Describe the bug When we have uptime service discovery on api definition, gateway crashes during api's loading.

Reproduction steps Steps to reproduce the behavior:

  1. Import this api definition 
    {
"id": "5cd440e8155be900010d2e99",
"name": "test-qa",
"slug": "test-qa",
"listen_port": 0,
"protocol": "",
"enable_proxy_protocol": false,
"api_id": "f309b7e8e0534cf77ba8fd402e2163f5",
"org_id": "588b52eab275ff0001cc7476",
"use_keyless": false,
"use_oauth2": false,
"use_openid": false,
"openid_options": {
    "providers": [],
    "segregate_by_client": false
},
"oauth_meta": {
    "allowed_access_types": [],
    "allowed_authorize_types": [],
    "auth_login_redirect": ""
},
"auth": {
    "use_param": true,
    "param_name": "auth",
    "use_cookie": false,
    "cookie_name": "",
    "auth_header_name": "Authorization",
    "use_certificate": false,
    "validate_signature": false,
    "signature": {
        "algorithm": "",
        "header": "",
        "secret": "",
        "allowed_clock_skew": 0,
        "error_code": 0,
        "error_message": ""
    }
},
"auth_configs": {},
"use_basic_auth": false,
"basic_auth": {
    "disable_caching": false,
    "cache_ttl": 0,
    "extract_from_body": false,
    "body_user_regexp": "",
    "body_password_regexp": ""
},
"use_mutual_tls_auth": false,
"client_certificates": [],
"upstream_certificates": {},
"pinned_public_keys": {},
"enable_jwt": false,
"use_standard_auth": true,
"use_go_plugin_auth": false,
"enable_coprocess_auth": false,
"jwt_signing_method": "",
"jwt_source": "",
"jwt_identity_base_field": "",
"jwt_client_base_field": "",
"jwt_policy_field_name": "",
"jwt_default_policies": [],
"jwt_issued_at_validation_skew": 0,
"jwt_expires_at_validation_skew": 0,
"jwt_not_before_validation_skew": 0,
"jwt_skip_kid": false,
"jwt_scope_to_policy_mapping": {},
"jwt_scope_claim_name": "",
"notifications": {
    "shared_secret": "",
    "oauth_on_keychange_url": ""
},
"enable_signature_checking": false,
"hmac_allowed_clock_skew": -1,
"hmac_allowed_algorithms": [],
"request_signing": {
    "is_enabled": false,
    "secret": "",
    "key_id": "",
    "algorithm": "",
    "header_list": [],
    "certificate_id": ""
},
"base_identity_provided_by": "",
"definition": {
    "location": "header",
    "key": "x-api-version",
    "strip_path": false
},
"version_data": {
    "not_versioned": true,
    "default_version": "",
    "versions": {
        "Default": {
            "name": "Default",
            "expires": "",
            "paths": {
                "ignored": [],
                "white_list": [],
                "black_list": []
            },
            "use_extended_paths": true,
            "extended_paths": {
                "ignored": [
                    {
                        "path": "/mock",
                        "ignore_case": false,
                        "method_actions": {
                            "GET": {
                                "action": "reply",
                                "code": 200,
                                "data": "{\n    \"mock\":\"response\"\n    }",
                                "headers": {
                                    "mock": "header"
                                }
                            }
                        }
                    },
                    {
                        "path": "/ip",
                        "ignore_case": false,
                        "method_actions": {
                            "GET": {
                                "action": "no_action",
                                "code": 200,
                                "data": "",
                                "headers": {}
                            }
                        }
                    }
                ],
                "cache": [
                    "/anything",
                    "rewrite5"
                ],
                "transform": [
                    {
                        "template_data": {
                            "input_type": "json",
                            "template_mode": "blob",
                            "enable_session": false,
                            "template_source": "ewogICAgInRyYW5zZm9ybWVkIjoie3suaGVsbG99fSIKfQ=="
                        },
                        "path": "/rewrite4",
                        "method": "GET"
                    }
                ],
                "transform_response": [
                    {
                        "template_data": {
                            "input_type": "json",
                            "template_mode": "blob",
                            "enable_session": false,
                            "template_source": "ewogICAgInRyYW5zZm9ybWVkIjoie3suZGF0YX19Igp9"
                        },
                        "path": "/rewrite4",
                        "method": "GET"
                    }
                ],
                "transform_headers": [
                    {
                        "delete_headers": [],
                        "add_headers": {
                            "request": "add"
                        },
                        "path": "/headers",
                        "method": "GET",
                        "act_on": false
                    }
                ],
                "transform_response_headers": [
                    {
                        "delete_headers": [],
                        "add_headers": {
                            "response": "add"
                        },
                        "path": "/headers",
                        "method": "GET",
                        "act_on": false
                    }
                ],
                "hard_timeouts": [
                    {
                        "path": "/delay",
                        "method": "GET",
                        "timeout": 2
                    }
                ],
                "url_rewrites": [
                    {
                        "path": "/rewrite4",
                        "method": "GET",
                        "match_pattern": "/rewrite4",
                        "rewrite_to": "/post",
                        "triggers": []
                    },
                    {
                        "path": "/rewrite3",
                        "method": "GET",
                        "match_pattern": "/rewrite3",
                        "rewrite_to": "",
                        "triggers": [
                            {
                                "on": "all",
                                "options": {
                                    "header_matches": {},
                                    "query_val_matches": {},
                                    "path_part_matches": {},
                                    "session_meta_matches": {},
                                    "request_context_matches": {},
                                    "payload_matches": {
                                        "match_rx": "hello",
                                        "reverse": false
                                    }
                                },
                                "rewrite_to": "/get"
                            }
                        ]
                    },
                    {
                        "path": "/rewrite2",
                        "method": "GET",
                        "match_pattern": "/rewrite2",
                        "rewrite_to": "",
                        "triggers": [
                            {
                                "on": "all",
                                "options": {
                                    "header_matches": {
                                        "test-qa": {
                                            "match_rx": "test-qa",
                                            "reverse": false
                                        }
                                    },
                                    "query_val_matches": {},
                                    "path_part_matches": {},
                                    "session_meta_matches": {},
                                    "request_context_matches": {},
                                    "payload_matches": {
                                        "match_rx": "",
                                        "reverse": false
                                    }
                                },
                                "rewrite_to": "/user-agent"
                            }
                        ]
                    },
                    {
                        "path": "/rewrite1",
                        "method": "GET",
                        "match_pattern": "/rewrite1",
                        "rewrite_to": "",
                        "triggers": [
                            {
                                "on": "all",
                                "options": {
                                    "header_matches": {},
                                    "query_val_matches": {
                                        "show_env": {
                                            "match_rx": "1",
                                            "reverse": false
                                        }
                                    },
                                    "path_part_matches": {},
                                    "session_meta_matches": {},
                                    "request_context_matches": {},
                                    "payload_matches": {
                                        "match_rx": "",
                                        "reverse": false
                                    }
                                },
                                "rewrite_to": "/get?show_env=2"
                            }
                        ]
                    },
                    {
                        "path": "rewrite5",
                        "method": "GET",
                        "match_pattern": "rewrite5",
                        "rewrite_to": "/ip",
                        "triggers": []
                    },
                    {
                        "path": "/rewrite",
                        "method": "GET",
                        "match_pattern": "/rewrite",
                        "rewrite_to": "/headers",
                        "triggers": []
                    }
                ],
                "virtual": [
                    {
                        "response_function_name": "testVirtData",
                        "function_source_type": "blob",
                        "function_source_uri": "ZnVuY3Rpb24gdGVzdFZpcnREYXRhKHJlcXVlc3QsIHNlc3Npb24sIGNvbmZpZykgewogICAgdmFyIHJlc3AgPSB7CiAgICAgICAgQm9keTogcmVxdWVzdC5Cb2R5ICsgIiBhZGRlZCBib2R5IiwgCiAgICAgICAgSGVhZGVyczogewogICAgICAgICAgICAiZGF0YS1mb28iOiBjb25maWcuY29uZmlnX2RhdGEuZm9vCiAgICAgICAgfSwKICAgICAgICBDb2RlOiAyMDIKICAgIH0KICAgIHJldHVybiBUeWtKc1Jlc3BvbnNlKHJlc3AsIHNlc3Npb24ubWV0YV9kYXRhKSAgIAp9",
                        "path": "/ilija2",
                        "method": "POST",
                        "use_session": false,
                        "proxy_on_error": false
                    },
                    {
                        "response_function_name": "batchTest",
                        "function_source_type": "blob",
                        "function_source_uri": "ZnVuY3Rpb24gYmF0Y2hUZXN0IChyZXF1ZXN0LCBzZXNzaW9uLCBjb25maWcpIHsKICAgICAgICAvLyBTZXQgdXAgYSByZXNwb25zZSBvYmplY3QKICAgICAgICB2YXIgcmVzcG9uc2UgPSB7CiAgICAgICAgICAgIEJvZHk6ICIiCiAgICAgICAgICAgIEhlYWRlcnM6IHsKICAgICAgICAgICAgICAgICJ0ZXN0IjogInZpcnR1YWwtaGVhZGVyLTEiLAogICAgICAgICAgICAgICAgInRlc3QtMiI6ICJ2aXJ0dWFsLWhlYWRlci0yIiwKICAgICAgICAgICAgICAgICJjb250ZW50LXR5cGUiOiAiYXBwbGljYXRpb24vanNvbiIKICAgICAgICAgICAgfSwKICAgICAgICAgICAgQ29kZTogMjAwCiAgICAgICAgfQogICAgCiAgICAgICAgLy8gQmF0Y2ggcmVxdWVzdAogICAgICAgIHZhciBiYXRjaCA9IHsKICAgICAgICAgICAgInJlcXVlc3RzIjogWwogICAgICAgICAgICAgICAgewogICAgICAgICAgICAgICAgICAgICJtZXRob2QiOiAiR0VUIiwKICAgICAgICAgICAgICAgICAgICAiaGVhZGVycyI6IHsKICAgICAgICAgICAgICAgICAgICAgICAgIngtdHlrLXRlc3QiOiAiMSIsCiAgICAgICAgICAgICAgICAgICAgICAgICJ4LXR5ay12ZXJzaW9uIjogIjEuMiIsCiAgICAgICAgICAgICAgICAgICAgICAgICJhdXRob3JpemF0aW9uIjogIjFkYmM4M2I5YzQzMTY0OWQ3Njk4ZmFhOTc5N2UyOTAwZiIKICAgICAgICAgICAgICAgICAgICB9LAogICAgICAgICAgICAgICAgICAgICJib2R5IjogIiIsCiAgICAgICAgICAgICAgICAgICAgInJlbGF0aXZlX3VybCI6ICJodHRwOi8vaHR0cGJpbi5vcmcvaGVhZGVycyIKICAgICAgICAgICAgICAgIH0sCiAgICAgICAgICAgICAgICB7CiAgICAgICAgICAgICAgICAgICAgIm1ldGhvZCI6ICJHRVQiLAogICAgICAgICAgICAgICAgICAgICJoZWFkZXJzIjoge30sCiAgICAgICAgICAgICAgICAgICAgImJvZHkiOiAiIiwKICAgICAgICAgICAgICAgICAgICAicmVsYXRpdmVfdXJsIjogImh0dHA6Ly9odHRwYmluLm9yZy91c2VyLWFnZW50IgogICAgICAgICAgICAgICAgfQogICAgICAgICAgICBdLAogICAgICAgICAgICAic3VwcHJlc3NfcGFyYWxsZWxfZXhlY3V0aW9uIjogZmFsc2UKICAgICAgICB9CiAgICAKICAgICAgICBsb2coIltWaXJ0dWFsIFRlc3RdIE1ha2luZyBVcHN0cmVhbSBCYXRjaCBSZXF1ZXN0IikKICAgICAgICB2YXIgbmV3Qm9keSA9IFR5a0JhdGNoUmVxdWVzdChKU09OLnN0cmluZ2lmeShiYXRjaCkpCiAgICAKICAgICAgICAvLyBXZSBrbm93IHRoYXQgdGhlIHJlcXVlc3RzIHJldHVybiBKU09OIGluIHRoZWlyIGJvZHksIGxldHMgZmxhdHRlbiBpdAogICAgICAgIHZhciBhc0pTID0gSlNPTi5wYXJzZShuZXdCb2R5KQogICAgICAgIGZvciAodmFyIGkgaW4gYXNKUykgewogICAgICAgICAgICBhc0pTW2ldLmJvZHkgPSBKU09OLnBhcnNlKGFzSlNbaV0uYm9keSkKICAgICAgICB9CiAgICAKICAgICAgICAvLyBXZSBuZWVkIHRvIHNlbmQgYSBzdHJpbmcgb2JqZWN0IGJhY2sgdG8gVHlrIHRvIGVtYmVkIGluIHRoZSByZXNwb25zZQogICAgICAgIHJlc3BvbnNlLkJvZHkgPSBKU09OLnN0cmluZ2lmeShhc0pTKQogICAgCiAgICAgICAgcmV0dXJuIFR5a0pzUmVzcG9uc2UocmVzcG9uc2UsIHNlc3Npb24ubWV0YV9kYXRhKQogICAgCiAgICB9CiAgICBsb2coIkJhdGNoIFRlc3QgaW5pdGlhbGlzZWQiKQ==",
                        "path": "/ilija",
                        "method": "GET",
                        "use_session": false,
                        "proxy_on_error": false
                    }
                ],
                "method_transforms": [
                    {
                        "path": "/rewrite4",
                        "method": "GET",
                        "to_method": "POST"
                    },
                    {
                        "path": "/post",
                        "method": "GET",
                        "to_method": "POST"
                    }
                ],
                "track_endpoints": [
                    {
                        "path": "rewrite5",
                        "method": "GET"
                    }
                ],
                "validate_json": [
                    {
                        "path": "/post",
                        "method": "POST",
                        "schema": {
                            "properties": {
                                "active": {
                                    "type": "boolean"
                                },
                                "email": {
                                    "type": "string"
                                },
                                "id": {
                                    "type": "number"
                                },
                                "name": {
                                    "type": "string"
                                },
                                "something": {
                                    "items": {
                                        "type": "string"
                                    },
                                    "type": "array"
                                }
                            },
                            "required": [
                                "name",
                                "email"
                            ],
                            "type": "object"
                        },
                        "error_response_code": 400
                    }
                ]
            },
            "global_headers": {
                "add_header": "qa"
            },
            "global_headers_remove": [],
            "global_size_limit": 0,
            "override_target": ""
        }
    }
},
"uptime_tests": {
    "check_list": [],
    "config": {
        "expire_utime_after": 0,
        "service_discovery": {
            "use_discovery_service": true,
            "query_endpoint": "http://test.com/api",
            "use_nested_query": false,
            "parent_data_path": "v",
            "data_path": "asd",
            "port_data_path": "",
            "target_path": "",
            "use_target_list": false,
            "cache_timeout": 60,
            "endpoint_returns_list": false
        },
        "recheck_wait": 0
    }
},
"proxy": {
    "preserve_host_header": false,
    "listen_path": "/f309b7e8e0534cf77ba8fd402e2163f5/",
    "target_url": "http://httpbin.org/",
    "disable_strip_slash": false,
    "strip_listen_path": true,
    "enable_load_balancing": false,
    "target_list": [],
    "check_host_against_uptime_tests": true,
    "service_discovery": {
        "use_discovery_service": false,
        "query_endpoint": "",
        "use_nested_query": false,
        "parent_data_path": "",
        "data_path": "",
        "port_data_path": "",
        "target_path": "asd",
        "use_target_list": false,
        "cache_timeout": 0,
        "endpoint_returns_list": false
    },
    "transport": {
        "ssl_insecure_skip_verify": false,
        "ssl_ciphers": [],
        "ssl_min_version": 0,
        "proxy_url": ""
    }
},
"disable_rate_limit": false,
"disable_quota": false,
"custom_middleware": {
    "pre": [],
    "post": [],
    "post_key_auth": [],
    "auth_check": {
        "name": "",
        "path": "",
        "require_session": false,
        "raw_body_only": false
    },
    "response": [],
    "driver": "",
    "id_extractor": {
        "extract_from": "",
        "extract_with": "",
        "extractor_config": {}
    }
},
"custom_middleware_bundle": "",
"cache_options": {
    "cache_timeout": 60,
    "enable_cache": true,
    "cache_all_safe_requests": false,
    "cache_response_codes": [],
    "enable_upstream_cache_control": false,
    "cache_control_ttl_header": ""
},
"session_lifetime": 0,
"active": true,
"internal": false,
"auth_provider": {
    "name": "",
    "storage_engine": "",
    "meta": {}
},
"session_provider": {
    "name": "",
    "storage_engine": "",
    "meta": {}
},
"event_handlers": {
    "events": {}
},
"enable_batch_request_support": true,
"enable_ip_whitelisting": false,
"allowed_ips": [],
"enable_ip_blacklisting": false,
"blacklisted_ips": [],
"dont_set_quota_on_create": false,
"expire_analytics_after": 0,
"response_processors": [
    {
        "name": "header_injector",
        "options": {}
    },
    {
        "name": "response_body_transform",
        "options": {}
    }
],
"CORS": {
    "enable": false,
    "allowed_origins": [],
    "allowed_methods": [],
    "allowed_headers": [],
    "exposed_headers": [],
    "allow_credentials": false,
    "max_age": 24,
    "options_passthrough": false,
    "debug": false
},
"domain": "",
"certificates": [],
"do_not_track": false,
"tags": [],
"enable_context_vars": true,
"config_data": {
    "foo": "bar"
},
"tag_headers": [],
"global_rate_limit": {
    "rate": 0,
    "per": 0
},
"strip_auth_data": true
}
  2. Watch console

Actual behavior Gateway will crash

[Dec 04 20:10:33]  INFO host-check-mgr: Loading uptime tests...
[Dec 04 20:10:33] ERROR invalid character '<' looking for beginning of value
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0x4abca57]

goroutine 1 [running]:
github.com/TykTechnologies/tyk/gateway.(*ServiceDiscovery).ParseObject(0xc0003dd940, 0xc000a92000, 0x615, 0xc000adefb0, 0x3, 0xc000044570)
    /Users/ilijabojanovic/work/src/github.com/TykTechnologies/tyk/gateway/service_discovery.go:207 +0x197
github.com/TykTechnologies/tyk/gateway.(*ServiceDiscovery).ProcessRawData(0xc0003dd940, 0xc000a92000, 0x615, 0xc000a92000, 0x615, 0x0)
    /Users/ilijabojanovic/work/src/github.com/TykTechnologies/tyk/gateway/service_discovery.go:248 +0x531
github.com/TykTechnologies/tyk/gateway.(*ServiceDiscovery).Target(0xc0003dd940, 0xc0008cbba0, 0x13, 0x7e3, 0x7e3, 0x0)
    /Users/ilijabojanovic/work/src/github.com/TykTechnologies/tyk/gateway/service_discovery.go:274 +0x91
github.com/TykTechnologies/tyk/gateway.(*HostCheckerManager).ListFromService(0x5aa8b60, 0xc0008cbb20, 0x20, 0x1, 0x1, 0x101, 0x0, 0x190)
    /Users/ilijabojanovic/work/src/github.com/TykTechnologies/tyk/gateway/host_checker_manager.go:395 +0x182
github.com/TykTechnologies/tyk/gateway.SetCheckerHostList()
    /Users/ilijabojanovic/work/src/github.com/TykTechnologies/tyk/gateway/host_checker_manager.go:521 +0x9d8
github.com/TykTechnologies/tyk/gateway.loadApps(0xc000b3a780, 0x3b, 0x3b)
    /Users/ilijabojanovic/work/src/github.com/TykTechnologies/tyk/gateway/api_loader.go:719 +0xb5e
github.com/TykTechnologies/tyk/gateway.loadGlobalApps()
    /Users/ilijabojanovic/work/src/github.com/TykTechnologies/tyk/gateway/api_loader.go:548 +0xe1
github.com/TykTechnologies/tyk/gateway.DoReload()
    /Users/ilijabojanovic/work/src/github.com/TykTechnologies/tyk/gateway/server.go:684 +0x20e
github.com/TykTechnologies/tyk/gateway.startServer()
    /Users/ilijabojanovic/work/src/github.com/TykTechnologies/tyk/gateway/server.go:1257 +0x808
github.com/TykTechnologies/tyk/gateway.Start()
    /Users/ilijabojanovic/work/src/github.com/TykTechnologies/tyk/gateway/server.go:1070 +0x5fc
main.main()
    /Users/ilijabojanovic/work/src/github.com/TykTechnologies/tyk/main.go:6 +0x20

Expected behavior Gateway should not crash

Screenshots/Video If applicable, add screenshots or video to help explain your problem.

Logs (debug mode or log file): Log from console or from log file.

Configuration (tyk config file): Attach tyk configuration file

Additional context Add any other context about the problem here.

stale[bot] commented 4 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs, please add comments to this ticket if you would like it to stay open. Thank you for your contributions.