plugin: coprocess auth doesn't apply policy check

[asoorm]

Branch/Environment/Version

all

Describe the bug

It is impossible to use policies with coprocess auth.

Expected behavior

An auth plugin should be able to check security policies. This will enable access rights (Authorization), rate-limits & quotas to be managed at the policy level, not the responsibility of the Auth plugin.

[ilijabojanovic]

@buger @asoorm Original issue is fixed, but we have problem with quotas.

On release-2.9 branch Reproduction steps:

1. Create policy with quota rates e.g 50 per hour
2. Add that policy in middleware
3. Generate traffic to that api

Actual: When we reach quota limit, quota counter resets and user can consume api without restrictions

Expected: When user reach quota limit we should stop traffic to that api due quota limits

Here is testing bundle: policy.zip

Here is video:

[ilijabojanovic]

@buger On latest release-2.9 branch Issue is still visible

Worth noticing that we apply in response headers X-Ratelimit-Reset but again issue is still visible

[buger]

I can't replicate so far. Do you use an ID extractor or similar? Can you try using a "new" unique key? How are your api and policy look like?

[ilijabojanovic]

@buger I we tried with new unique key, same issue X-Ratelimit-Reset was not set

Here is middleware: ilijafix-10.zip

Here is policy obj:

{
    "auth_type": "custom",
    "_id": "5e66c4d29633d71135a3b00a",
    "id": "",
    "org_id": "5dcaa92a9633d70d2397e557",
    "rate": 20,
    "per": 60,
    "throttle_interval": -1,
    "throttle_retry_limit": -1,
    "quota_max": 50,
    "quota_renewal_rate": 3600,
    "access_rights": {
        "273df2d6308d457e59cb5b8d3e391cd7": {
            "api_name": "auth",
            "api_id": "273df2d6308d457e59cb5b8d3e391cd7",
            "versions": [
                "v1",
                "v2"
            ],
            "allowed_urls": [
                {
                    "url": "get",
                    "methods": [
                        "GET"
                    ]
                },
                {
                    "url": "ip",
                    "methods": [
                        "GET"
                    ]
                },
                {
                    "url": "post",
                    "methods": [
                        "POST"
                    ]
                }
            ],
            "limit": null,
            "allowance_scope": ""
        }
    },
    "hmac_enabled": false,
    "active": true,
    "name": "python",
    "is_inactive": false,
    "date_created": "2020-03-09T23:36:02.306+01:00",
    "tags": [],
    "key_expires_in": 0,
    "partitions": {
        "quota": false,
        "rate_limit": false,
        "acl": false,
        "per_api": false
    },
    "last_updated": "1584053782",
    "meta_data": {
        "qa": "qa"
    }
}

[ilijabojanovic]

@buger Issue is visible again after merge.

[matiasinsaurralde]

@ilija I've added an additional fix here: #2955 Main point is that middleware.go checks for access_rights so only copying quota related fields in the first level of the session object isn't enough. This is found here in middleware.go. 👍

[ilijabojanovic]

@buger @matiasinsaurralde

On release-2.9 branch: ce0bc725a951e81f89627b539405f2a189c2e645

This issue is still visible:

BUNDLE: bundle-org.zip

TYK CONFIG

{
    "listen_address": "",
    "listen_port": 8181,
    "secret": "352d20ee67be67f6340b4c0605b044b7",
    "node_secret": "352d20ee67be67f6340b4c0605b044b7",
    "template_path": "templates",
    "tyk_js_path": "./js",
    "middleware_path": "middleware",
    "policies": {
        "policy_source": "service",
        "policy_connection_string": "http://www.tyk-test.com:3000",
        "policy_record_name": "tyk_policies",
        "allow_explicit_policy_id": false
    },
    "use_db_app_configs": true,
    "db_app_conf_options": {
        "connection_string": "http://www.tyk-test.com:3000",
        "node_is_segmented": false,
        "tags": null
    },
    "disable_dashboard_zeroconf": false,
    "app_path": "apps/",
    "storage": {
        "type": "redis",
        "host": "localhost",
        "port": 6379,
        "username": "",
        "password": "",
        "database": 0,
        "timeout": 0,
        "optimisation_max_idle": 2000,
        "optimisation_max_active": 4000,
        "enable_cluster": false,
        "use_ssl": false,
        "ssl_insecure_skip_verify": false
    },
    "enable_separate_cache_store": false,
    "cache_storage": {
        "type": "redis",
        "host": "localhost",
        "port": 6379,
        "hosts": null,
        "username": "",
        "password": "",
        "database": 0,
        "timeout": 5,
        "optimisation_max_idle": 0,
        "optimisation_max_active": 0,
        "enable_cluster": false,
        "use_ssl": false,
        "ssl_insecure_skip_verify": false
    },
    "enable_analytics": true,
    "analytics_config": {
        "type": "mongo",
        "ignored_ips": [],
        "enable_detailed_recording": true,
        "enable_geo_ip": true,
        "geo_ip_db_path": "./GeoLite2-City.mmdb",
        "normalise_urls": {
            "enabled": true,
            "normalise_uuids": true,
            "normalise_numbers": true,
            "custom_patterns": null
        },
        "pool_size": 0,
        "records_buffer_size": 0,
        "storage_expiration_time": 60
    },
    "health_check": {
        "enable_health_checks": false,
        "health_check_value_timeouts": 0
    },
    "optimisations_use_async_session_write": false,
    "session_update_pool_size": 0,
    "session_update_buffer_size": 0,
    "allow_master_keys": false,
    "hash_keys": true,
    "hash_key_function": "murmur64",
    "suppress_redis_signal_reload": false,
    "suppress_default_org_store": false,
    "use_redis_log": true,
    "sentry_code": "",
    "use_sentry": false,
    "use_syslog": false,
    "use_graylog": false,
    "use_logstash": false,
    "graylog_network_addr": "",
    "logstash_network_addr": "",
    "syslog_transport": "",
    "logstash_transport": "",
    "syslog_network_addr": "",
    "statsd_connection_string": "",
    "statsd_prefix": "",
    "enforce_org_data_age": true,
    "enforce_org_data_detail_logging": false,
    "enforce_org_quotas": true,
    "experimental_process_org_off_thread": true,
    "enable_non_transactional_rate_limiter": true,
    "enable_sentinel_rate_limiter": false,
    "enable_redis_rolling_limiter": false,
    "enable_http_profiler": true,
    "management_node": false,
    "monitor": {
        "enable_trigger_monitors": true,
        "configuration": {
            "method": "POST",
            "target_path": "https://webhook.site/bb8652e0-1d09-4ce7-80e2-11d40802b194",
            "template_path": "./templates/monitor_template.json",
            "header_map": null,
            "event_timeout": 0
        },
        "global_trigger_limit": 50,
        "monitor_user_keys": true,
        "monitor_org_keys": true
    },
    "oauth_refresh_token_expire": 0,
    "oauth_token_expire": 0,
    "oauth_token_expired_retain_period": 0,
    "oauth_redirect_uri_separator": "",
    "slave_options": {
        "use_rpc": false,
        "use_ssl": false,
        "ssl_insecure_skip_verify": false,
        "connection_string": "",
        "rpc_key": "",
        "api_key": "",
        "enable_rpc_cache": false,
        "bind_to_slugs": false,
        "disable_keyspace_sync": false,
        "group_id": "",
        "call_timeout": 0,
        "ping_timeout": 0,
        "rpc_pool_size": 0
    },
    "disable_virtual_path_blobs": false,
    "local_session_cache": {
        "disable_cached_session_state": false,
          "cached_session_timeout": 5,
          "cached_session_eviction": 10
    },
    "http_server_options": {
        "override_defaults": false,
        "read_timeout": 0,
        "write_timeout": 0,
        "use_ssl": true,
        "use_ssl_le": false,
        "ssl_insecure_skip_verify": true,
        "enable_websockets": true,
        "certificates": [
            {
                "domain_name": "*.tyk-test.com",
                "cert_file": "./server.crt",
                "key_file": "./server.key"
            }
        ],
        "ssl_certificates": null,
        "server_name": "www.tyk-test.com",
        "min_version": 0,
        "flush_interval": 0,
        "skip_url_cleaning": true,
        "skip_target_path_escaping": true,
        "ssl_ciphers": null
    },
    "service_discovery": {
        "default_cache_timeout": 0
    },
    "proxy_close_connections": false,
    "close_connections": true,
    "auth_override": {
        "force_auth_provider": false,
        "auth_provider": {
            "name": "",
            "storage_engine": "",
            "meta": null
        },
        "force_session_provider": false,
        "session_provider": {
            "name": "",
            "storage_engine": "",
            "meta": null
        }
    },
    "uptime_tests": {
        "disable": false,
        "config": {
            "failure_trigger_sample_size": 2,
            "time_wait": 5,
            "checker_pool_size": 50,
            "enable_uptime_analytics": true
        }
    },
    "hostname": "",
    "enable_api_segregation": false,
    "control_api_hostname": "",
    "enable_custom_domains": true,
    "enable_jsvm": true,
    "jsvm_timeout": 10,
    "coprocess_options": {
        "enable_coprocess": true,
        "coprocess_grpc_server": "",
        "python_path_prefix": "",
        "python_version": ""
    },
    "hide_generator_header": false,
    "event_handlers": {},
    "event_trigers_defunct": null,
    "pid_file_location": "",
    "allow_insecure_configs": true,
    "public_key_path": "/Users/ilijabojanovic/work/src/github.com/TykTechnologies/pubkey.pem",
    "close_idle_connections": false,
    "drl_notification_frequency": 0,
    "global_session_lifetime": 0,
    "force_global_session_lifetime": false,
    "bundle_base_url": "http://192.168.1.50:8000/",
    "bundle_insecure_skip_verify": true,
    "enable_bundle_downloader": true,
    "allow_remote_config": true,
    "legacy_enable_allowance_countdown": false,
    "max_idle_connections": 400,
    "max_idle_connections_per_host": 0,
    "max_conn_time": 0,
    "reload_wait_time": 0,
    "proxy_ssl_insecure_skip_verify": false,
    "proxy_ssl_min_version": 0,
    "proxy_ssl_ciphers": null,
    "proxy_default_timeout": 10,
    "proxy_ssl_disable_renegotiation": false,
    "log_level": "info",
    "control_api_port": 0,
    "enable_websockets": true,
    "security": {
        "private_certificate_encoding_secret": "",
        "control_api_use_mutual_tls": false,
        "pinned_public_keys": null,
        "certificates": {
            "apis": null,
            "upstream": null,
            "control_api": null,
            "dashboard_api": null,
            "mdcb_api": null
        }
    },
    "enable_key_logging": false,
    "newrelic": {
        "app_name": "",
        "license_key": ""
    },
    "health_check_endpoint_name": "",
    "version_header": "",
    "enable_hashed_keys_listing": true,
    "min_token_length": 0,
    "disable_regexp_cache": false,
    "regexp_cache_expire": 0,
    "disable_ports_whitelist": true,
    "ports_whitelist": {
        "http": {
          "ranges": [
            {
              "from": 8000,
              "to": 9000
            }
          ]
        },
        "tcp": {
          "ranges": [
            {
              "from": 7001,
              "to": 7900
            }
          ]
        },
        "tls": {
          "ports": [
            6000,
            6015
          ]
        }
      }
}

[ilijabojanovic]

Verified