Kong won't create acl and jwt plugins on startup using a customized kong.yml

[francisco-andrade]

Hi there,

I'm using Kong 0.7 and I have an issue where Kong is not creating the acl and jwt plugins on startup when using a customized kong.yml file. I've tracked down the problem to my kong.yml file, because when I start the service with the default file these plugins are installed.

When running the service with the customized kong.yml, I'm able to associate the acl plugin to an api:

curl -X POST --url http://localhost:8001/apis/myapi/plugins/ --data 'name=acl' --data "config.whitelist=dacota" {"api_id":"c4f5e0e9-297b-4628-8657-54033f8eca06","id":"167e5ab0-497f-496f-a3dc-459bc2cc5567","created_at":1460554831000,"enabled":true,"name":"acl","config":{"whitelist":["dacota"]}}

But when I try to add a group to an existing consumer, kong returns the following error:

curl -X POST http://localhost:8001/consumers/ee9b1d27-022f-42de-99d0-030d7b9d5282/acls --data "group=dacota" {"message":"Not found"}

I get the same message when I try to get the acls for the user:

curl -X GET http://localhost:8001/consumers/ee9b1d27-022f-42de-99d0-030d7b9d5282/acls {"message":"Not found"}

Both commands works using the default file. I compared the working with the not working kong.yml but I'm not finding where the problem could be.

Do anyone knows where are the plugins listed to be installed on startup?

Thanks, Francisco Andrade

[francisco-andrade]

Here's my customized kong.yml:

custom_plugins:
  - helloworld
  - q-to-filter
  - json-filter
proxy_listen: "0.0.0.0:8000"
proxy_listen_ssl: "0.0.0.0:8443"
admin_api_listen: "0.0.0.0:8001"
cluster_listen: "10.166.69.206:7946"
cluster_listen_rpc: "127.0.0.1:7373"
dns_resolver: dnsmasq
dns_resolvers_available:
 server:
  address: "8.8.8.8:53"
 dnsmasq:
  port: 8053
cluster:
 advertise: "10.166.69.206:7946"

database: cassandra
cassandra:
  username: cassandra
  password: cassandra
  contact_points:
    - "localhost:9042"
nginx: |
  {{user}}
  worker_processes auto;
  error_log logs/error.log debug;
  daemon off;

  worker_rlimit_nofile {{auto_worker_rlimit_nofile}};

  env KONG_CONF;
  env PATH;

  events {
    worker_connections {{auto_worker_connections}};
    multi_accept on;
  }

  http {
    resolver {{dns_resolver}} ipv6=off;
    charset UTF-8;

    proxy_cache_path /tmp/cache keys_zone=mem_cache_zone:25m loader_threshold=300 loader_files=200 max_size=100m;

    access_log logs/access.log;
    access_log on;

    keepalive_timeout 60s;
    client_header_timeout 60s;
    client_body_timeout 60s;
    send_timeout 60s;

    proxy_buffer_size 128k;
    proxy_buffers 4 256k;
    proxy_busy_buffers_size 256k;
    proxy_ssl_server_name on;

    real_ip_header X-Forwarded-For;
    set_real_ip_from 0.0.0.0/0;
    real_ip_recursive on;

    client_max_body_size 0;
    underscores_in_headers on;
    reset_timedout_connection on;
    tcp_nopush on;

    lua_package_path ';;';
    lua_code_cache on;
    lua_max_running_timers 4096;
    lua_max_pending_timers 16384;
    lua_shared_dict reports_locks 100k;
    lua_shared_dict cluster_locks 100k;
    lua_shared_dict cluster_autojoin_locks 100k;
    lua_shared_dict cache {{memory_cache_size}}m;
    lua_shared_dict cassandra 1m;
    lua_shared_dict cassandra_prepared 5m;
    lua_socket_log_errors off;
    {{lua_ssl_trusted_certificate}}

    init_by_lua_block {
      kong = require "kong"
      kong.init()
    }

    init_worker_by_lua_block {
      kong.init_worker()
    }

    server {
      server_name _;
      proxy_cache mem_cache_zone;

      listen {{proxy_listen}};
      listen {{proxy_listen_ssl}} ssl;

      ssl_certificate_by_lua_block {
        kong.ssl_certificate()
      }

      ssl_certificate {{ssl_cert}};
      ssl_certificate_key {{ssl_key}};
      ssl_protocols TLSv1 TLSv1.1 TLSv1.2;# omit SSLv3 because of POODLE (CVE-2014-3566)

      location / {
        default_type 'text/plain';

        proxy_cache_valid 200 206 20m; #http_codes and time in [1-9]m or [1-24]h
        proxy_cache_min_uses 1;

        set $upstream_host nil;
        set $upstream_url nil;

        access_by_lua_block {
          kong.access()
        }

        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header Host $upstream_host;
        proxy_pass $upstream_url;
        proxy_pass_header Server;

        header_filter_by_lua_block {
          kong.header_filter()
        }

        body_filter_by_lua_block {
          kong.body_filter()
        }

        log_by_lua_block {
          kong.log()
        }
      }

      location /robots.txt {
        return 200 'User-agent: *\nDisallow: /';
      }

      error_page 500 502 503 504 /50x;

      location = /50x {
        internal;
        content_by_lua_block {
          require("kong.core.error_handlers")(ngx)
        }
      }

      location /healthcheck {
        return 200 'Kong APIs is healthy!';
      }
    }

    server {
      listen {{admin_api_listen}};

      client_max_body_size 10m;
      client_body_buffer_size 10m;

      location / {
        default_type application/json;
        content_by_lua_block {
          ngx.header["Access-Control-Allow-Origin"] = "*"
          if ngx.req.get_method() == "OPTIONS" then
            ngx.header["Access-Control-Allow-Methods"] = "GET,HEAD,PUT,PATCH,POST,DELETE"
            ngx.header["Access-Control-Allow-Headers"] = "Content-Type"
            ngx.exit(204)
          end
          local lapis = require "lapis"
          lapis.serve "kong.api.app"
        }
      }

      location /nginx_status {
        internal;
        access_log off;
        stub_status;
      }

      location /robots.txt {
        return 200 'User-agent: *\nDisallow: /';
      }

      location /healthcheck {
        return 200 'Kong ADMIN is healthy!';
      }
    }
  }

[francisco-andrade]

I've commented the custom_plugins section on my kong.yml and now kong only installs jwt and acl. Not sure why right now.

[UPDATE] On a second startup, all plugins were installed

[francisco-andrade]

I just figured out that the problem only happens when I enable a specific custom_plugin. Commenting the "json-filter" plugin on kong.yml makes acl to be installed. This plugin rewrites the json response.

I'll check if there's any problem on the plugin sources.

[UPDATE] I removed the hello-world plugin and added back the json-filter and acl is installed, but jwt is not. So the problem may not be on a specific plugin.

[francisco-andrade]

Latest results on the analysis: When I have 3 plugins listed on the custom_plugin section on kong.yml both jwt and acl are not been installed. When I have 2 plugins listed on the custom_plugin, acl is installed but jwt is not. When I have only 1 or none plugin listed, both acl and jwt are installed.

It looks like we are reaching a plugin limit.

[thibaultcha]

Hi,

What do you mean by plugins being "installed" or "not installed"?

[francisco-andrade]

Hi @thibaultCha , at first they are not included on the kong startup migrations:

[INFO] Migrating acl on keyspace "kong" (cassandra) [INFO] acl migrated up to: 2015-08-25-841841_init_acl

We don't see them at the cassandra tables (because they were not migrated) and when I try to use the acl plugin I get an error:

curl -X POST http://localhost:8001/consumers/ee9b1d27-022f-42de-99d0-030d7b9d5282/acls --data "group=dacota" {"message":"Not found"}

curl -X GET http://localhost:8001/consumers/ee9b1d27-022f-42de-99d0-030d7b9d5282/acls {"message":"Not found"}

I tried to create the acl tables manually, but it didn't worked. I didn't tested the jwt to see how it is affected.

[Tieske]

@chicoandrade does this problem still persist? I tried to replicate it on 0.8.3, but failed.

I added three custom plugins, and then ran against an empty database again, the migration log was exactly the same as without the three custom plugins.

I understood you didn't see the migrations, so hence that's what I checked.

[francisco-andrade]

Hi @Tieske , sorry for the absence of feedback. The problem seems to be solved on Kong's latest versions. We updated the version and everything is working fine. Thanks!

[Tieske]

ok, closing this then.