Invalid column name algorithm because it conflicts with an existing column

[jl91]

I updated Kong for 0.8.0rc1 version to fix my problem with https://github.com/Mashape/kong/issues/1067

but now when a try to start kong a have this problem :

[INFO] Kong 0.8.0rc1 [INFO] Using configuration: /etc/kong/kong.yml [INFO] Setting working directory to /usr/local/kong [INFO] database...........cassandra keyspace=kong ssl=verify=false enabled=false consistency=ONE contact_points=127.0.0.1:9042 replication_strategy=SimpleStrategy timeout=5000 replication_factor=1 data_centers= [INFO] Migrating jwt (cassandra) [INFO] Leaving cluster.. [ERR] Error during migration 2016-03-07-jwt-alg: [Invalid] Invalid column name algorithm because it conflicts with an existing column [ERR] Could not start Kong

[jl91]

This is all my error log from today

2016/03/31 15:14:51 [error] 27004#0: *638 lua coroutine: runtime error: don't know how to respond to PATCH stack traceback: coroutine 0:

coroutine 1: C: in function 'resume' /usr/local/share/lua/5.1/lapis/application.lua:656: in function 'handler' /usr/local/share/lua/5.1/lapis/application.lua:393: in function 'resolve' ./kong/api/app.lua:99: in function 'handler' /usr/local/share/lua/5.1/lapis/application.lua:393: in function 'handler' /usr/local/share/lua/5.1/lapis/application.lua:404: in function </usr/local/share/lua/5.1/lapis/application.lua:400> C: in function 'xpcall' /usr/local/share/lua/5.1/lapis/application.lua:400: in function 'dispatch' /usr/local/share/lua/5.1/lapis/nginx.lua:205: in function 'serve' content_by_lua(nginx.conf:155):9: in function <content_by_lua(nginx.conf:155):1>, client: 127.0.0.1, server: , request: "PATCH /apis/identity/plugins HTTP/1.1", host: "localhost:8001" 2016/03/31 15:15:16 [error] 27004#0: *638 lua coroutine: runtime error: don't know how to respond to PATCH stack traceback: coroutine 0:

coroutine 1: C: in function 'resume' /usr/local/share/lua/5.1/lapis/application.lua:656: in function 'handler' /usr/local/share/lua/5.1/lapis/application.lua:393: in function 'resolve' ./kong/api/app.lua:99: in function 'handler' /usr/local/share/lua/5.1/lapis/application.lua:393: in function 'handler' /usr/local/share/lua/5.1/lapis/application.lua:404: in function </usr/local/share/lua/5.1/lapis/application.lua:400> C: in function 'xpcall' /usr/local/share/lua/5.1/lapis/application.lua:400: in function 'dispatch' /usr/local/share/lua/5.1/lapis/nginx.lua:205: in function 'serve' content_by_lua(nginx.conf:155):9: in function <content_by_lua(nginx.conf:155):1>, client: 127.0.0.1, server: , request: "PATCH /apis/identity/plugins HTTP/1.1", host: "localhost:8001" 2016/03/31 15:17:59 [error] 27006#0: *685 lua coroutine: runtime error: don't know how to respond to PATCH stack traceback: coroutine 0:

coroutine 1: C: in function 'resume' /usr/local/share/lua/5.1/lapis/application.lua:656: in function 'handler' /usr/local/share/lua/5.1/lapis/application.lua:393: in function 'resolve' ./kong/api/app.lua:99: in function 'handler' /usr/local/share/lua/5.1/lapis/application.lua:393: in function 'handler' /usr/local/share/lua/5.1/lapis/application.lua:404: in function </usr/local/share/lua/5.1/lapis/application.lua:400> C: in function 'xpcall' /usr/local/share/lua/5.1/lapis/application.lua:400: in function 'dispatch' /usr/local/share/lua/5.1/lapis/nginx.lua:205: in function 'serve' content_by_lua(nginx.conf:155):9: in function <content_by_lua(nginx.conf:155):1>, client: 127.0.0.1, server: , request: "PATCH /apis/identity/plugins HTTP/1.1", host: "localhost:8001" 2016/03/31 15:35:17 [error] 27000#0: *933 lua coroutine: runtime error: don't know how to respond to PATCH stack traceback: coroutine 0:

coroutine 1: C: in function 'resume' /usr/local/share/lua/5.1/lapis/application.lua:656: in function 'handler' /usr/local/share/lua/5.1/lapis/application.lua:393: in function 'resolve' /usr/local/share/lua/5.1/lapis/application.lua:402: in function </usr/local/share/lua/5.1/lapis/application.lua:400> C: in function 'xpcall' /usr/local/share/lua/5.1/lapis/application.lua:400: in function 'dispatch' /usr/local/share/lua/5.1/lapis/nginx.lua:205: in function 'serve' content_by_lua(nginx.conf:155):9: in function <content_by_lua(nginx.conf:155):1>, client: 127.0.0.1, server: , request: "PATCH /apis/identity/plugins/ HTTP/1.1", host: "localhost:8001" 2016/03/31 16:12:35 [notice] 7574#0: signal process started 2016/03/31 16:12:36 [error] 7710#0: [lua] kong.lua:144: init(): Startup error: /usr/local/share/lua/5.1/kong/tools/utils.lua:221: /usr/local/share/lua/5.1/kong/plugins/runscope/handler.lua:3: module 'kong.plugins.runscope.log' not found: no field package.preload['kong.plugins.runscope.log'] no file '/usr/local/openresty/lualib/kong/plugins/runscope/log.lua' no file '/usr/local/openresty/lualib/kong/plugins/runscope/log/init.lua' no file './kong/plugins/runscope/log.lua' no file '/usr/local/openresty/luajit/share/luajit-2.1.0-beta1/kong/plugins/runscope/log.lua' no file '/usr/local/share/lua/5.1/kong/plugins/runscope/log.lua' no file '/usr/local/share/lua/5.1/kong/plugins/runscope/log/init.lua' no file '/usr/local/openresty/luajit/share/lua/5.1/kong/plugins/runscope/log.lua' no file '/usr/local/openresty/luajit/share/lua/5.1/kong/plugins/runscope/log/init.lua' no file '/usr/local/openresty/lualib/kong/plugins/runscope/log.so' no file './kong/plugins/runscope/log.so' no file '/usr/local/lib/lua/5.1/kong/plugins/runscope/log.so' no file '/usr/local/openresty/luajit/lib/lua/5.1/kong/plugins/runscope/log.so' no file '/usr/local/lib/lua/5.1/loadall.so' no file '/usr/local/openresty/lualib/kong.so' no file './kong.so' no file '/usr/local/lib/lua/5.1/kong.so' no file '/usr/local/openresty/luajit/lib/lua/5.1/kong.so' no file '/usr/local/lib/lua/5.1/loadall.so' 2016/03/31 16:12:37 [notice] 7733#0: signal process started 2016/03/31 16:12:37 [error] 7733#0: open() "/usr/local/kong/nginx.pid" failed (2: No such file or directory) 2016/03/31 16:17:53 [notice] 11488#0: signal process started 2016/03/31 16:17:53 [error] 11488#0: open() "/usr/local/kong/nginx.pid" failed (2: No such file or directory) 2016/03/31 16:36:21 [notice] 17567#0: signal process started 2016/03/31 16:36:21 [alert] 17567#0: kill(11602, 15) failed (3: No such process) 2016/03/31 16:36:30 [notice] 17621#0: signal process started 2016/03/31 16:36:30 [alert] 17621#0: kill(11602, 15) failed (3: No such process) 2016/03/31 16:38:24 [notice] 18617#0: signal process started 2016/03/31 16:38:24 [alert] 18617#0: kill(11602, 15) failed (3: No such process) 2016/03/31 16:53:23 [warn] 23739#0: the "user" directive makes sense only if the master process runs with super-user privileges, ignored in /usr/local/kong/nginx.conf:2 2016/03/31 16:53:23 [notice] 23739#0: signal process started 2016/03/31 16:53:23 [alert] 23739#0: kill(11602, 15) failed (3: No such process) 2016/03/31 16:53:29 [notice] 23803#0: signal process started 2016/03/31 16:53:29 [alert] 23803#0: kill(11602, 15) failed (3: No such process)

[subnetmarco]

@jl91 can you try with a fresh Cassandra datastore?

[jl91]

I formated my pc yesterday and when i start install kong 8rc1 that was ok. but, after i reboot it. i have the same problem ...

╰─λ sudo kong start 0 < 13:40:01 [INFO] Kong 0.8.0rc1 [INFO] Using configuration: /etc/kong/kong.yml [INFO] Setting working directory to /usr/local/kong [INFO] database...........cassandra keyspace=kong ssl=verify=false enabled=false consistency=ONE contact_points=127.0.0.1:9042 replication_strategy=SimpleStrategy timeout=5000 replication_factor=1 data_centers= [INFO] Migrating core (cassandra) [INFO] core migrated up to: 2015-01-12-175310_skeleton [INFO] core migrated up to: 2015-01-12-175310_init_schema [INFO] core migrated up to: 2015-11-23-817313_nodes [INFO] core migrated up to: 2016-02-25-160900_remove_null_consumer_id [INFO] core migrated up to: 2016-02-29-121813_remove_ttls [INFO] Migrating key-auth (cassandra) [INFO] key-auth migrated up to: 2015-07-31-172400_init_keyauth [INFO] Migrating rate-limiting (cassandra) [INFO] rate-limiting migrated up to: 2015-08-03-132400_init_ratelimiting [INFO] Migrating mashape-analytics (cassandra) [INFO] mashape-analytics migrated up to: 2015-12-03-161400_mashape-analytics-config [INFO] Migrating acl (cassandra) [INFO] acl migrated up to: 2015-08-25-841841_init_acl [INFO] Migrating response-ratelimiting (cassandra) [INFO] response-ratelimiting migrated up to: 2015-08-21_init_response-rate-limiting [INFO] Migrating oauth2 (cassandra) [INFO] oauth2 migrated up to: 2015-08-03-132400_init_oauth2 [INFO] oauth2 migrated up to: 2015-08-24-215800_cascade_delete_index [INFO] oauth2 migrated up to: 2016-02-29-435612_remove_ttl [INFO] Migrating request-transformer (cassandra) [INFO] request-transformer migrated up to: 2016-03-10-160000_req_trans_schema_changes [INFO] Migrating jwt (cassandra) [INFO] jwt migrated up to: 2015-06-09-jwt-auth [INFO] Leaving cluster.. [ERR] Error during migration 2016-03-07-jwt-alg: [Invalid] Invalid column name algorithm because it conflicts with an existing column [ERR] Could not start Kong

and i do not create any api i just install cassandra, and kong

[thibaultcha]

I have no problem running the migrations up and down again from an empty keyspace, and then starting/stopping both Kong and Cassandra.

Those migrations were introduced during development, and one of them had to be renamed because was not properly named in #1053. I believe it could cause issue if Kong was installed in between #1053 and the renaming, but not otherwise.

Could you connect via cqlsh into your node, and run:

DESCRIBE kong;
SELECT * FROM kong.schema_migrations;

[davideagle]

I'm seeing the same behaviour, was there any solution for this issue?

[hatchetation]

We're hitting this issue as well, with an upgrade to 0.8.3

Our current schema and migrations: https://gist.github.com/hatchetation/87dc5c2daee6e06d18b5fc0ea1b552aa

Some thoughts:

1) It seems like the root issue is that these columns already exist, and that the cassandra ALTER TABLE __ ADD action isn't expressed idempotently. I'm not sure csql allows for native idempotent column additions - the migration would need to be guarded better in code?

2) I don't understand why kong is trying to repeat this migration if it's already been applied and reflected in the schema_migrations table.

kong migrations list shows:

[INFO] jwt: 2015-06-09-jwt-auth, 2015-06-09-jwt-auth, 2016-03-07-jwt-alg

FWIW, we see migrations against core and oauth2 being run each time kong is started. Is this normal? I don't have a firm grasp on how kong migrations are supposed to be triggered.

[hatchetation]

@davideagle do you have duplicate migrations recorded in schema_migrations? We do, and think that may cause problems with how factory.lua#migrate evaluates migrations to run.

We're experimenting with manually clearing out the duplicate migrations, so far it's promising.

[subnetmarco]

Could you paste here the result of

SELECT * FROM kong.schema_migrations; ​ where we can see the double migration?

[chrisst]

I don't have the full migrations table (we've been cleaning it up) but here's an example of core before it got clean:

core | ['2015-01-12-175310_skeleton', '2015-01-12-175310_init_schema', '2015-11-23-817313_nodes', '2015-01-12-175310_skeleton', '2015-01-12-175310_init_schema', '2015-11-23-817313_nodes', '2016-02-25-160900_remove_null_consumer_id', '2016-02-29-121813_remove_ttls', '2016-02-25-160900_remove_null_consumer_id', '2016-02-29-121813_remove_ttls', '2016-02-25-160900_remove_null_consumer_id', '2016-02-29-121813_remove_ttls', '2016-02-25-160900_remove_null_consumer_id', '2016-02-29-121813_remove_ttls', '2016-02-25-160900_remove_null_consumer_id', '2016-02-29-121813_remove_ttls', '2016-02-25-160900_remove_null_consumer_id', '2016-02-29-121813_remove_ttls', '2016-02-25-160900_remove_null_consumer_id', '2016-02-29-121813_remove_ttls', '2016-02-25-160900_remove_null_consumer_id', '2016-02-29-121813_remove_ttls', '2016-02-25-160900_remove_null_consumer_id', '2016-02-29-121813_remove_ttls', '2016-02-25-160900_remove_null_consumer_id', '2016-02-29-121813_remove_ttls', '2016-02-25-160900_remove_null_consumer_id', '2016-02-29-121813_remove_ttls', '2016-02-25-160900_remove_null_consumer_id', '2016-02-29-121813_remove_ttls', '2016-02-25-160900_remove_null_consumer_id', '2016-02-29-121813_remove_ttls', '2016-02-25-160900_remove_null_consumer_id', '2016-02-29-121813_remove_ttls', '2016-02-25-160900_remove_null_consumer_id', '2016-02-29-121813_remove_ttls', '2016-02-25-160900_remove_null_consumer_id', '2016-02-29-121813_remove_ttls']

It looks like when the migration table gets 'bad' aka no longer sorted correctly or a duplicate gets inserted, etc. It will keep running, and then writing to the table, any new migrations over and over again.

[hatchetation]

@thefosk you can see our duplicateschema_migrations in the gist I linked earlier

[subnetmarco]

Thank you - I wonder if this problem is caused by a wrong consistency setting in the client when doing the migration in Cassandra.

Are you able to consistently replicate this migration problem from scratch? Because if that is the case, I wonder if setting

consistency: ALL https://github.com/Mashape/kong/blob/master/kong.yml#L158 ​ in kong.yml just for the migration fixes this problem. If it indeed fixes the problem, we may want to manually override the setting to "ALL" during our migrations without requiring a manual change in the configuration file.

[chrisst]

As soon as a single duplicate row gets in there, then kong will run any new migration (with that key) every single time kong gets restarted. I'm pretty sure it's because kong expects the migrations table to only have a single entry per migration date/time key and it relies on them being in the correct order as well. So once it gets out of sync it keeps appending the migrations over and over as it will never find migration 3 in the 3rd spot ever again. If I'm reading this correctly, changing write consistency won't really have any effect once the data is bad, as the migration will never be detected as being run.

We're not sure exactly how we got into this state. One current theory is that we brought two new kong instances online at the same time and they may have tried running migrations simultaneously resulting in a race condition with multiple entries into a couple tables.

[gavinlove]

We had this same problem, brand new kong and cassandra, no apis configured on it yet. It started the first time, next time I started the server I got this error.

[ERR] Error during migration 2016-03-07-jwt-alg: [Invalid] Invalid column name algorithm because it conflicts with an existing column

 id                    | migrations
-----------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------
         rate-limiting |                                                                                                                                  ['2015-08-03-132400_init_ratelimiting']
                   acl |                                                                                                                                           ['2015-08-25-841841_init_acl']
        ip-restriction |                                                                                                                                              ['2016-05-24-remove-cache']
 response-ratelimiting |                                                                                                                               ['2015-08-21_init_response-rate-limiting']
                oauth2 |                  ['2015-08-03-132400_init_oauth2', '2015-08-24-215800_cascade_delete_index', '2016-02-29-435612_remove_ttl', '2016-04-14-283949_serialize_redirect_uri']
                   jwt |                                                                                                                                                  ['2015-06-09-jwt-auth']
                  core | ['2015-01-12-175310_skeleton', '2015-01-12-175310_init_schema', '2015-11-23-817313_nodes', '2016-02-25-160900_remove_null_consumer_id', '2016-02-29-121813_remove_ttls']
   request-transformer |                                                                                                                           ['2016-03-10-160000_req_trans_schema_changes']
              key-auth |                                                                                                                                       ['2015-07-31-172400_init_keyauth']
               galileo |                                                                                                                          ['2016-04-15_galileo-import-mashape-analytics']

DESCRIBE kong.jwt_secrets;

CREATE TABLE kong.jwt_secrets (
    id uuid PRIMARY KEY,
    algorithm text,
    consumer_id uuid,
    created_at timestamp,
    key text,
    rsa_public_key text,
    secret text
) WITH bloom_filter_fp_chance = 0.01
    AND caching = '{"keys":"ALL", "rows_per_partition":"NONE"}'
    AND comment = ''
    AND compaction = {'class': 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy'}
    AND compression = {'sstable_compression': 'org.apache.cassandra.io.compress.LZ4Compressor'}
    AND dclocal_read_repair_chance = 0.1
    AND default_time_to_live = 0
    AND gc_grace_seconds = 864000
    AND max_index_interval = 2048
    AND memtable_flush_period_in_ms = 0
    AND min_index_interval = 128
    AND read_repair_chance = 0.0
    AND speculative_retry = '99.0PERCENTILE';
CREATE INDEX jwt_secrets_secret_idx ON kong.jwt_secrets (secret);
CREATE INDEX jwt_secrets_key_idx ON kong.jwt_secrets (key);
CREATE INDEX jwt_secrets_consumer_id_idx ON kong.jwt_secrets (consumer_id);

[gavinlove]

I fixed it by just reseting the database as there was nothing in it anyway.

kong migrations reset --config /etc/kong/kong.yml

kong migrations up --config  /etc/kong/kong.yml

and now my schema is like this

id                    | migrations
-----------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------
         rate-limiting |                                                                                                                                  ['2015-08-03-132400_init_ratelimiting']
                   acl |                                                                                                                                           ['2015-08-25-841841_init_acl']
            basic-auth |                                                                                                                                     ['2015-08-03-132400_init_basicauth']
  response-transformer |                                                                                                                          ['2016-03-10-160000_resp_trans_schema_changes']
        ip-restriction |                                                                                                                                              ['2016-05-24-remove-cache']
 response-ratelimiting |                                                                                                                               ['2015-08-21_init_response-rate-limiting']
                oauth2 |                  ['2015-08-03-132400_init_oauth2', '2015-08-24-215800_cascade_delete_index', '2016-02-29-435612_remove_ttl', '2016-04-14-283949_serialize_redirect_uri']
                   jwt |                                                                                                                            ['2015-06-09-jwt-auth', '2016-03-07-jwt-alg']
                  core | ['2015-01-12-175310_skeleton', '2015-01-12-175310_init_schema', '2015-11-23-817313_nodes', '2016-02-25-160900_remove_null_consumer_id', '2016-02-29-121813_remove_ttls']
   request-transformer |                                                                                                                           ['2016-03-10-160000_req_trans_schema_changes']
             hmac-auth |                                                                                                                                      ['2015-09-16-132400_init_hmacauth']
              key-auth |                                                                                                                                       ['2015-07-31-172400_init_keyauth']
               galileo |                                                                                                                          ['2016-04-15_galileo-import-mashape-analytics']

[premdass]

I am also facing the same problem. I am starting 2 kong nodes together (using Kube Replication controller) , and i am consistently facing the issue, even on the clean cassandra datastore..

[someshmishra]

Hi guys,

I am also facing a similar problem and unable to start kong.

kong start [INFO] kong 0.8.3 [INFO] Using configuration: /etc/kong/kong.yml [INFO] Setting working directory to /usr/local/kong [INFO] database...........cassandra contact_points=127.0.0.1:9042 data_centers= ssl=verify=false enabled=false port=9042 timeout=5000 replication_strategy=SimpleStrategy keyspace=kong replication_factor=1 consistency=ONE [INFO] Migrating core (cassandra) [INFO] Leaving cluster.. [ERR] Error during migration 2016-02-25-160900_remove_null_consumer_id: config=Plugin "mashape-analytics" not found [ERR] Could not start Kong

[tobilg]

Same problem here, started the first time, but after each restart I'm facing different schema migration error messages in different stages. Started with a clean Cassandra install and an empty keyspace.

Sample error:

[INFO] kong 0.8.3
[INFO] Using configuration: /etc/kong/kong.yml
[INFO] Setting working directory to /usr/local/kong
[INFO] database...........cassandra contact_points=192.168.200.166:9042 data_centers= port=9042 ssl=verify=false enabled=false keyspace=kong123 timeout=5000 replication_strategy=SimpleStrategy replication_factor=2 consistency=ONE
[INFO] Migrating jwt (cassandra)
[INFO] Leaving cluster..
[ERR] Error during migration 2016-03-07-jwt-alg: [Invalid] Invalid column name algorithm because it conflicts with an existing column
[ERR] Could not start Kong

[premdass]

Are you starting more than one server at the same time? i had faced this problem when i tried starting more than 1 at the same time

On 11 August 2016 at 13:19, Tobi notifications@github.com wrote:

Same problem here, started the first time, but after each restart I'm facing different schema migration error messages in different stages. Started with a clean Cassandra install and an empty keyspace.

Sample error:

[INFO] kong 0.8.3 [INFO] Using configuration: /etc/kong/kong.yml [INFO] Setting working directory to /usr/local/kong [INFO] database...........cassandra contact_points=192.168.200.166:9042 data_centers= port=9042 ssl=verify=false enabled=false keyspace=kong123 timeout=5000 replication_strategy=SimpleStrategy replication_factor=2 consistency=ONE [INFO] Migrating jwt (cassandra) [INFO] Leaving cluster.. [ERR] Error during migration 2016-03-07-jwt-alg: [Invalid] Invalid column name algorithm because it conflicts with an existing column [ERR] Could not start Kong

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/Mashape/kong/issues/1118#issuecomment-239158169, or mute the thread https://github.com/notifications/unsubscribe-auth/AE2nTqize-SWaE6xxy177hsrncicUoBkks5qeyFngaJpZM4H9HAU .

[tobilg]

Initially, I was trying to start a Kong cluster on DC/OS... As it didn't work, I tried with single instances and brand new keyspaces. But that didn't help unfortunately.

[thibaultcha]

Please keep in mind the migration step should be run from a single node, preferably as separate command: kong migrations up and then several nodes can start at once. Considering this resolved for now.

[chrisst]

I think running the migration from multiple machines is only one of the ways that schema_migrations might get into a bad state. But what could be considered a root issue is that if the migration table gets a bad entry in it for any reason then each restart of kong will make the migrations table progressively more corrupt.

It would be nice to have the migrations recognize when the table has gotten into a bad state (either duplicate entries or entries in the wrong order) and raise an appropriate error. Or it would just scan the table for any occurrences of a migration (and not just in the order that it was expected) and not attempt to re-run a migration that has already been run even if it was in an unexpected place.

[thibaultcha]

We're planning on improving the migrations system before 1.0. For now we'll have to do with it...

[revandarth]

@thibaultcha I am facing the same issue with kong 0.9.9, I run kong-app that pointing to cassandra on openshift platform, it was failing during the migration

I tried running the following commands as suggested above, but no luck.

`sh-4.2$ kong migrations up

2017/02/15 10:36:08 [warn] 53#0: 2 [lua] log.lua:22: warn(): No cluster infos in shared dict cassandra, context: ngx.timer 2017/02/15 10:36:08 [warn] 53#0: 2 [lua] log.lua:22: warn(): Setting host 10.0.31.9:9042 as DOWN. Next retry in: 1000ms., context: ngx.timer migrating core for keyspace kong core migrated up to: 2015-01-12-175310_init_schema core migrated up to: 2015-11-23-817313_nodes core migrated up to: 2016-02-25-160900_remove_null_consumer_id core migrated up to: 2016-02-29-121813_remove_ttls migrating request-transformer for keyspace kong request-transformer migrated up to: 2016-03-10-160000_req_trans_schema_changes migrating rate-limiting for keyspace kong rate-limiting migrated up to: 2015-08-03-132400_init_ratelimiting rate-limiting migrated up to: 2016-07-25-471385_ratelimiting_policies migrating jwt for keyspace kong jwt migrated up to: 2015-06-09-jwt-auth Error: Error during migration 2016-03-07-jwt-alg: [Invalid] Invalid column name algorithm because it conflicts with an existing column `

sh-4.2$ kong migrations list 2017/02/15 10:52:03 [warn] 154#0: *2 [lua] log.lua:22: warn(): No cluster infos in shared dict cassandra, context: ngx.timer Executed migrations for keyspace 'kong': rate-limiting: 2015-08-03-132400_init_ratelimiting, 2015-08-03-132400_init_ratelimiting, 2016-07-25-471385_ratelimiting_policies request-transformer: 2016-03-10-160000_req_trans_schema_changes jwt: 2015-06-09-jwt-auth core: 2015-01-12-175310_skeleton, 2015-01-12-175310_init_schema, 2015-01-12-175310_init_schema, 2015-11-23-817313_nodes, 2015-11-23-817313_nodes, 2016-02-25-160900_remove_null_consumer_id, 2016-02-25-160900_remove_null_consumer_id, 2016-02-29-121813_remove_ttls

Note: I have cassandra cluster(3 nodes) running on openshift platform. from cassandra node cqlsh :

id | migrations ---------------------+------------------------------------------------------------------------------------------------------ rate-limiting | ['2015-08-03-132400_init_ratelimiting', '2015-08-03-132400_init_ratelimiting', '2016-07-25-471385_ratelimiting_policies', '2016-07-25-471385_ratelimiting_policies', '2016-07-25-471385_ratelimiting_policies', '2016-07-25-471385_ratelimiting_policies', '2016-07-25-471385_ratelimiting_policies', '2016-07-25-471385_ratelimiting_policies'] jwt | ['2015-06-09-jwt-auth'] core | ['2015-01-12-175310_skeleton', '2015-01-12-175310_init_schema', '2015-01-12-175310_init_schema', '2015-11-23-817313_nodes', '2015-11-23-817313_nodes', '2016-02-25-160900_remove_null_consumer_id', '2016-02-25-160900_remove_null_consumer_id', '2016-02-29-121813_remove_ttls', '2015-11-23-817313_nodes', '2016-02-25-160900_remove_null_consumer_id', '2016-02-29-121813_remove_ttls', '2015-11-23-817313_nodes', '2016-02-25-160900_remove_null_consumer_id', '2016-02-29-121813_remove_ttls', '2015-11-23-817313_nodes', '2016-02-25-160900_remove_null_consumer_id', '2016-02-29-121813_remove_ttls', '2015-11-23-817313_nodes', '2016-02-25-160900_remove_null_consumer_id', '2016-02-29-121813_remove_ttls', '2015-11-23-817313_nodes', '2016-02-25-160900_remove_null_consumer_id', '2016-02-29-121813_remove_ttls', '2015-11-23-817313_nodes', '2016-02-25-160900_remove_null_consumer_id', '2015-11-23-817313_nodes', '2016-02-25-160900_remove_null_consumer_id', '2015-11-23-817313_nodes', '2016-02-25-160900_remove_null_consumer_id', '2016-02-25-160900_remove_null_consumer_id', '2015-11-23-817313_nodes', '2016-02-25-160900_remove_null_consumer_id'] request-transformer | ['2016-03-10-160000_req_trans_schema_changes'] can you please help me to understand this behavior? Thanks in advance

[chrisst]

@revandarth Right now you're migrations table is in a corrupted state, and unfortunately without manual intervention the table will never recover. You can see that 2016-02-25-160900_remove_null_consumer_id has been run several times and it will keep running over and over each time you restart kong.

The reason is because there is a list of migrations and it compares the migration name at index[n] with the migration name coming back from the migrations table also at index[n]. When it doesn't find it at the right index it will re-run the migration but it will continue to append the migration name on the end of the array. So it will never actually 'fix' the index in question.

The solution we've arrived at in this position is to go in and de-dupe the migration table by hand, paying attention to the migration files per plugin (rate-limiting in this case) and making sure the order is correct. You might be able to simply truncate your migrations and, assuming the migrations are mostly idempotent, it will re-build the table the next time you start kong up.

[revandarth]

@ThorArakis Thanks for the response

I have tried with fresh cassandra cluster, still i have the same issue

/usr/local/share/lua/5.1/kong/cmd/start.lua:37: /usr/local/share/lua/5.1/kong/cmd/start.lua:18: Error during migration 2016-03-07-jwt-alg: [Invalid] Invalid column name algorithm because it conflicts with an existing column stack traceback:

from cassandra cqlsh: cqlsh> select * from kong.schema_migrations;

id | migrations ---------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------- rate-limiting | ['2015-08-03-132400_init_ratelimiting', '2016-07-25-471385_ratelimiting_policies'] jwt | ['2015-06-09-jwt-auth'] core | ['2015-01-12-175310_skeleton', '2015-01-12-175310_init_schema', '2015-11-23-817313_nodes', '2016-02-25-160900_remove_null_consumer_id', '2016-02-29-121813_remove_ttls'] request-transformer | ['2016-03-10-160000_req_trans_schema_changes']

FYI: Environment openshift yaml files: https://github.com/revandarth/kong-cassandra-openshift

@ThorArakis mind helping me to understand where it is wrong? Thanks - Revan

[chrisst]

@revandarth You should look into your table definitions specifically for your jwt table. It looks like 2016-03-07-jwt-alg is trying to add a column that already exists. You can either rip out that column so the migration will succeed or manually add 2016-03-07-jwt-alg to the migrations table. Either way I recommend looking at the migrations so you know what they are doing, it will help you troubleshoot why each individual migration is failing.

[revandarth]

@ThorArakis make sense. Thank you so much for detailed information.

[DaveAvigad]

simply delete any existing kong schema from your data base (don’t forget to BACKUP your data before if it’s not empty...) and run again kong migrations up

[subnetmarco]

Note that deleting the schema will also delete all the data.