search

Kong / kong

🦍 The Cloud-Native API Gateway and AI Gateway.
https://konghq.com/install/#kong-community
Apache License 2.0
39.24k stars 4.81k forks source link

Runtime error in Cassandra #420

Closed subnetmarco closed 9 years ago

subnetmarco commented 9 years ago

Just got this from the logs:

Jul 22 15:09:42 45.79.129.126 Jul 22 22:09:42 kong-1 nginx: 2015/07/22 22:09:42 [error] 274#0: *2259 lua entry thread aborted: runtime error: /usr/local/share/lua/5.1/kong/dao/cassandra/apis.lua:16: attempt to call a nil value
stack traceback:
coroutine 0:
    /usr/local/share/lua/5.1/kong/dao/cassandra/apis.lua: in function 'find_all'
    /usr/local/share/lua/5.1/kong/resolver/access.lua:24: in function 'cb'
    /usr/local/share/lua/5.1/kong/tools/database_cache.lua:103: in function 'get_or_set'
    /usr/local/share/lua/5.1/kong/resolver/access.lua:97: in function 'find_api'
    /usr/local/share/lua/5.1/kong/resolver/access.lua:154: in function 'execute'
    /usr/local/share/lua/5.1/kong/resolver/handler.lua:22: in function 'access'
    /usr/local/share/lua/5.1/kong.lua:198: in function 'exec_plugins_access'
    access_by_lua(nginx.conf:87):1: in function <access_by_lua(nginx.conf:87):1>, client: 74.86.158.106, server: _, request: "HEAD / HTTP/1.1", host: "www.voilatheapp.com"
subnetmarco commented 9 years ago

Cannot replicate.

maxfi commented 9 years ago

I'm getting the same error running in Tutum (Docker 1.8.3):

2015/10/20 20:02:13 [error] 51#0: *7 lua entry thread aborted: runtime error: /usr/local/share/lua/5.1/kong/dao/cassandra/apis.lua:16: attempt to call a nil value                                                                              
stack traceback:                                                                
coroutine 0:                                                                    
        /usr/local/share/lua/5.1/kong/dao/cassandra/apis.lua: in function 'find_all'                                                                            
        /usr/local/share/lua/5.1/kong/resolver/access.lua:55: in function 'cb'  
        /usr/local/share/lua/5.1/kong/tools/database_cache.lua:109: in function 'get_or_set'                                                                    
        /usr/local/share/lua/5.1/kong/resolver/access.lua:177: in function 'find_api'                                                                           
        /usr/local/share/lua/5.1/kong/resolver/access.lua:198: in function 'execute'                                                                            
        /usr/local/share/lua/5.1/kong/resolver/handler.lua:22: in function 'access'                                                                             
        /usr/local/share/lua/5.1/kong.lua:191: in function 'exec_plugins_access'
        access_by_lua(nginx.conf:90):1: in function <access_by_lua(nginx.conf:90
):1>

Stack file:

cassandra:
  image: 'mashape/cassandra:latest'
  restart: always
  tags:
    - aux
kong:
  image: 'mashape/kong:latest'
  links:
    - cassandra
  ports:
    - '8000:8000'
    - '8001:8000'
  restart: always
  tags:
    - aux

Could this have something to do with the docker version?

nricklin commented 8 years ago

I just got this same error on kong 0.5.2, using the AWS AMI.

The logs show first a whole bunch of errors like this:

2015/12/07 16:54:31 [error] 25775#0: *30967142 [lua] responses.lua:61: cb(): Cassandra error: connection refused, client: XXX.YY.ZZ.AA, server: _, request: "GET /XXXXXXXXX HTTP/1.1", host: 
"XXXXXXXX"

and then endless errors like this:

2015/12/07 16:54:31 [error] 25775#0: *30967139 lua entry thread aborted: runtime error: /usr/local/share/lua/5.1/kong/dao/cassandra/apis.lua:16: attempt to call a nil value
stack traceback:
coroutine 0:
    /usr/local/share/lua/5.1/kong/dao/cassandra/apis.lua: in function 'find_all'
    /usr/local/share/lua/5.1/kong/resolver/access.lua:55: in function 'cb'
    /usr/local/share/lua/5.1/kong/tools/database_cache.lua:109: in function 'get_or_set'
    /usr/local/share/lua/5.1/kong/resolver/access.lua:177: in function 'find_api'
    /usr/local/share/lua/5.1/kong/resolver/access.lua:198: in function 'execute'
    /usr/local/share/lua/5.1/kong/resolver/handler.lua:22: in function 'access'
    /usr/local/share/lua/5.1/kong.lua:191: in function 'exec_plugins_access'
    access_by_lua(nginx.conf:90):1: in function <access_by_lua(nginx.conf:90):1>, client: XX.YY.ZZ.AA, server: _, request: "GET /XXXXXX HTTP/1.1", host: "XXXXXXX"

Here's what I did to remedy the situation:

  1. kill the cassandra process
  2. sudo service cassandra restart
  3. sudo  /usr/local/bin/kong restart

I'm not sure it was necessary to bounce cassandra

nricklin commented 8 years ago

Here's the Cassandra system.log:

ERROR [COMMIT-LOG-ALLOCATOR] 2015-12-07 16:54:02,762 StorageService.java:458 - Stopping gossiper
WARN  [COMMIT-LOG-ALLOCATOR] 2015-12-07 16:54:02,762 StorageService.java:364 - Stopping gossip by operator request
INFO  [COMMIT-LOG-ALLOCATOR] 2015-12-07 16:54:02,762 Gossiper.java:1439 - Announcing shutdown
INFO  [COMMIT-LOG-ALLOCATOR] 2015-12-07 16:54:02,763 StorageService.java:1904 - Node localhost/127.0.0.1 state jump to normal
INFO  [IndexSummaryManager:1] 2015-12-07 16:54:04,377 IndexSummaryManager.java:256 - Redistributing index summaries
ERROR [COMMIT-LOG-ALLOCATOR] 2015-12-07 16:54:04,860 StorageService.java:468 - Stopping native transport
INFO  [COMMIT-LOG-ALLOCATOR] 2015-12-07 16:54:04,863 Server.java:208 - Stop listening for CQL clients
ERROR [COMMIT-LOG-ALLOCATOR] 2015-12-07 16:54:04,867 CommitLog.java:486 - Failed managing commit log segments. Commit disk failure policy is stop; terminating thread
java.lang.AssertionError: attempted to delete non-existing file CommitLog-5-1447358040710.log
        at org.apache.cassandra.io.util.FileUtils.deleteWithConfirm(FileUtils.java:122) ~[apache-cassandra-2.2.3.jar:2.2.3]
        at org.apache.cassandra.io.util.FileUtils.deleteWithConfirm(FileUtils.java:149) ~[apache-cassandra-2.2.3.jar:2.2.3]
        at org.apache.cassandra.db.commitlog.CommitLogSegment.discard(CommitLogSegment.java:314) ~[apache-cassandra-2.2.3.jar:2.2.3]
        at org.apache.cassandra.db.commitlog.CommitLogSegmentManager$2.run(CommitLogSegmentManager.java:374) ~[apache-cassandra-2.2.3.jar:2.2.3]
        at org.apache.cassandra.db.commitlog.CommitLogSegmentManager$1.runMayThrow(CommitLogSegmentManager.java:155) ~[apache-cassandra-2.2.3.jar:2.2.3]
        at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) [apache-cassandra-2.2.3.jar:2.2.3]
        at java.lang.Thread.run(Thread.java:745) [na:1.7.0_91]
nricklin commented 8 years ago

Some more investigating:

This may be the issue: https://issues.apache.org/jira/browse/CASSANDRA-10377

The 0.5.2 kong AMI uses cassandra 2.2.3. Looks like the issue is fixed in Cassandra 2.2.4 or 3.0.0.