Error trying load services in aws

[CriztianiX]

Hey. i am using the library inside a web app using lapis framework (running on openresty) and i get this error when i try to use it:

ls: cannot access /usr/local/openresty/lualib//usr/local/share/lua/5.1/lua-aws/services//definitions/dynamodb*: No such file or directory

Any idea?

[CriztianiX]

The complete error:

2015/05/13 19:12:06 [notice] 24324#0: signal 17 (SIGCHLD) received 2015/05/13 19:12:06 [notice] 24324#0: unknown process 24342 exited with code 0 2015/05/13 19:12:06 [notice] 24324#0: signal 17 (SIGCHLD) received 2015/05/13 19:12:06 [notice] 24324#0: unknown process 24343 exited with code 0 2015/05/13 19:12:06 [error] 24324#0: _66 lua coroutine: runtime error: /usr/local/share/lua/5.1/lua-aws/services/base.lua:32: dynamodb:no api defined. stack traceback: coroutine 0: [C]: in function 'assert' /usr/local/share/lua/5.1/lua-aws/services/base.lua:32: in function 'build_apis' /usr/local/share/lua/5.1/lua-aws/services/base.lua:9: in function 'initialize' /usr/local/share/lua/5.1/lua-aws/class.lua:56: in function 'new' /usr/local/share/lua/5.1/lua-aws/core.lua:11: in function 'initialize' /usr/local/share/lua/5.1/lua-aws/class.lua:56: in function 'get_instance' ./apps/users.lua:137: in function <./apps/users.lua:99> coroutine 1: [C]: in function 'resume' /usr/local/share/lua/5.1/lapis/application.lua:652: in function 'handler' /usr/local/share/lua/5.1/lapis/application.lua:393: in function 'resolve' /usr/local/share/lua/5.1/lapis/application.lua:402: in function </usr/local/share/lua/5.1/lapis/application.lua:400> [C]: in function 'xpcall' /usr/local/share/lua/5.1/lapis/application.lua:400: in function 'dispatch' /usr/local/share/lua/5.1/lapis/nginx.lua:200: in function 'serve' content_by_lua(nginx.conf.compiled:23):2: in function <content_by_lua(nginx.conf.compiled:23):1>, client: xxx.xxx.xxx.xxx, server: , request: "POST /api/v1/users/1/register_device HTTP/1.1", host: xxx.xxx.xxx.xxx, referrer: "http://xxx.xxx.xxx.xxx/api/v1" 2015/05/13 19:12:06 [notice] 24324#0: *66 [lua] application.lua:464: handle_error(): [500] POST /api/v1/users/1/register_device - {}, client: xxx.xxx.xxx.xxx, server: , request: "POST /api/v1/users/1/registerdevice HTTP/1.1", host: "xx.xxx.xxx.xxx", referrer: "http://xxx.xxx.xxx.xxx/api/v1" ls: cannot access /usr/local/openresty/lualib//usr/local/share/lua/5.1/lua-aws/services//definitions/dynamodb: No such file or directory 2015/05/13 19:12:06 [notice] 24324#0: signal 17 (SIGCHLD) received

[CriztianiX]

The problems seems to be in util.lua#242, the function file_exists return true when the file does not exists

I devel environment i get: "testing: /usr/local/openresty/lualib//usr/local/share/lua/5.1/lua-aws/services//definitions/" = false

but in ec2 i get; "testing: /usr/local/openresty/lualib//usr/local/share/lua/5.1/lua-aws/services//definitions/" = true

[umegaya]

hi, because I didn't write any test for dynamoDB, I write the one to confirm it is not dynamoDB only error. at https://github.com/umegaya/lua-aws/tree/umegaya/issue/11 and in my local environment, dynamoDB API seems to work fine so I believe this is some environment problem.

as you say, if /usr/local/openresty/lualib//usr/local/share/lua/5.1/lua-aws/services//definitions/ does not exist in ec2 environment, create the one and manually copy json files under definitions/ can solve the problem?

thanks.

[CriztianiX]

Hey.. there is no a problem with dinamodb, It's a generic problem with all services... If i comment self.DynamoDB = require('lua-aws.services.dynamodb').new(self) in core.lua, the error jumps to next line (self.EC2 = require('lua-aws.services.ec2').new(self)), etc.

The problem is than strange reason file_exists in util.lua is returning true if the file does not exists.

[umegaya]

ok, I think I got what you report. util.search_path_with_lua_config returns wrong path due to file_exists wrongly return true fo /usr/local/openresty/lualib//usr/local/share/lua/5.1/lua-aws/services//definitions/, before try to test /usr/local/share/lua/5.1/lua-aws/services//definitions/ is exists or not, right?

if so, still I think there is some problem in environment, because util.file_exist(path) is literally equivalent to [ -r $path ] in shell script.

can you execute following shell script and give me output? if [ -r "/usr/local/openresty/lualib//usr/local/share/lua/5.1/lua-aws/services//definitions/" ]; then echo '1' else echo '0' fi

thanks

[umegaya]

by the way, have you used docker? I try to use lua-aws with openresty on docker, and there is no problem with using it. also I push this image to docker registry as umegaya/openresty, so that you can try it in your machine like following. (you need to set some environment variables) I hope this would help.

dokyougemusu-no-MacBook-Pro:openresty iyatomi$ docker run --rm -ti -e AWS_ACCESS_KEY=$AWS_ACCESS_KEY -e AWS_SECRET_KEY=$AWS_SECRET_KEY -e EC2_URL=$EC2_URL umegaya/openresty bash
root@ce5fd0bd4fdb:/# nginx
nginx: [alert] lua_code_cache is off; this will hurt performance in /opt/openresty/nginx/conf/nginx.conf:30
root@ce5fd0bd4fdb:/# curl http://localhost:8080/
[true,{"TableNames":{}}]
root@ce5fd0bd4fdb:/# cat /opt/openresty/nginx/
app/  conf/ html/ logs/ sbin/ 
root@ce5fd0bd4fdb:/# cat /opt/openresty/nginx/app/dynamo.lua 
local cjson = require 'cjson'
local AWS = require 'lua-aws'
assert(os.getenv('AWS_ACCESS_KEY'))
AWS = AWS.new({
    accessKeyId = os.getenv('AWS_ACCESS_KEY'),
    secretAccessKey = os.getenv('AWS_SECRET_KEY')
})

local ok,r = AWS.DynamoDB:api():listTables({
    Limit = 3,
})

ngx.say(cjson.encode({ok, r}))

root@ce5fd0bd4fdb:/# 

[lordnynex]

I am experiencing the same issue. In my case, I am running ec2 describeinstances. Sometimes the request succeeds but there are many notices in the logs:

2015/08/12 16:52:32 [notice] 18192#0: signal 17 (SIGCHLD) received
2015/08/12 16:52:32 [notice] 18192#0: unknown process 26280 exited with code 0
2015/08/12 16:52:32 [notice] 18192#0: signal 17 (SIGCHLD) received
2015/08/12 16:52:32 [notice] 18192#0: unknown process 26282 exited with code 0

Sometimes the request fails outright

2015/08/12 16:52:19 [error] 18192#0: *24 lua entry thread aborted: runtime error: /usr/local/share/lua/5.1/lua-aws/services/base.lua:32: ec2:no api defined.
stack traceback:
coroutine 0:
        [C]: in function 'require'
        /usr/local/share/lua/5.1/lapis/init.lua:15: in function 'serve'
        content_by_lua:2: in function <content_by_lua:1>, client: 127.0.0.1, server: , request: "GET /ec2 HTTP/1.1", host: "localhost:8080"
ls: cannot access /usr/local/openresty/lualib//usr/local/share/lua/5.1/lua-aws/services//definitions/ec2*: No such file or directory
2015/08/12 16:52:19 [notice] 18192#0: signal 17 (SIGCHLD) received

Sometimes the error references kinesis, however, I am not using kinesis.

2015/08/12 16:49:22 [notice] 18192#0: unknown process 25226 exited with code 0
2015/08/12 16:49:22 [notice] 18192#0: signal 17 (SIGCHLD) received
2015/08/12 16:49:22 [notice] 18192#0: unknown process 25228 exited with code 0
2015/08/12 16:49:22 [notice] 18192#0: signal 17 (SIGCHLD) received
2015/08/12 16:49:22 [notice] 18192#0: unknown process 25230 exited with code 0
2015/08/12 16:49:22 [error] 18192#0: *23 lua entry thread aborted: runtime error: /usr/local/share/lua/5.1/lua-aws/services/base.lua:32: kinesis:no api defined.
stack traceback:
coroutine 0:
        [C]: in function 'require'
        /usr/local/share/lua/5.1/lapis/init.lua:15: in function 'serve'
        content_by_lua:2: in function <content_by_lua:1>, client: 127.0.0.1, server: , request: "GET /ec2 HTTP/1.1", host: "localhost:8080"
ls: write error: Broken pipe
2015/08/12 16:49:22 [notice] 18192#0: signal 17 (SIGCHLD) received

[lordnynex]

Also, I'm curious what the motivation for subshell calls where? I think this module would be better served by LFS.

[umegaya]

@lordnynex thanks for your information! references kinesis is really strange... I will investigate again.

Also, I'm curious what the motivation for subshell calls where? I think this module would be better served by LFS.

motivation is keeping dependency as small as possible. but it seems that most of user of lua-aws runs it under the openresty environment, probably it is better using lfs at least if it is available. (like http_engine approach)

[umegaya]

@lordnynex with my openresty docker container umegaya/openresty, everything seems to work fine.

2015/08/12 16:49:22 [notice] 18192#0: signal 17 (SIGCHLD) received
2015/08/12 16:49:22 [notice] 18192#0: unknown process 25230 exited with code 0

even this kind log is not seen in /var/nginx/error.log. its like following.

2015/05/14 04:31:24 [notice] 44#0: signal process started
2015/05/14 04:31:24 [alert] 45#0: lua_code_cache is off; this will hurt performance in /opt/openresty/nginx/conf/nginx.conf:30
2015/08/13 09:27:03 [alert] 7#0: lua_code_cache is off; this will hurt performance in /opt/openresty/nginx/conf/nginx.conf:30

maybe I made complete mistake to build environment like yours. can you give me more detail about how to setup your environment?

or maybe better to modify lua-aws to use LFS if available. I will work on this...

[CriztianiX]

+1 LFS

[lordnynex]

I think the environment is ok. The only appreciable difference is that both @CriztianiX and I are using lapis so package.path would also include '.'.

sudo luarocks install lapis
mkdir ~/lapis-test && cd ~/lapis-test
lapis new

Now have a look at app.moon and make it

lapis = require "lapis"
local AWS = require ('lua-aws.init')

import capture_errors_json, yield_error from require "lapis.application"

AWS = AWS.new({
    accessKeyId = os.getenv('AWS_ACCESS_KEY'),
    secretAccessKey = os.getenv('AWS_SECRET_KEY')
})

class extends lapis.Application
  "/": =>
    local ok,r = AWS.EC2:api():describeInstances()

    yield_error r if not ok

    status: 200, json: r

Now run:

moonc app.moon
lapis server

and hit http://localhost:8080/ in a web browser. That should reproduce the issue.

It's important to note lapis wraps everything to capture errors gracefully and the amount of failing assertions in lua-aws would yield poor results. I'm still not confident in sub shell calls though. I think I will fork this and begin integrating LFS and cjson.

[umegaya]

@lordnynex @CriztianiX thank you for your information :D I managed to reproduce SIGCHLD issue. but still cannot repro "no api defined" one.

but it seems to be related with SIGCHLD issue, so I change json and fs modules to be pluggable and prefer to use cjson and lfs by default. (by #15)

if using lfs, guilty code path like util.file_exists or util.search_path_with_lua_config is not used. and at least in my environment, SIGCHLD issues are gone. can you try in your side? thanks.

[lordnynex]

@umegaya Sorry it took me so long. I've just tested your branch and so far everything looks good. Thanks for making this.

[CriztianiX]

Work for me too!

i've merged in my forked master! https://github.com/umegaya/lua-aws/pull/16

[umegaya]

@loadnynex I'm glad to here that. thank you for your information and suggestion. @CriztianiX sorry for taking so long time to fix your report, because of my ignorance, I did not notice the possibility that you used lapis :<

[CriztianiX]

Hey! I've notice what:

    preferred_engines: {
      json: "cjson",
      fs: require "lfs"
    }

runtime error: /usr/local/share/lua/5.1/lua-aws/services/base.lua:23: attempt to call field 'scandir' (a nil value)
stack traceback:
coroutine 0:
    /usr/local/share/lua/5.1/lua-aws/services/base.lua: in function 'build_apis'
    /usr/local/share/lua/5.1/lua-aws/services/base.lua:9: in function 'initialize'
    /usr/local/share/lua/5.1/lua-aws/class.lua:56: in function 'new'
    /usr/local/share/lua/5.1/lua-aws/core.lua:23: in function 'initialize'
    /usr/local/share/lua/5.1/lua-aws/class.lua:56: in function 'get_instance'

lfs no seems to have scandir function,! PD: Luafilesystem (lfs is installed and working in this env)

[lordnynex]

@CriztianiX yea this is just kindof a weird implementation issue. It took me a bit to realize you should just use:

preferred_engines: {
  json: "cjson"
  fs: "lfs"
}

EDIT:

For clarity, the message in the PR was misleading. It allows you to use your own module to provide scandir. In the case of LFS, it's now supported by this module directly. Take a look at https://github.com/umegaya/lua-aws/blob/master/lua-aws/engines/fs/lfs.lua to understand why/how.

[CriztianiX]

Very thanks! Now working with cjson and lfs!

This is realted to aws?

2015/08/20 04:23:05 [notice] 14877#0: signal 28 (SIGWINCH) received
2015/08/20 04:23:05 [notice] 14875#0: signal 28 (SIGWINCH) received
2015/08/20 04:23:29 [notice] 14877#0: signal 28 (SIGWINCH) received
2015/08/20 04:23:29 [notice] 14875#0: signal 28 (SIGWINCH) received

[umegaya]

@CriztianiX hi, since I made json and fs module pluggable, when you give preferred_engines to table, these tables need to have some function at specified key, like a concept of "interface" of Java, Go, etc... so directly giving lfs module to preferred_engines will not work. please see lua-aws/engines/fs/lfs.lua,os.lua, it provides scandir interface by using lfs API, os.execute respectively.

@lordnynex for openresty or lapis user, cjson and lfs is so common that this may seem to be verbose. but other users of lua-aws may not want to use lfs or cjson. (e.g. I want to use lua-aws with my own FFI yajl parser and fs module). so I think keeping these modules pluggable is still important for fitting various use-case. but I have to admit that current error message is not informative, I will do some improvement about it.

2015年8月21日(金) 4:51 Cristian Haunsen notifications@github.com:

Very thanks! Now working with cjson and lfs!

— Reply to this email directly or view it on GitHub https://github.com/umegaya/lua-aws/issues/13#issuecomment-133152101.