Erros listing API's

[rogeriollacerda]

Hi, After create something like 1000 api's at Umbrella, I can't acess the api list via web interface anymore.

My nginx log (current):

2016-07-15T20:42:49.96257 2016/07/15 20:42:49 [error] 152#0: [lua] interval_lock.lua:41: timeout_exec(): timeout exec pcall failed: ...e/current/src/api-umbrella/proxy/jobs/load_db_config.lua:43: attempt to compare number with nil, context: ngx.timer 2016-07-15T20:42:50.26315 2016/07/15 20:42:50 [error] 152#0: [lua] interval_lock.lua:41: timeout_exec(): timeout exec pcall failed: ...e/current/src/api-umbrella/proxy/jobs/load_db_config.lua:43: attempt to compare number with nil, context: ngx.timer

Browser error popup:

DataTables warning: table id=ember679 - Ajax error. For more information about this error, please see http://datatables.net/tn/7

Browser web developer error:

WARNING: New Ember version detected. URL hash monkey patch possibly no longer needed or broken. Check for compatibility. :8001/web-assets/admin-360139f49b27c28ad0eeabcbb3dcf025.js:339 DEBUG: ------------------------------- :8001/web-assets/admin-360139f49b27c28ad0eeabcbb3dcf025.js:339 DEBUG: Ember : 1.7.1 :8001/web-assets/admin-360139f49b27c28ad0eeabcbb3dcf025.js:339 DEBUG: Handlebars : 1.3.0 :8001/web-assets/admin-360139f49b27c28ad0eeabcbb3dcf025.js:339 DEBUG: jQuery : 1.11.2 admin-360139f49b27c28ad0eeabcbb3dcf025.js:339 DEBUG: Ember Model : 0.0.11 admin-360139f49b27c28ad0eeabcbb3dcf025.js:339 DEBUG: ------------------------------- admin-360139f49b27c28ad0eeabcbb3dcf025.js:339 Attempting URL transition to /apis admin-360139f49b27c28ad0eeabcbb3dcf025.js:339 Transition #0: application: calling beforeModel hook admin-360139f49b27c28ad0eeabcbb3dcf025.js:339 Transition #0: application: calling deserialize hook admin-360139f49b27c28ad0eeabcbb3dcf025.js:339 Transition #0: application: calling afterModel hook admin-360139f49b27c28ad0eeabcbb3dcf025.js:339 Transition #0: apis: calling beforeModel hook admin-360139f49b27c28ad0eeabcbb3dcf025.js:339 Transition #0: apis: calling deserialize hook admin-360139f49b27c28ad0eeabcbb3dcf025.js:339 Transition #0: apis: calling afterModel hook admin-360139f49b27c28ad0eeabcbb3dcf025.js:339 Transition #0: apis.index: calling beforeModel hook admin-360139f49b27c28ad0eeabcbb3dcf025.js:339 Transition #0: apis.index: calling deserialize hook admin-360139f49b27c28ad0eeabcbb3dcf025.js:339 Transition #0: apis.index: calling afterModel hook admin-360139f49b27c28ad0eeabcbb3dcf025.js:339 Transition #0: Resolved all models on destination route; finalizing transition. admin-360139f49b27c28ad0eeabcbb3dcf025.js:339 Transitioned into 'apis.index' admin-360139f49b27c28ad0eeabcbb3dcf025.js:339 Transition #0: TRANSITION COMPLETE. admin-360139f49b27c28ad0eeabcbb3dcf025.js:339 DEBUG: For more advanced debugging, install the Ember Inspector from https://chrome.google.com/webstore/detail/ember-inspector/bmdblncegkenkacieihfhpjfppoconhi https://10.13.12.59:8001/api-umbrella/v1/apis.json?draw=1&columns%5B0%5D%5B…art=0&length=50&search%5Bvalue%5D=&search%5Bregex%5D=false&_=1468614708689 Failed to load resource: the server responded with a status of 404 (Not Found) Navigated to https://10.13.12.59:8001/admin/ admin-360139f….js:339 WARNING: New Ember version detected. URL hash monkey patch possibly no longer needed or broken. Check for compatibility. admin-360139f….js:339 DEBUG: ------------------------------- admin-360139f….js:339 DEBUG: Ember : 1.7.1 admin-360139f….js:339 DEBUG: Handlebars : 1.3.0 admin-360139f….js:339 DEBUG: jQuery : 1.11.2 admin-360139f….js:339 DEBUG: Ember Model : 0.0.11 admin-360139f….js:339 DEBUG: ------------------------------- admin-360139f….js:339 Attempting URL transition to /apis admin-360139f….js:339 Transition #0: application: calling beforeModel hook admin-360139f….js:339 Transition #0: application: calling deserialize hook admin-360139f….js:339 Transition #0: application: calling afterModel hook admin-360139f….js:339 Transition #0: apis: calling beforeModel hook admin-360139f….js:339 Transition #0: apis: calling deserialize hook admin-360139f….js:339 Transition #0: apis: calling afterModel hook admin-360139f….js:339 Transition #0: apis.index: calling beforeModel hook admin-360139f….js:339 Transition #0: apis.index: calling deserialize hook admin-360139f….js:339 Transition #0: apis.index: calling afterModel hook admin-360139f….js:339 Transition #0: Resolved all models on destination route; finalizing transition. admin-360139f….js:339 Transitioned into 'apis.index' admin-360139f….js:339 Transition #0: TRANSITION COMPLETE. admin-360139f….js:339 DEBUG: For more advanced debugging, install the Ember Inspector from https://chrome.google.com/webstore/detail/ember-inspector/bmdblncegkenkacieihfhpjfppoconhi admin-360139f….js:26 GET https://10.13.12.59:8001/api-umbrella/v1/apis.json?draw=1&columns%5B0%5D%5B…art=0&length=50&search%5Bvalue%5D=&search%5Bregex%5D=false&_=1468614747744 404 (Not Found)

[rogeriollacerda]

In this scenario, I'm running 6 instances of router and web-app running on exclusive instance. When that error was ocurred, all of our instances stoped work.

[GUI]

The exact place it's erroring is a bit odd, but I suspect what's happening is that with 1,000 API Backends, it's exceeded the amount of memory we allocate in nginx to storing the API backend configuration data. Do you perhaps see anything further up in /var/log/api-umbrella/nginx/current about errors related to no memory or something else memory related?

In any case, if this is the issue, I think it should be easily solvable by allocating more memory to this. Basically, you have to have enough memory to store the full JSON representation of all the API backend objects. To increase this memory, in your /etc/api-umbrella/api-umbrella.yml file, you'll want to add this config:

nginx:
  shared_dicts:
    active_config:
      size: 5m

You can adjust the size as needed. By default, we allocate 600KB, which I think is usually enough for somewhere between 500-1000 API backends (I forget the exact math).

Let us know if this doesn't solve the issue. But if it does solve it, that would also be useful to know, since we need to add some better documentation around this (what the defaults allow for, and how to increase it). We might also see if we can somehow detect this situation a bit more cleanly so the error messages are more obvious.

[rogeriollacerda]

@GUI Thanks for help.

You're right. I've change my configs on api-umbrella.yml and didn't have the problem again. Our first charge will insert more than 2000 api's. Next monday we'll make a test with 25k req/sec running 6 instances of api-umbrella. That is our charge in a commom day. My configs now:

 nginx:
   workers: 15
   shared_dicts:
     active_config:
       size: 5000m
     api_users:
       size: 10m
     dns_cache:
       size: 900k
     resolved_hosts:
       size: 500k
     upstream_checksums:
       size: 5m
 gatekeeper:
   workers: 10

[GUI]

Glad that did the trick! Although, 5000MB of memory for active_config is probably a bit much, even for 2000+ APIs. For reference, I did some quick math based on our production API configuration, and it looks like each API Backend consumes on average 2.6KB of memory (you can find actual memory use in your own servers by using systemtap and ngx-shm, if you're interested). That can definitely vary, but even assuming each API Backend takes 10KB (for a very inflated number), 30MB for active_config should be plenty for up to 3000 API backends.

But thanks for filing this issue, and confirming that fixed it. This is definitely an area we should better document.

And if you are able to, and don't mind sharing details, we'd of course be interested to hear how your benchmarking goes (either here or on the mailing list). But let us know if you run into any other issues in the meantime.