search

strongloop / strong-arc

StrongLoop Arc has been replaced by API Connect. We recommend Arc users move to the Essentials edition of API Connect. If you have questions, please email reachsl@us.ibm.com.
http://strongloop.com/
Other
114 stars 36 forks source link

discovery chokes on 30+ model import PostgreSQL #558

Open seanbrookes opened 9 years ago

seanbrookes commented 9 years ago

Trying to import all (~30) tables vis PostgreSQL connector in composer the screen stays spinning and never recovers. It seems the Angular UI stacks up a bunch of api calls and then uses $q.all() to post them to the backed for model definition creation. Somewhere in that process it stalls and never finishes resolving the stack. You can see the http requests getting resolved in the network tab of chrome dev tools

seanbrookes commented 9 years ago

here is a sample from the terminal:

(node) warning: possible EventEmitter memory leak detected. 11 listeners added. Use emitter.setMaxListeners() to increase limit.

Trace

    at WriteStream.addListener (events.js:160:15)

    at WriteStream.Readable.on (_stream_readable.js:691:33)

    at WriteStream.once (events.js:185:8)

    at Socket.Readable.pipe (_stream_readable.js:543:8)

    at ModelConstructor.DataSourceDefinition.invokeMethodInWorkspace (/usr/local/lib/node_modules/strong-studio/node_modules/loopback-workspace/models/data-source-definition.js:285:16)

    at ModelConstructor.DataSourceDefinition.discoverModelDefinition (/usr/local/lib/node_modules/strong-studio/node_modules/loopback-workspace/models/data-source-definition.js:142:8)

    at SharedMethod.invoke (/usr/local/lib/node_modules/strong-studio/node_modules/loopback-workspace/node_modules/loopback/node_modules/strong-remoting/lib/shared-method.js:225:19)

    at HttpContext.invoke (/usr/local/lib/node_modules/strong-studio/node_modules/loopback-workspace/node_modules/loopback/node_modules/strong-remoting/lib/http-context.js:255:12)

    at /usr/local/lib/node_modules/strong-studio/node_modules/loopback-workspace/node_modules/loopback/node_modules/strong-remoting/lib/remote-objects.js:477:9

    at execStack (/usr/local/lib/node_modules/strong-studio/node_modules/loopback-workspace/node_modules/loopback/node_modules/strong-remoting/lib/remote-objects.js:348:7)

(node) warning: possible EventEmitter memory leak detected. 11 listeners added. Use emitter.setMaxListeners() to increase limit.

Trace

    at WriteStream.addListener (events.js:160:15)

    at WriteStream.Readable.on (_stream_readable.js:691:33)

    at Socket.Readable.pipe (_stream_readable.js:478:8)

    at ModelConstructor.DataSourceDefinition.invokeMethodInWorkspace (/usr/local/lib/node_modules/strong-studio/node_modules/loopback-workspace/models/data-source-definition.js:285:16)

    at ModelConstructor.DataSourceDefinition.discoverModelDefinition (/usr/local/lib/node_modules/strong-studio/node_modules/loopback-workspace/models/data-source-definition.js:142:8)

    at SharedMethod.invoke (/usr/local/lib/node_modules/strong-studio/node_modules/loopback-workspace/node_modules/loopback/node_modules/strong-remoting/lib/shared-method.js:225:19)

    at HttpContext.invoke (/usr/local/lib/node_modules/strong-studio/node_modules/loopback-workspace/node_modules/loopback/node_modules/strong-remoting/lib/http-context.js:255:12)

    at /usr/local/lib/node_modules/strong-studio/node_modules/loopback-workspace/node_modules/loopback/node_modules/strong-remoting/lib/remote-objects.js:477:9

    at execStack (/usr/local/lib/node_modules/strong-studio/node_modules/loopback-workspace/node_modules/loopback/node_modules/strong-remoting/lib/remote-objects.js:348:7)

    at /usr/local/lib/node_modules/strong-studio/node_modules/loopback-workspace/node_modules/loopback/lib/application.js:319:13

(node) warning: possible EventEmitter memory leak detected. 11 listeners added. Use emitter.setMaxListeners() to increase limit.

Trace

    at WriteStream.addListener (events.js:160:15)

    at WriteStream.Readable.on (_stream_readable.js:691:33)

    at Socket.Readable.pipe (_stream_readable.js:493:8)

    at ModelConstructor.DataSourceDefinition.invokeMethodInWorkspace (/usr/local/lib/node_modules/strong-studio/node_modules/loopback-workspace/models/data-source-definition.js:285:16)

    at ModelConstructor.DataSourceDefinition.discoverModelDefinition (/usr/local/lib/node_modules/strong-studio/node_modules/loopback-workspace/models/data-source-definition.js:142:8)

    at SharedMethod.invoke (/usr/local/lib/node_modules/strong-studio/node_modules/loopback-workspace/node_modules/loopback/node_modules/strong-remoting/lib/shared-method.js:225:19)

    at HttpContext.invoke (/usr/local/lib/node_modules/strong-studio/node_modules/loopback-workspace/node_modules/loopback/node_modules/strong-remoting/lib/http-context.js:255:12)

    at /usr/local/lib/node_modules/strong-studio/node_modules/loopback-workspace/node_modules/loopback/node_modules/strong-remoting/lib/remote-objects.js:477:9

    at execStack (/usr/local/lib/node_modules/strong-studio/node_modules/loopback-workspace/node_modules/loopback/node_modules/strong-remoting/lib/remote-objects.js:348:7)

    at /usr/local/lib/node_modules/strong-studio/node_modules/loopback-workspace/node_modules/loopback/lib/application.js:319:13

(node) warning: possible EventEmitter memory leak detected. 11 listeners added. Use emitter.setMaxListeners() to increase limit.

Trace

    at WriteStream.addListener (events.js:160:15)

    at WriteStream.Readable.on (_stream_readable.js:691:33)

    at WriteStream.once (events.js:185:8)

    at Socket.Readable.pipe (_stream_readable.js:538:8)

    at ModelConstructor.DataSourceDefinition.invokeMethodInWorkspace (/usr/local/lib/node_modules/strong-studio/node_modules/loopback-workspace/models/data-source-definition.js:285:16)

    at ModelConstructor.DataSourceDefinition.discoverModelDefinition (/usr/local/lib/node_modules/strong-studio/node_modules/loopback-workspace/models/data-source-definition.js:142:8)

    at SharedMethod.invoke (/usr/local/lib/node_modules/strong-studio/node_modules/loopback-workspace/node_modules/loopback/node_modules/strong-remoting/lib/shared-method.js:225:19)

    at HttpContext.invoke (/usr/local/lib/node_modules/strong-studio/node_modules/loopback-workspace/node_modules/loopback/node_modules/strong-remoting/lib/http-context.js:255:12)

    at /usr/local/lib/node_modules/strong-studio/node_modules/loopback-workspace/node_modules/loopback/node_modules/strong-remoting/lib/remote-objects.js:477:9

    at execStack (/usr/local/lib/node_modules/strong-studio/node_modules/loopback-workspace/node_modules/loopback/node_modules/strong-remoting/lib/remote-objects.js:348:7)
seanbrookes commented 9 years ago

screenshot of the browser console: image

seanbrookes commented 9 years ago

link to offending schema definition: https://drive.google.com/a/strongloop.com/file/d/0B7cRRPpbtkFKc0x5ejdNcVdaekk/view?usp=sharing

raymondfeng commented 9 years ago

I tried the server side using the same set of tables using the following code:

var async = require('async');
var server = require('./server');
var dataSource = server.dataSources.accountDB;

dataSource.discoverModelDefinitions({ schema: 'public' },
  function(err, tables) {
    if (err) {
      throw err;
    }
    async.each(tables, function(table, done) {
      dataSource.discoverModelProperties(table.name, {schema: table.owner},
        function(err, props) {
          console.log(props);
          done();
        });
    }, function(err) {
      console.log('done');
      dataSource.disconnect();
    });
  });

Everything works as expected in reasonable time.

I think the problem is probably on how we use Angular to call multiple model discovery APIs in parallel.

seanbrookes commented 9 years ago

I suspect it is the way the ui is posting all of the queries at once?

$q.all(pStack).then(function(result) { var resolveArray = [];
angular.forEach(result, function(response) { resolveArray.push(response.
status); }); deferred.resolve(resolveArray); });

pStack is an array of discovery calls to create modeldefinitions and propertydefinitions. The cod never recovers from this $q call if there are too many tables.

I was thinking we might throttle/chunk the requests to the server into blocks of 10 tables at a time or something separated by a few milliseconds to give the server a chance to 'breathe'

On Thu, Dec 4, 2014 at 4:53 PM, Raymond Feng notifications@github.com wrote:

I tried the server side using the same set of tables using the following code:

var async = require('async');var server = require('./server');var dataSource = server.dataSources.accountDB;

dataSource.discoverModelDefinitions({ schema: 'public' }, function(err, tables) { if (err) { throw err; } async.each(tables, function(table, done) { dataSource.discoverModelProperties(table.name, {schema: table.owner}, function(err, props) { console.log(props); done(); }); }, function(err) { console.log('done'); dataSource.disconnect(); }); });

Everything works as expected in reasonable time.

I think the problem is probably on how we use Angular to call multiple model discovery APIs in parallel.

— Reply to this email directly or view it on GitHub https://github.com/strongloop/strong-arc/issues/558#issuecomment-65731323 .

StrongLoop http://strongloop.com/* makes it easy to develop APIs http://strongloop.com/mobile-application-development/loopback/ in Node, plus get DevOps capabilities http://strongloop.com/node-js-performance/strongops/ like monitoring, debugging and clustering*.

raymondfeng commented 9 years ago

The other option is to add a new DataSourceDefinition.discoverModelDefinitions() API to discover an array of models in one call.

Thanks,

Raymond Feng Co-Founder and Architect @ StrongLoop, Inc.

StrongLoop http://strongloop.com/ makes it easy to develop APIs http://strongloop.com/mobile-application-development/loopback/ in Node, plus get DevOps capabilities http://strongloop.com/node-js-performance/strongops/ like monitoring, debugging and clustering.

On Dec 5, 2014, at 8:11 AM, Sean Brookes notifications@github.com wrote:

I suspect it is the way the ui is posting all of the queries at once? 

$q.all(pStack).then(function(result) { var resolveArray = []; 
angular.forEach(result, function(response) { resolveArray.push(response. 
status); }); deferred.resolve(resolveArray); });

pStack is an array of discovery calls to create modeldefinitions and propertydefinitions. The cod never recovers from this $q call if there are too many tables.

I was thinking we might throttle/chunk the requests to the server into blocks of 10 tables at a time or something separated by a few milliseconds to give the server a chance to 'breathe'

On Thu, Dec 4, 2014 at 4:53 PM, Raymond Feng notifications@github.com wrote:

I tried the server side using the same set of tables using the following code:

var async = require('async');var server = require('./server');var dataSource = server.dataSources.accountDB;

dataSource.discoverModelDefinitions({ schema: 'public' }, function(err, tables) { if (err) { throw err; } async.each(tables, function(table, done) { dataSource.discoverModelProperties(table.name, {schema: table.owner}, function(err, props) { console.log(props); done(); }); }, function(err) { console.log('done'); dataSource.disconnect(); }); });

Everything works as expected in reasonable time.

I think the problem is probably on how we use Angular to call multiple model discovery APIs in parallel.

— Reply to this email directly or view it on GitHub https://github.com/strongloop/strong-arc/issues/558#issuecomment-65731323 .

StrongLoop http://strongloop.com/* makes it easy to develop APIs http://strongloop.com/mobile-application-development/loopback/ in Node, plus get DevOps capabilities http://strongloop.com/node-js-performance/strongops/ like monitoring, debugging and clustering*. — Reply to this email directly or view it on GitHub https://github.com/strongloop/strong-arc/issues/558#issuecomment-65811504.

altsang commented 9 years ago

note: this has been brought up externally by another prospect, this affects all discovery - please see these notes: https://docs.google.com/a/strongloop.com/document/d/1-SdZPqJbh-0mDVEhJeHpki-IW9AfkIQ9HrpkP-SkSJ4/edit

note: UX issues from feedback in the doc will be handled separately