apinf / platform

Apinf - Open source API management platform with multi proxy and protocol support
https://apinf.com/
European Union Public License 1.1
74 stars 35 forks source link

SyncedCron tor Analytics data throws exception #3560

Open matleppa opened 6 years ago

matleppa commented 6 years ago

Background

SyncedCron is run every half an hour to get data from ElasticSearch in order to write it to local MongoDB.

Problem

In case there is no living connection, an exception occurs:

I20180906-16:00:02.693(3) (packages\logging.js:101) SyncedCron: Exception "Analytics data: GmcdSMJ3XmpGH8Dx9" Error: No Living connections at sendReqWithConnection (C:\Users\matleppa\api\apinf\platform\node_modules\elasticsearch\src\lib\transport.js:207:15) at next (C:\Users\matleppa\api\apinf\platform\node_modules\elasticsearch\src\lib\connection_pool.js:213:7) at nextTickCallbackWith0Args (node.js:489:9) at process._tickDomainCallback (node.js:459:13) => awaited here: at Promise.await (C:\Users\matleppa\AppData\Local.meteor\packages\promise\0.9.0\npm\node_modules\meteor-promise\promiseserver.js:60:12) at [object Object]..extend.apply (packages/ddp-server/livedataserver.js:1635:22) at [object Object]..extend.call (packages/ddp-server/livedata_server.js:1604:17) at _loop (apinf_packages/analytics/server/store.js:90:31) at [object Object].Meteor.methods.storeAnalyticsData (apinf_packages/analytics/server/store.js:75:41) at [object Object].methodMap.(anonymous function) (apinf_packages/core/capture_exception/server/wrap_meteor_methods.js:17:30) at maybeAuditArgumentChecks (packages/ddp-server/livedata_server.js:1768:12) at packages/ddp-server/livedata_server.js:1686:15 at [object Object].EVp.withValue (packages\meteor.js:1135:15) at packages/ddp-server/livedataserver.js:1684:36 at [object Object]..extend.applyAsync (packages/ddp-server/livedataserver.js:1683:12) at [object Object]..extend.apply (packages/ddp-server/livedataserver.js:1622:26) at [object Object]..extend.call (packages/ddp-server/livedata_server.js:1604:17) at [object Object].Meteor.methods.proxyBackendAnalyticsData (apinf_packages/analytics/server/store.js:47:16) at [object Object].methodMap.(anonymous function) (apinf_packages/core/capture_exception/server/wrap_meteor_methods.js:17:30) at maybeAuditArgumentChecks (packages/ddp-server/livedata_server.js:1768:12) at packages/ddp-server/livedata_server.js:1686:15 at [object Object].EVp.withValue (packages\meteor.js:1135:15) at packages/ddp-server/livedataserver.js:1684:36 at [object Object]..extend.applyAsync (packages/ddp-server/livedataserver.js:1683:12) at [object Object]..extend.apply (packages/ddp-server/livedataserver.js:1622:26) at [object Object]..extend.call (packages/ddp-server/livedata_server.js:1604:17) at Object.SyncedCron.add.job (apinf_packages/analytics/server/cron.js:41:16) at packages\percolate_synced-cron.js:242:26 at scheduleTimeout (packages\percolate_synced-cron.js:290:7) at packages\percolate_synced-cron.js:338:42 at [object Object].EVp.withValue (packages\meteor.js:1135:15) at packages\meteor.js:452:25 at runWithEnvironment (packages\meteor.js:1189:24) => awaited here: at Promise.await (C:\Users\matleppa\AppData\Local.meteor\packages\promise\0.9.0\npm\node_modules\meteor-promise\promiseserver.js:60:12) at [object Object]..extend.apply (packages/ddp-server/livedataserver.js:1635:22) at [object Object]..extend.call (packages/ddp-server/livedata_server.js:1604:17) at [object Object].Meteor.methods.proxyBackendAnalyticsData (apinf_packages/analytics/server/store.js:47:16) at [object Object].methodMap.(anonymous function) (apinf_packages/core/capture_exception/server/wrap_meteor_methods.js:17:30) at maybeAuditArgumentChecks (packages/ddp-server/livedata_server.js:1768:12) at packages/ddp-server/livedata_server.js:1686:15 at [object Object].EVp.withValue (packages\meteor.js:1135:15) at packages/ddp-server/livedataserver.js:1684:36 at [object Object]..extend.applyAsync (packages/ddp-server/livedataserver.js:1683:12) at [object Object]..extend.apply (packages/ddp-server/livedataserver.js:1622:26) at [object Object]..extend.call (packages/ddp-server/livedata_server.js:1604:17) at Object.SyncedCron.add.job (apinf_packages/analytics/server/cron.js:41:16) at packages\percolate_synced-cron.js:242:26 at scheduleTimeout (packages\percolate_synced-cron.js:290:7) at packages\percolate_synced-cron.js:338:42 at [object Object].EVp.withValue (packages\meteor.js:1135:15) at packages\meteor.js:452:25 at runWithEnvironment (packages\meteor.js:1189:24) => awaited here: at Promise.await (C:\Users\matleppa\AppData\Local.meteor\packages\promise\0.9.0\npm\node_modules\meteor-promise\promiseserver.js:60:12) at [object Object]..extend.apply (packages/ddp-server/livedataserver.js:1635:22) at [object Object]..extend.call (packages/ddp-server/livedata_server.js:1604:17) at Object.SyncedCron.add.job (apinf_packages/analytics/server/cron.js:41:16) at packages\percolate_synced-cron.js:242:26 at scheduleTimeout (packages\percolate_synced-cron.js:290:7) at packages\percolate_synced-cron.js:338:42 at [object Object].EVp.withValue (packages\meteor.js:1135:15) at packages\meteor.js:452:25 at runWithEnvironment (packages\meteor.js:1189:24)

It seems, that there is missing a checking for case empty data is returned when handling the response from ElasticSearch.

lc3t35 commented 5 years ago

No try / catch on Meteor.call in analytics/server/store.js

const response = Meteor.call('getElasticsearchData', elasticsearchHost, query);
// Get data that are grouped by requested path
const aggregatedDataResponse = response.aggregations.group_by_request_path.buckets;

what happens if response is undefined ? or if if crashes in "getElasticsearchData" in elasticsearch/server/methods.js

const esClient = new ElasticSearch.Client({ host });
return await esClient.search(queryParams);

if esClient is undefined, you can't search -> error is not handled within code.