search

CMPUT401Group / analytic-engine

Bleeding edge of analytic things and stuff
1 stars 0 forks source link

Can't complete an entailment search #2

Open deanbittner opened 7 years ago

deanbittner commented 7 years ago

Script: node --max-old-space-size=4096 ./dist/cli.js entailment_search --goal-metric invidi.webapp.localhost_localdomain.request.total_response_time.mean --goal-metric-time-begin 05:00_20160917 --goal-metric-time-end 12:00_20160917 --time-begin 00:00_20160917 --time-end 23:00_20160917 --iteration-count 10000 --out /tmp/temp-result.json --dashboard-out /tmp/dashboard.json

Expected outcome: Completed entailment search

Actual: Aborted due to timeout in Graphite as below. Tried decreasing the interval and iterations without effect. Suspect a bad metric or metric name? I will look at the results of the graphite query for all names in the database and attach it to see if anything obvious jumps out.

Console output of failure follows --

GraphiteAdapter.renderAllMetrics: 4 in 11 GraphiteAdapter.renderAllMetrics: 5 in 11 GraphiteAdapter.renderAllMetrics: 6 in 11 GraphiteAdapter.renderAllMetrics: 7 in 11 GraphiteAdapter.renderAllMetrics: 8 in 11 GraphiteAdapter.renderAllMetrics: 9 in 11 GraphiteAdapter.renderAllMetrics: 10 in 11 /home/dean/analytic-engine/node_modules/sync-request/node_modules/http-response-object/index.js:40 throw err; ^

Error: Server responded with status code 504: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">

504 Gateway Timeout

Gateway Timeout

The gateway did not receive a timely response from the upstream server or application.

Apache/2.4.10 (Ubuntu) Server at 10.120.1.240 Port 81
at Response.getBody (/home/dean/analytic-engine/node_modules/sync-request/node_modules/http-response-object/index.js:34:15)
at RenderAPIAdapter.render (/home/dean/analytic-engine/dist/render-api-adapter.js:64:35)
at /home/dean/analytic-engine/dist/graphite-adapter.js:66:46
at Array.forEach (native)
at GraphiteAdapter.renderAllMetrics (/home/dean/analytic-engine/dist/graphite-adapter.js:65:21)
at RLAdapter.acquireMetrics (/home/dean/analytic-engine/dist/rl-adapater.js:51:23)
at RLAdapter.train (/home/dean/analytic-engine/dist/rl-adapater.js:65:12)
at /home/dean/analytic-engine/dist/cli.js:325:45
at Object.<anonymous> (/home/dean/analytic-engine/dist/cli.js:353:4)
at Module._compile (module.js:409:26)
deanbittner commented 7 years ago

Here's the full and rather 'unclean' list of metrics from the server, zipped. metrics_list_12122016.csv.zip

deanbittner commented 7 years ago

Maybe there's just too many and it takes longer than the graphite timeout of 360 (set in apache wigs settings) seconds to retrieve? Will try a longer time.

deanbittner commented 7 years ago

Increased the timeout in the apache settings --

WSGIDaemonProcess _graphite processes=5 threads=5 display-name='%{GROUP}' inactivity-timeout=1200 user=_graphite group=_graphite

inactivity-timeout set to 1200 seconds and got past this error, now have another error, will file separately

deanbittner commented 7 years ago

Can be closed.