DC/OS package doesn't start with a network error (connection refused)

[mimmus]

Describe the bug Package available on DC/OS Catalog doesn't start with a network error (connection refused to internal address during startup).

To Reproduce Steps to reproduce the behavior:

1. Go to 'Catalog' and install "predator' package
2. Select appropriate backend (SQLITE doesn't seem to work, I used MySQL and it is OK)
3. Look at logs during startup of container

Expected behavior Predator service starts automatically and I'm able connect to Web UI

Logs {"level":30,"time":1603874944320,"pid":1,"hostname":"coll-dcos9","name":"predator","msg":"Predator listening on port 1884"} {"level":30,"time":1603874944323,"pid":1,"hostname":"coll-dcos9","name":"predator","msg":"Checking http://predator.marathon.l4lb.thisdcos.directory:80/config to verify predator-runners will be able connect to Predator"} {"level":50,"time":1603874945398,"pid":1,"hostname":"coll-dcos9","name":"predator","name":"RequestError","message":"Error: connect ECONNREFUSED 11.31.104.67:80","cause":{"errno":"ECONNREFUSED","code":"ECONNREFUSED","syscall":"connect","address":"11.31.104.67","port":80},"error":{"errno":"ECONNREFUSED","code":"ECONNREFUSED","syscall":"connect","address":"11.31.104.67","port":80},"options":{"json":true,"simple":false,"resolveWithFullResponse":true,"timeout":5000,"uri":"http://predator.marathon.l4lb.thisdcos.directory:80/config","method":"GET","transform2xxOnly":false},"stack":"RequestError: Error: connect ECONNREFUSED 11.31.104.67:80\n at new RequestError (/usr/node_modules/request-promise-core/lib/errors.js:14:15)\n at Request.plumbing.callback (/usr/node_modules/request-promise-core/lib/plumbing.js:87:29)\n at Request.RP$callback [as _callback] (/usr/node_modules/request-promise-core/lib/plumbing.js:46:31)\n at self.callback (/usr/node_modules/request/request.js:185:22)\n at Request.emit (events.js:310:20)\n at Request.onRequestError (/usr/node_modules/request/request.js:877:8)\n at ClientRequest.emit (events.js:322:22)\n at Socket.socketErrorListener (_http_client.js:426:9)\n at Socket.emit (events.js:310:20)\n at emitErrorNT (internal/streams/destroy.js:92:8)\n at emitErrorAndCloseNT (internal/streams/destroy.js:60:3)\n at processTicksAndRejections (internal/process/task_queues.js:84:21)","type":"Error","msg":"Encountered an error during start up"}

Version:

• Predator DC/OS package: [e.g. 1.0.0]
• Database [e.g. MySQL]

Additional context I know that DC/OS does not have a large use but it is actually our container orchestration platform. I have some in-depth knowledge of this product but I was not able to find a workaround.

[enudler]

Hi @mimmus can you try set env var: SKIP_INTERNAL_ADDRESS_CHECK = "true"

[enudler]

@NivLipetz we will need to fix our DC/OS deployment

[NivLipetz]

@enudler no problem will get on it today

[mimmus]

I was able to set env var using (deprecated) Marathon endpoint (http://<cluster-ip>:8080). Service started successfully!

Now I will try to get into the product...

Thanks for very quick help

[enudler]

great @mimmus let us know if you any further help

[mimmus]

Yes, UI home page: https://<cluster-ip>/service/predator/ui/ is blank. I'm able to see its source but result is a blank web page.

[enudler]

Yes it's an issue we have with webpack where the UI has to be at https://service/ui

so there are two options

1. create DNS record for Predator for example https://predator.dcos and access https://predator.dcos/ui
2. build predator image like this docker build --build-arg BUCKET_PATH=/predator . -t predator

more info here https://github.com/Zooz/predator#opening-the-predator-ui

[mimmus]

I'm trying to expose UI on marathon-lb instead of accessing it by admin-router (https://<cluster-ip>/service/....)

[mimmus]

If you add following env vars:

    HAPROXY_GROUP=external
    HAPROXY_0_MODE=http
    HAPROXY_0_VHOST=predator.yourdomain.com

(and configure DNS to resolve predator.yourdomain.com) it works simply using: http://predator.yourdomain.com/ui/

During first tests, I'm having failures on all jobs with:

/usr/app/utils/verifier.js:39
        throw error;
        ^

Error: Missing mandatory variables: RUN_ID
    at Object.module.exports.verifyEnvironmentVars (/usr/app/utils/verifier.js:37:21)
    at Object.<anonymous> (/usr/app/app.js:2:29)
    at Module._compile (module.js:653:30)
    at Object.Module._extensions..js (module.js:664:10)
    at Module.load (module.js:566:32)
    at tryModuleLoad (module.js:506:12)
    at Function.Module._load (module.js:498:3)
    at Function.Module.runMain (module.js:694:10)
    at startup (bootstrap_node.js:204:16)
    at bootstrap_node.js:625:3

but my knowledge of Predator is almost zero.

[enudler]

@mimmus The version of the predator-runner and predator are not matching, verify that docker image predator is zooz/predator:1.5 (in marathon docker image) after predator starts go to setting in the ui and set predator-runner image: zooz/predator-runner:1.5

[mimmus]

Set this too!

Now an error when I run a simple test:

Error: Request failed with status code 422
Description: 422 - {"message":"Object is not valid","details":[{"path":"","errors":["Additional properties are not allowed, but found properties '0' and '1' and '2' and '3' and '4' and '5' and '6' and '7'."]}]}

probably POST-ing the Job request to DC/OS Metronome.

[enudler]

ohh to many bugs in our dcos deployment but actually, this is fixed on master so please use

zooz/predator:latest zooz/predator-runner:latest

[mimmus]

No problem, I understand.

'latest' on both solves the issue but Job test still fails with:

{"name":"predator-runner","hostname":"coll-dcos14","pid":1,"level":30,"runner_config":{"environment":"test","cluster":"default","logLevel":"info","statsInterval":30,"testId":"c477d3c9-e73c-49a2-9d22-2b28eb02dc11","jobId":"22ba2e7e-9dac-4bbe-b7ef-2cdf4439cb0b","jobType":"load_test","predatorUrl":"http://predator.marathon.l4lb.thisdcos.directory/v1","duration":60,"arrivalRate":1,"maxVusers":250,"httpPoolSize":250,"delayRunnerMs":0,"reportId":"55a3ed63-220e-42a8-8069-358a4584a2cd","predatorVersion":"1.5.4","containerId":"20201028132746RNUgn"},"msg":"Initialized test runner","time":"2020-10-28T12:28:05.114Z","v":0}
{"name":"predator-runner","hostname":"coll-dcos14","pid":1,"level":30,"method":"GET","url":"http://predator.marathon.l4lb.thisdcos.directory/v1/tests/c477d3c9-e73c-49a2-9d22-2b28eb02dc11","headers":{"x-runner-id":"20201028132746RNUgn"},"msg":"GET test file","time":"2020-10-28T12:28:05.116Z","v":0}
{"name":"predator-runner","hostname":"coll-dcos14","pid":1,"level":30,"msg":"Request to http://predator.marathon.l4lb.thisdcos.directory/v1/tests/c477d3c9-e73c-49a2-9d22-2b28eb02dc11 succeeded with status code 200","time":"2020-10-28T12:28:05.141Z","v":0}
{"name":"predator-runner","hostname":"coll-dcos14","pid":1,"level":30,"test_file":{"id":"c477d3c9-e73c-49a2-9d22-2b28eb02dc11","name":"Test1","description":"","type":"basic","updated_at":"2020-10-28T12:17:09.000Z","revision_id":"51398502-8fcf-45d8-975f-763ec28cfc85","artillery_test":{"config":{"target":"https://company.cerved.com","plugins":{"expect":{}}},"scenarios":[{"name":"Scenario 1","weight":100,"flow":[{"get":{"url":"/","headers":{},"capture":[],"expect":[{"statusCode":"200"}],"forever":true}}]}]}},"msg":"Retrieved test file successfully","time":"2020-10-28T12:28:05.142Z","v":0}
{"name":"predator-runner","hostname":"coll-dcos14","pid":1,"level":50,"msg":"Request to http://predator.marathon.l4lb.thisdcos.directory/v1/tests/c477d3c9-e73c-49a2-9d22-2b28eb02dc11/reports/55a3ed63-220e-42a8-8069-358a4584a2cd/subscribe failed on the 1 attempt with error 400 - \"{\"message\":\"Input validation error\",\"validation_errors\":[\"headers/x-runner-id should match format \\\"uuid\\\"\"]}\"","time":"2020-10-28T12:28:05.148Z","v":0}
{"name":"predator-runner","hostname":"coll-dcos14","pid":1,"level":50,"err":{"message":"400 - \"{\"message\":\"Input validation error\",\"validation_errors\":[\"headers/x-runner-id should match format \\\"uuid\\\"\"]}\"","name":"StatusCodeError","stack":"StatusCodeError: 400 - \"{\"message\":\"Input validation error\",\"validation_errors\":[\"headers/x-runner-id should match format \\\"uuid\\\"\"]}\"\n    at buildResponse (/usr/node_modules/requestxn/lib/index.js:147:23)\n    at retry.then.response (/usr/node_modules/requestxn/lib/index.js:76:31)\n    at <anonymous>\n    at process._tickCallback (internal/process/next_tick.js:189:7)"},"msg":"Test failed","time":"2020-10-28T12:28:05.149Z","v":0}
{"name":"predator-runner","hostname":"coll-dcos14","pid":1,"level":50,"msg":"Request to http://predator.marathon.l4lb.thisdcos.directory/v1/tests/c477d3c9-e73c-49a2-9d22-2b28eb02dc11/reports/55a3ed63-220e-42a8-8069-358a4584a2cd/stats failed on the 1 attempt with error 500 - \"{\"message\":\"Internal server error\"}\"","time":"2020-10-28T12:28:05.162Z","v":0}
{"name":"predator-runner","hostname":"coll-dcos14","pid":1,"level":50,"msg":"Request to http://predator.marathon.l4lb.thisdcos.directory/v1/tests/c477d3c9-e73c-49a2-9d22-2b28eb02dc11/reports/55a3ed63-220e-42a8-8069-358a4584a2cd/stats failed on the 2 attempt with error 500 - \"{\"message\":\"Internal server error\"}\"","time":"2020-10-28T12:28:06.176Z","v":0}
{"name":"predator-runner","hostname":"coll-dcos14","pid":1,"level":50,"msg":"Request to http://predator.marathon.l4lb.thisdcos.directory/v1/tests/c477d3c9-e73c-49a2-9d22-2b28eb02dc11/reports/55a3ed63-220e-42a8-8069-358a4584a2cd/stats failed on the 3 attempt with error 500 - \"{\"message\":\"Internal server error\"}\"","time":"2020-10-28T12:28:08.184Z","v":0}
{"name":"predator-runner","hostname":"coll-dcos14","pid":1,"level":50,"err":{"message":"500 - \"{\"message\":\"Internal server error\"}\"","name":"StatusCodeError","stack":"StatusCodeError: 500 - \"{\"message\":\"Internal server error\"}\"\n    at buildResponse (/usr/node_modules/requestxn/lib/index.js:147:23)\n    at retry.then.response (/usr/node_modules/requestxn/lib/index.js:76:31)\n    at <anonymous>\n    at process._tickCallback (internal/process/next_tick.js:189:7)"},"msg":"Failed to update reporter of test failure","time":"2020-10-28T12:28:08.184Z","v":0}

[enudler]

Seems that the runner is old and not compatible with the predator version metronome sometimes doesn't pull images on jobs

What I suggest is to:

1. verify you are using zooz/predator-runner:latest in the ui setting screen
2. if yes, pull that docker on your machine, tag, push to some registery dcos has access to, and change the predator-runner

[mimmus]

1. yes
2. I pulled zooz/predator-runner:latest, tagged it to our-registry/predator-runner:1 and to pushed, after I changed the predator-running setting by the UI but I get always same error "500"

[enudler]

@mimmus i will check it today on our DC/OS cluster and update afaik it should work although long time no one touched it

[NivLipetz]

@mimmus i see the problem, will build a fix and update you once it's built, should take 15 min +/-. The fix will be built in the zooz/predator:latest docker image

[mimmus]

OK, take your time. I will indagate furtherly if I missed something

[NivLipetz]

New image with fix is built and pushed to our Dockerhub registry, you can redeploy predator now. Waiting to hear the test runs successfully 😄

[mimmus]

It works!

Now I can pass the ball to interested teams.

Thank you for excellent support, we used a Github issue like a chat but it's not really important :-)

[NivLipetz]

Great to hear, we will close this issue and implement the internal address check differently so that the app doesn't crash.

We would be happy if you joined our slacks workspace for updates on the project and for any further questions and suggestions you might have.