search

kdebisschop / rundeck-rancher-node-plugin

Manage Rancher-controlled Docker containers in Rundeck
Apache License 2.0
5 stars 0 forks source link

Cannot run cmd on container after upgraded it #32

Closed mashwino closed 4 years ago

mashwino commented 4 years ago

Hi, here under a problem I've got with this job. (Rundeck 3.2.3 docker image) The goal of the job is to upgrade the container and launch a command on it Job type: Dispatch to node step 1: Node step => command: hostname step 2: Node step => Upgrade Node (Start before stopping: whatever) step 3: Workflow step => Local command: sleep 10 step 4: Node step => command: hostname

Choose your node and launch, every thing is going find till the last step where you've got: Failed dispatching to node Node_selected: java.lang.IllegalArgumentException: Expected URL scheme 'http' or 'https' but no colon was found

If, instead of making the last step on the node newly upgraded we do it on an other node, every thing is fine.

Thank you.

kdebisschop commented 4 years ago

I suspect this is because you have node caching enabled - which is not necessarily a bad thing. But if so, you need to follow the upgrade by

  1. refreshing the nodes
  2. run the subsequent step in a job reference and not directly

I do not think this is an issue with the plugin or even anything the plugin can provide a workaround for.

mashwino commented 4 years ago

Hi, could be a cache problem, but I tried disabling it, waiting 1min, launching a job to refresh it before the action, running the step on a subsequent job ... well, almost all combinations and still the same error. On the docker logs output, I always get this: [2020-03-01 10:42:10,090] INFO web.requests "POST /execution/tailExecutionOutput" 10.42.229.125 http admin form 10 ? [application/json;charset=utf-8] (Mozilla/5.0 (X11; Linux x86_64; rv:73.0) Gecko/20100101 Firefox/73.0) [2020-03-01 10:42:10,828] INFO web.requests "GET /execution/ajaxExecState/8" 10.42.229.125 http admin form 8 ? [application/json;charset=utf-8] (Mozilla/5.0 (X11; Linux x86_64; rv:73.0) Gecko/20100101 Firefox/73.0) ... [2020-03-01 10:42:17,573] INFO web.requests "POST /execution/tailExecutionOutput" 10.42.229.125 http admin form 8 ? [application/json;charset=utf-8] (Mozilla/5.0 (X11; Linux x86_64; rv:73.0) Gecko/20100101 Firefox/73.0) [2020-03-01 10:42:18,421] WARN authorization.RuleEvaluator RuleEvaluator - Evaluating Decision for: res<name:access.key, type:storage, path:keys/rancher/access.key> subject action env<rundeck: auth:env:application:rundeck>: authorized: false: No context matches subject or environment => REJECTED_NO_SUBJECT_OR_ENV_FOUND (0ms) [2020-03-01 10:42:18,423] INFO storage.events get file keys/rancher/access.key - {Rundeck-content-type=application/octet-stream, Rundeck-auth-modified-username=admin, Rundeck-content-modify-time=2020-03-01T10:04:40Z, Rundeck-content-crea tion-time=2020-03-01T10:04:40Z, Rundeck-content-size=20, Rundeck-auth-created-username=admin, Rundeck-key-type=private, Rundeck-content-mask=content} [2020-03-01 10:42:18,423] WARN authorization.RuleEvaluator RuleEvaluator - Evaluating Decision for: res<name:secret.key, type:storage, path:keys/rancher/secret.key> subject action env<rundeck: auth:env:application:rundeck>: authorized: false: No context matches subject or environment => REJECTED_NO_SUBJECT_OR_ENV_FOUND (0ms) [2020-03-01 10:42:18,425] INFO storage.events get file keys/rancher/secret.key - {Rundeck-content-type=application/octet-stream, Rundeck-auth-modified-username=admin, Rundeck-content-modify-time=2020-03-01T10:05:01Z, Rundeck-content-crea tion-time=2020-03-01T10:05:01Z, Rundeck-content-size=40, Rundeck-auth-created-username=admin, Rundeck-key-type=private, Rundeck-content-mask=content} [2020-03-01 10:42:18.479] ERROR ExecutionUtilService --- [eduler_Worker-8] Execution failed: 8 in project rancher: [Workflow result: , step failures: {3=Dispatch failed on 1 nodes: [NAS_music-koel-front-1: Unknown: java.lang.IllegalArgume ntException: Expected URL scheme 'http' or 'https' but no colon was found]}, Node failures: {ENV_teststack-1=[Unknown: java.lang.IllegalArgumentException: Expected URL scheme 'http' or 'https' but no colon was found]}, status: fail ed]

Fresh install, nothing else than this job. Thank you.

kdebisschop commented 4 years ago

Did you try the sequence I recommended? You need to understand that you have started with a node that is captured at the beginning of the job and once you upgrade that node it no longer exists. This is definitely the case for nodes that represent containers, and the steps you propose cannot work in cases where the node is a service because I do not yet have support for running commands on services.

I have seen exactly the behavior you described, the way I solved it was as I described. (Cache may not be relevant because as I noted the node set needs to change in the middle of the job sequence and I think the only way RunDeck can do that is to reference another job that will re-calculate the node set)

kdebisschop commented 4 years ago

I am going to assume my diagnosis is correct since there is no reply to my last comment. I plan to resolve this issue with additional clarifying text in the README to make users aware of this pitfall.