Infinite fail loop : limit function rate / retries

[SaMnCo]

I have a very simple workflow like:

apiVersion: 1
description: Collect a form - computes a risk then sends SMS with result
output: CreateResult
tasks:
  # Fetch incoming data and update status. Computes risk.
  SetToPending:
    run: set-to-pending
    inputs: "{$.Invocation.Inputs.default}"

  # Adds a status depending on value of the risk
  MakeDecision:
    run: make-decision
    inputs: "{$.Tasks.SetToPending.Output}"
    requires:
    - SetToPending

  SendSMS:
    run: if
    inputs:
      if: "{$.Tasks.SetToPending.Output.risk > 70}"
      then: 
        run: send-sms
        inputs: "{$.Tasks.MakeDecision.Output}"
      else: 
        run: send-sms
        inputs: "{$.Tasks.MakeDecision.Output}"
    requires:
    - SetToPending
    - MakeDecision

  CreateResult:
    run: compose
    inputs:
      id: "{$.Tasks.SetToPending.Output.id}"
      status: "{$.Tasks.MakeDecision.Output.status}"
      risk: "{$.Tasks.MakeDecision.Output.risk}"
      sid: "{$.Tasks.SendSMS.Output.sid}"
    requires:
    - SetToPending
    - MakeDecision
    - SendSMS

the inputs are supposed to be JSON. I made a mistake calling the function and my JSON was malformed. Then the workflow engine goes crazy and retries for ever:

$ stern workflow --namespace fission-function
"2017-10-20T08:47:17Z" level=info msg="Invoking function 'if' for task 'SendSMS'" 
"2017-10-20T08:47:17Z" level=info msg="Resolved expression" key=then resolved=type:"flow" value:"\n\004then\032\010send-sms\"0\n\007default\022%\n\004expr\022\035{$.Tasks.MakeDecision.Output}0\000"  val=type:"flow" value:"\n\004then\032\010send-sms\"0\n\007default\022%\n\004expr\022\035{$.Tasks.MakeDecision.Output}0\000"  
"2017-10-20T08:47:17Z" level=info msg="Resolved expression" key=else resolved=type:"flow" value:"\n\004else\032\010send-sms\"0\n\007default\022%\n\004expr\022\035{$.Tasks.MakeDecision.Output}0\000"  val=type:"flow" value:"\n\004else\032\010send-sms\"0\n\007default\022%\n\004expr\022\035{$.Tasks.MakeDecision.Output}0\000"  
"2017-10-20T08:47:17Z" level=error msg="Failed to parse input: TypeError: Cannot access member 'risk' of undefined" inputKey=if val=type:"expr" value:"{$.Tasks.SetToPending.Output.risk > 70}"  
"2017-10-20T08:47:17Z" level=error msg="WorkflowInvocation action failed" action=&{0x19fffbf0 0x1a122670 0x1a123210 0x1a33cba0 0x1a2e9050} 
"2017-10-20T08:47:18Z" level=info msg="Scheduler evaluating..." invoke=wi-da56801c-3125-4a21-bbf7-dd2a9563f52d workflow=74926bd6-b4e5-11e7-8ff6-42010a9a010b 
"2017-10-20T08:47:18Z" level=info msg="Checking if dependencies have been satisfied" completedDeps=2 max=2 task=SendSMS 
"2017-10-20T08:47:18Z" level=info msg="task found on horizon" task=SendSMS 
"2017-10-20T08:47:18Z" level=info msg="Checking if dependencies have been satisfied" completedDeps=3 max=3 task=CreateResult 
"2017-10-20T08:47:18Z" level=info msg="Determined schedule" invocation=wi-da56801c-3125-4a21-bbf7-dd2a9563f52d invoke=wi-da56801c-3125-4a21-bbf7-dd2a9563f52d schedule=1 workflow=74926bd6-b4e5-11e7-8ff6-42010a9a010b 
"2017-10-20T08:47:18Z" level=info msg="Invoking function 'if' for task 'SendSMS'" 
"2017-10-20T08:47:18Z" level=info msg="Resolved expression" key=then resolved=type:"flow" value:"\n\004then\032\010send-sms\"0\n\007default\022%\n\004expr\022\035{$.Tasks.MakeDecision.Output}0\000"  val=type:"flow" value:"\n\004then\032\010send-sms\"0\n\007default\022%\n\004expr\022\035{$.Tasks.MakeDecision.Output}0\000"  
"2017-10-20T08:47:18Z" level=info msg="Resolved expression" key=else resolved=type:"flow" value:"\n\004else\032\010send-sms\"0\n\007default\022%\n\004expr\022\035{$.Tasks.MakeDecision.Output}0\000"  val=type:"flow" value:"\n\004else\032\010send-sms\"0\n\007default\022%\n\004expr\022\035{$.Tasks.MakeDecision.Output}0\000"  
"2017-10-20T08:47:18Z" level=error msg="Failed to parse input: TypeError: Cannot access member 'risk' of undefined" inputKey=if val=type:"expr" value:"{$.Tasks.SetToPending.Output.risk > 70}"  
"2017-10-20T08:47:18Z" level=error msg="WorkflowInvocation action failed" action=&{0x19fffbf0 0x1a122670 0x1a123210 0x1a33cba0 0x1a41aa70} 
"2017-10-20T08:47:19Z" level=info msg="Scheduler evaluating..." invoke=wi-da56801c-3125-4a21-bbf7-dd2a9563f52d workflow=74926bd6-b4e5-11e7-8ff6-42010a9a010b 
"2017-10-20T08:47:19Z" level=info msg="Checking if dependencies have been satisfied" completedDeps=3 max=3 task=CreateResult 
"2017-10-20T08:47:19Z" level=info msg="Checking if dependencies have been satisfied" completedDeps=2 max=2 task=SendSMS 
"2017-10-20T08:47:19Z" level=info msg="task found on horizon" task=SendSMS 
"2017-10-20T08:47:19Z" level=info msg="Determined schedule" invocation=wi-da56801c-3125-4a21-bbf7-dd2a9563f52d invoke=wi-da56801c-3125-4a21-bbf7-dd2a9563f52d schedule=1 workflow=74926bd6-b4e5-11e7-8ff6-42010a9a010b 
"2017-10-20T08:47:19Z" level=info msg="Invoking function 'if' for task 'SendSMS'" 
"2017-10-20T08:47:19Z" level=info msg="Resolved expression" key=then resolved=type:"flow" value:"\n\004then\032\010send-sms\"0\n\007default\022%\n\004expr\022\035{$.Tasks.MakeDecision.Output}0\000"  val=type:"flow" value:"\n\004then\032\010send-sms\"0\n\007default\022%\n\004expr\022\035{$.Tasks.MakeDecision.Output}0\000"  
"2017-10-20T08:47:19Z" level=info msg="Resolved expression" key=else resolved=type:"flow" value:"\n\004else\032\010send-sms\"0\n\007default\022%\n\004expr\022\035{$.Tasks.MakeDecision.Output}0\000"  val=type:"flow" value:"\n\004else\032\010send-sms\"0\n\007default\022%\n\004expr\022\035{$.Tasks.MakeDecision.Output}0\000"  
"2017-10-20T08:47:19Z" level=error msg="Failed to parse input: TypeError: Cannot access member 'risk' of undefined" inputKey=if val=type:"expr" value:"{$.Tasks.SetToPending.Output.risk > 70}"  
"2017-10-20T08:47:19Z" level=error msg="WorkflowInvocation action failed" action=&{0x19fffbf0 0x1a122670 0x1a123210 0x1a33cba0 0x1a326120} 
"2017-10-20T08:47:20Z" level=info msg="Scheduler evaluating..." invoke=wi-da56801c-3125-4a21-bbf7-dd2a9563f52d workflow=74926bd6-b4e5-11e7-8ff6-42010a9a010b 
"2017-10-20T08:47:20Z" level=info msg="Checking if dependencies have been satisfied" completedDeps=3 max=3 task=CreateResult 
"2017-10-20T08:47:20Z" level=info msg="Checking if dependencies have been satisfied" completedDeps=2 max=2 task=SendSMS 
"2017-10-20T08:47:20Z" level=info msg="task found on horizon" task=SendSMS 
"2017-10-20T08:47:20Z" level=info msg="Determined schedule" invocation=wi-da56801c-3125-4a21-bbf7-dd2a9563f52d invoke=wi-da56801c-3125-4a21-bbf7-dd2a9563f52d schedule=1 workflow=74926bd6-b4e5-11e7-8ff6-42010a9a010b 
"2017-10-20T08:47:20Z" level=info msg="Invoking function 'if' for task 'SendSMS'" 
"2017-10-20T08:47:20Z" level=info msg="Resolved expression" key=then resolved=type:"flow" value:"\n\004then\032\010send-sms\"0\n\007default\022%\n\004expr\022\035{$.Tasks.MakeDecision.Output}0\000"  val=type:"flow" value:"\n\004then\032\010send-sms\"0\n\007default\022%\n\004expr\022\035{$.Tasks.MakeDecision.Output}0\000"  
"2017-10-20T08:47:20Z" level=info msg="Resolved expression" key=else resolved=type:"flow" value:"\n\004else\032\010send-sms\"0\n\007default\022%\n\004expr\022\035{$.Tasks.MakeDecision.Output}0\000"  val=type:"flow" value:"\n\004else\032\010send-sms\"0\n\007default\022%\n\004expr\022\035{$.Tasks.MakeDecision.Output}0\000"  
"2017-10-20T08:47:20Z" level=error msg="Failed to parse input: TypeError: Cannot access member 'risk' of undefined" inputKey=if val=type:"expr" value:"{$.Tasks.SetToPending.Output.risk > 70}"  
"2017-10-20T08:47:20Z" level=error msg="WorkflowInvocation action failed" action=&{0x19fffbf0 0x1a122670 0x1a123210 0x1a33cba0 0x1a098310} 
"2017-10-20T08:47:21Z" level=info msg="Scheduler evaluating..." invoke=wi-da56801c-3125-4a21-bbf7-dd2a9563f52d workflow=74926bd6-b4e5-11e7-8ff6-42010a9a010b 
"2017-10-20T08:47:21Z" level=info msg="Checking if dependencies have been satisfied" completedDeps=2 max=2 task=SendSMS 
"2017-10-20T08:47:21Z" level=info msg="task found on horizon" task=SendSMS 
"2017-10-20T08:47:21Z" level=info msg="Checking if dependencies have been satisfied" completedDeps=3 max=3 task=CreateResult 
"2017-10-20T08:47:21Z" level=info msg="Determined schedule" invocation=wi-da56801c-3125-4a21-bbf7-dd2a9563f52d invoke=wi-da56801c-3125-4a21-bbf7-dd2a9563f52d schedule=1 workflow=74926bd6-b4e5-11e7-8ff6-42010a9a010b 
"2017-10-20T08:47:21Z" level=info msg="Invoking function 'if' for task 'SendSMS'" 
"2017-10-20T08:47:21Z" level=info msg="Resolved expression" key=then resolved=type:"flow" value:"\n\004then\032\010send-sms\"0\n\007default\022%\n\004expr\022\035{$.Tasks.MakeDecision.Output}0\000"  val=type:"flow" value:"\n\004then\032\010send-sms\"0\n\007default\022%\n\004expr\022\035{$.Tasks.MakeDecision.Output}0\000"  
"2017-10-20T08:47:21Z" level=info msg="Resolved expression" key=else resolved=type:"flow" value:"\n\004else\032\010send-sms\"0\n\007default\022%\n\004expr\022\035{$.Tasks.MakeDecision.Output}0\000"  val=type:"flow" value:"\n\004else\032\010send-sms\"0\n\007default\022%\n\004expr\022\035{$.Tasks.MakeDecision.Output}0\000"  
"2017-10-20T08:47:21Z" level=error msg="Failed to parse input: TypeError: Cannot access member 'risk' of undefined" inputKey=if val=type:"expr" value:"{$.Tasks.SetToPending.Output.risk > 70}"  
"2017-10-20T08:47:21Z" level=error msg="WorkflowInvocation action failed" action=&{0x19fffbf0 0x1a122670 0x1a123210 0x1a33cba0 0x1a3bdb00} 
"2017-10-20T08:47:22Z" level=info msg="Scheduler evaluating..." invoke=wi-da56801c-3125-4a21-bbf7-dd2a9563f52d workflow=74926bd6-b4e5-11e7-8ff6-42010a9a010b 
"2017-10-20T08:47:22Z" level=info msg="Checking if dependencies have been satisfied" completedDeps=3 max=3 task=CreateResult 
"2017-10-20T08:47:22Z" level=info msg="Checking if dependencies have been satisfied" completedDeps=2 max=2 task=SendSMS 
"2017-10-20T08:47:22Z" level=info msg="task found on horizon" task=SendSMS 
"2017-10-20T08:47:22Z" level=info msg="Determined schedule" invocation=wi-da56801c-3125-4a21-bbf7-dd2a9563f52d invoke=wi-da56801c-3125-4a21-bbf7-dd2a9563f52d schedule=1 workflow=74926bd6-b4e5-11e7-8ff6-42010a9a010b 
"2017-10-20T08:47:22Z" level=info msg="Invoking function 'if' for task 'SendSMS'" 
"2017-10-20T08:47:22Z" level=error msg="Failed to parse input: TypeError: Cannot access member 'risk' of undefined" inputKey=if val=type:"expr" value:"{$.Tasks.SetToPending.Output.risk > 70}"  
"2017-10-20T08:47:22Z" level=error msg="WorkflowInvocation action failed" action=&{0x19fffbf0 0x1a122670 0x1a123210 0x1a33cba0 0x1b678bf0} 
"2017-10-20T08:47:23Z" level=info msg="Scheduler evaluating..." invoke=wi-da56801c-3125-4a21-bbf7-dd2a9563f52d workflow=74926bd6-b4e5-11e7-8ff6-42010a9a010b 
"2017-10-20T08:47:23Z" level=info msg="Checking if dependencies have been satisfied" completedDeps=3 max=3 task=CreateResult 
"2017-10-20T08:47:23Z" level=info msg="Checking if dependencies have been satisfied" completedDeps=2 max=2 task=SendSMS 
"2017-10-20T08:47:23Z" level=info msg="task found on horizon" task=SendSMS 
"2017-10-20T08:47:23Z" level=info msg="Determined schedule" invocation=wi-da56801c-3125-4a21-bbf7-dd2a9563f52d invoke=wi-da56801c-3125-4a21-bbf7-dd2a9563f52d schedule=1 workflow=74926bd6-b4e5-11e7-8ff6-42010a9a010b 
"2017-10-20T08:47:23Z" level=info msg="Invoking function 'if' for task 'SendSMS'" 
"2017-10-20T08:47:23Z" level=info msg="Resolved expression" key=then resolved=type:"flow" value:"\n\004then\032\010send-sms\"0\n\007default\022%\n\004expr\022\035{$.Tasks.MakeDecision.Output}0\000"  val=type:"flow" value:"\n\004then\032\010send-sms\"0\n\007default\022%\n\004expr\022\035{$.Tasks.MakeDecision.Output}0\000"  
"2017-10-20T08:47:23Z" level=info msg="Resolved expression" key=else resolved=type:"flow" value:"\n\004else\032\010send-sms\"0\n\007default\022%\n\004expr\022\035{$.Tasks.MakeDecision.Output}0\000"  val=type:"flow" value:"\n\004else\032\010send-sms\"0\n\007default\022%\n\004expr\022\035{$.Tasks.MakeDecision.Output}0\000"  
"2017-10-20T08:47:23Z" level=error msg="Failed to parse input: TypeError: Cannot access member 'risk' of undefined" inputKey=if val=type:"expr" value:"{$.Tasks.SetToPending.Output.risk > 70}"  
"2017-10-20T08:47:23Z" level=error msg="WorkflowInvocation action failed" action=&{0x19fffbf0 0x1a122670 0x1a123210 0x1a33cba0 0x1a09e090} 
"2017-10-20T08:47:24Z" level=info msg="Scheduler evaluating..." invoke=wi-da56801c-3125-4a21-bbf7-dd2a9563f52d workflow=74926bd6-b4e5-11e7-8ff6-42010a9a010b 
"2017-10-20T08:47:24Z" level=info msg="Checking if dependencies have been satisfied" completedDeps=2 max=2 task=SendSMS 
"2017-10-20T08:47:24Z" level=info msg="task found on horizon" task=SendSMS 
"2017-10-20T08:47:24Z" level=info msg="Checking if dependencies have been satisfied" completedDeps=3 max=3 task=CreateResult 
"2017-10-20T08:47:24Z" level=info msg="Determined schedule" invocation=wi-da56801c-3125-4a21-bbf7-dd2a9563f52d invoke=wi-da56801c-3125-4a21-bbf7-dd2a9563f52d schedule=1 workflow=74926bd6-b4e5-11e7-8ff6-42010a9a010b 
"2017-10-20T08:47:24Z" level=info msg="Invoking function 'if' for task 'SendSMS'" 
"2017-10-20T08:47:24Z" level=info msg="Resolved expression" key=then resolved=type:"flow" value:"\n\004then\032\010send-sms\"0\n\007default\022%\n\004expr\022\035{$.Tasks.MakeDecision.Output}0\000"  val=type:"flow" value:"\n\004then\032\010send-sms\"0\n\007default\022%\n\004expr\022\035{$.Tasks.MakeDecision.Output}0\000"  
"2017-10-20T08:47:24Z" level=info msg="Resolved expression" key=else resolved=type:"flow" value:"\n\004else\032\010send-sms\"0\n\007default\022%\n\004expr\022\035{$.Tasks.MakeDecision.Output}0\000"  val=type:"flow" value:"\n\004else\032\010send-sms\"0\n\007default\022%\n\004expr\022\035{$.Tasks.MakeDecision.Output}0\000"  
"2017-10-20T08:47:24Z" level=error msg="Failed to parse input: TypeError: Cannot access member 'risk' of undefined" inputKey=if val=type:"expr" value:"{$.Tasks.SetToPending.Output.risk > 70}"  
"2017-10-20T08:47:24Z" level=error msg="WorkflowInvocation action failed" action=&{0x19fffbf0 0x1a122670 0x1a123210 0x1a33cba0 0x1a098160} 
"2017-10-20T08:47:25Z" level=info msg="Scheduler evaluating..." invoke=wi-da56801c-3125-4a21-bbf7-dd2a9563f52d workflow=74926bd6-b4e5-11e7-8ff6-42010a9a010b 
"2017-10-20T08:47:25Z" level=info msg="Checking if dependencies have been satisfied" completedDeps=2 max=2 task=SendSMS 
"2017-10-20T08:47:25Z" level=info msg="task found on horizon" task=SendSMS 
"2017-10-20T08:47:25Z" level=info msg="Checking if dependencies have been satisfied" completedDeps=3 max=3 task=CreateResult 
"2017-10-20T08:47:25Z" level=info msg="Determined schedule" invocation=wi-da56801c-3125-4a21-bbf7-dd2a9563f52d invoke=wi-da56801c-3125-4a21-bbf7-dd2a9563f52d schedule=1 workflow=74926bd6-b4e5-11e7-8ff6-42010a9a010b 
"2017-10-20T08:47:25Z" level=info msg="Invoking function 'if' for task 'SendSMS'" 
"2017-10-20T08:47:25Z" level=info msg="Resolved expression" key=then resolved=type:"flow" value:"\n\004then\032\010send-sms\"0\n\007default\022%\n\004expr\022\035{$.Tasks.MakeDecision.Output}0\000"  val=type:"flow" value:"\n\004then\032\010send-sms\"0\n\007default\022%\n\004expr\022\035{$.Tasks.MakeDecision.Output}0\000"  
"2017-10-20T08:47:25Z" level=info msg="Resolved expression" key=else resolved=type:"flow" value:"\n\004else\032\010send-sms\"0\n\007default\022%\n\004expr\022\035{$.Tasks.MakeDecision.Output}0\000"  val=type:"flow" value:"\n\004else\032\010send-sms\"0\n\007default\022%\n\004expr\022\035{$.Tasks.MakeDecision.Output}0\000"  
"2017-10-20T08:47:25Z" level=error msg="Failed to parse input: TypeError: Cannot access member 'risk' of undefined" inputKey=if val=type:"expr" value:"{$.Tasks.SetToPending.Output.risk > 70}"  
"2017-10-20T08:47:25Z" level=error msg="WorkflowInvocation action failed" action=&{0x19fffbf0 0x1a122670 0x1a123210 0x1a33cba0 0x1a3d7ef0} 
"2017-10-20T08:47:26Z" level=info msg="Scheduler evaluating..." invoke=wi-da56801c-3125-4a21-bbf7-dd2a9563f52d workflow=74926bd6-b4e5-11e7-8ff6-42010a9a010b 
"2017-10-20T08:47:26Z" level=info msg="Checking if dependencies have been satisfied" completedDeps=3 max=3 task=CreateResult 
"2017-10-20T08:47:26Z" level=info msg="Checking if dependencies have been satisfied" completedDeps=2 max=2 task=SendSMS 
"2017-10-20T08:47:26Z" level=info msg="task found on horizon" task=SendSMS 
"2017-10-20T08:47:26Z" level=info msg="Determined schedule" invocation=wi-da56801c-3125-4a21-bbf7-dd2a9563f52d invoke=wi-da56801c-3125-4a21-bbf7-dd2a9563f52d schedule=1 workflow=74926bd6-b4e5-11e7-8ff6-42010a9a010b 
"2017-10-20T08:47:26Z" level=info msg="Invoking function 'if' for task 'SendSMS'" 
"2017-10-20T08:47:26Z" level=info msg="Resolved expression" key=then resolved=type:"flow" value:"\n\004then\032\010send-sms\"0\n\007default\022%\n\004expr\022\035{$.Tasks.MakeDecision.Output}0\000"  val=type:"flow" value:"\n\004then\032\010send-sms\"0\n\007default\022%\n\004expr\022\035{$.Tasks.MakeDecision.Output}0\000"  
"2017-10-20T08:47:26Z" level=info msg="Resolved expression" key=else resolved=type:"flow" value:"\n\004else\032\010send-sms\"0\n\007default\022%\n\004expr\022\035{$.Tasks.MakeDecision.Output}0\000"  val=type:"flow" value:"\n\004else\032\010send-sms\"0\n\007default\022%\n\004expr\022\035{$.Tasks.MakeDecision.Output}0\000"  
"2017-10-20T08:47:26Z" level=error msg="Failed to parse input: TypeError: Cannot access member 'risk' of undefined" inputKey=if val=type:"expr" value:"{$.Tasks.SetToPending.Output.risk > 70}"  
"2017-10-20T08:47:26Z" level=error msg="WorkflowInvocation action failed" action=&{0x19fffbf0 0x1a122670 0x1a123210 0x1a33cba0 0x1a3f1df0} 

I could not find a way to stop that. Currently the only option I found is to do

1. Delete the NATS deployment and recreate it
2. Delete the workflow pod in the fission-function namespace

Is there another and better way?

Thanks!

[erwinvaneyk]

Hi @SaMnCo, sorry for the late response; I just got back from vacation.

Good catch! There is a overall timeout set for an workflow execution. So the execution would eventually stop after 10 or so minutes. But of course an error like this should stop the workflow earlier than that. Not sure if we even want to retry at all on an expression error.

Aside, we are working on removing the manual YAML->JSON step. :)

[erwinvaneyk]

Fixed in #60 - workflows no longer require the manual compilation to JSON.

[soamvasani]

"2017-10-20T08:47:26Z" level=error msg="Failed to parse input: TypeError: Cannot access member 'risk' of undefined" inputKey=if val=type:"expr" value:"{$.Tasks.SetToPending.Output.risk > 70}"

As I understand it, this issue is not about compilation to JSON, but about errors in JSON expressions in input fields of tasks.

[erwinvaneyk]

My bad! My checking on open issues, I misread that this issue was regarding the yaml conversion. Thanks @soamvasani !

[soamvasani]

Errors in json now fail the task immediately.