search

StackStorm / st2

StackStorm (aka "IFTTT for Ops") is event-driven automation for auto-remediation, incident responses, troubleshooting, deployments, and more for DevOps and SREs. Includes rules engine, workflow, 160 integration packs with 6000+ actions (see https://exchange.stackstorm.org) and ChatOps. Installer at https://docs.stackstorm.com/install/index.html
https://stackstorm.com/
Apache License 2.0
6.07k stars 749 forks source link

Array of Dictionaries Serialization #2903

Closed bigmstone closed 8 years ago

bigmstone commented 8 years ago

I've encountered an issue serializing dictionaries inside an array with StackStorm + Mistral.

The following wokflow is being executed: Metadata:


---
description: Mistral Array Test
enabled: true
entry_point: workflow/mistral_array.yaml
name: mistral_array
pack: mistral_array_test
parameters:
  iter_data:
    required: true
    type: array
runner_type: mistral-v2

Workflow:

version: '2.0'

mistral_array_test.mistral_array:
    description: Mistral Array
    type: direct
    input:
        - iter_data
    output:
        stdout: <% $.stdout %>
    tasks:
      iterate_data:
        with-items: item in <% $.iter_data %>
        workflow: mistral_array_test.mistral_array_mock
        input:
          bob: <% $.bob %>
          sally: <% $.sally %>
st2 run mistral_array_test.mistral_array iter_data='{"bob":"1","sally":"1"},{"bob":"2","sally":"2"}'

When I attempt to run this workflow using the above command I receive the following error output from stackstorm:

...
id: 57d96694c4da5f434d2ccd84
action.ref: mistral_array_test.mistral_array
parameters:
  iter_data:
  - '{"bob":"1"'
  - '"sally":"1"}'
  - '{"bob":"2"'
  - '"sally":"2"}'
status: failed
start_timestamp: 2016-09-14T15:02:44.361797Z
end_timestamp: 2016-09-14T15:02:48.898389Z
result:
  extra:
    state: ERROR
    state_info: 'Failed to start task [task_ex=TaskExecution {''state_info'': ''Can not evaluate YAQL expression:  $.bob , data = {u''iter_data'': [u''{"bob":"1"'', u''"sally":"1"}'', u''{"bob":"2"'', u''"sally":"2"}''], u''item'': u''{"bob":"1"'', u''__env'': {u''st2_liveaction_id'': u''57d96694c4da5f434d2ccd83'', u''st2_execution_id'': u''57d96694c4da5f434d2ccd84'', u''__actions'': {u''st2.action'': {u''st2_context'': {u''endpoint'': u''http://0.0.0.0:9101/v1/actionexecutions'', u''api_url'': u''http://0.0.0.0:9101/v1'', u''parent'': {u''user'': u''st2admin'', u''execution_id'': u''57d96694c4da5f434d2ccd84''}, u''auth_token'': u''8972baddd5194a01bb76f9f1b870c9a4'', u''skip_notify_tasks'': [], u''notify'': {}}}}, u''st2_action_api_url'': u''http://0.0.0.0:9101/v1''}, u''__execution'': {u''input'': {u''iter_data'': [u''{"bob":"1"'', u''"sally":"1"}'', u''{"bob":"2"'', u''"sally":"2"}'']}, u''params'': {u''env'': {u''st2_liveaction_id'': u''57d96694c4da5f434d2ccd83'', u''st2_execution_id'': u''57d96694c4da5f434d2ccd84'', u''__actions'': {u''st2.action'': {u''st2_context'': {u''endpoint'': u''http://0.0.0.0:9101/v1/actionexecutions'', u''api_url'': u''http://0.0.0.0:9101/v1'', u''parent'': {u''user'': u''st2admin'', u''execution_id'': u''57d96694c4da5f434d2ccd84''}, u''auth_token'': u''8972baddd5194a01bb76f9f1b870c9a4'', u''notify'': {}, u''skip_notify_tasks'': []}}}, u''st2_action_api_url'': u''http://0.0.0.0:9101/v1''}}, u''id'': u''61be1c76-b8e0-4f8e-b362-9feed766b553'', u''spec'': {u''tasks'': {u''iterate_data'': {u''with-items'': u''item in <% $.iter_data %>'', ''name'': u''iterate_data'', u''workflow'': u''mistral_array_test.mistral_array_mock'', u''version'': u''2.0'', u''input'': {u''bob'': u''<% $.bob %>'', u''sally'': u''<% $.sally %>''}, u''type'': u''direct''}}, u''description'': u''Mistral Array'', u''version'': u''2.0'', u''input'': [u''iter_data''], u''output'': {u''stdout'': u''<% $.stdout %>''}, u''type'': u''direct'', u''name'': u''mistral_array_test.mistral_array''}}}; error: u''bob'''', ''in_context'': {u''iter_data'': [u''{"bob":"1"'', u''"sally":"1"}'', u''{"bob":"2"'', u''"sally":"2"}''], u''__env'': {u''st2_liveaction_id'': u''57d96694c4da5f434d2ccd83'', u''st2_execution_id'': u''57d96694c4da5f434d2ccd84'', u''__actions'': {u''st2.action'': {u''st2_context'': {u''endpoint'': u''http://0.0.0.0:9101/v1/actionexecutions'', u''api_url'': u''http://0.0.0.0:9101/v1'', u''parent'': {u''user'': u''st2admin'', u''execution_id'': u''57d96694c4da5f434d2ccd84''}, u''auth_token'': u''8972baddd5194a01bb76f9f1b870c9a4'', u''skip_notify_tasks'': [], u''notify'': {}}}}, u''st2_action_api_url'': u''http://0.0.0.0:9101/v1''}, u''__execution'': {u''input'': {u''iter_data'': [u''{"bob":"1"'', u''"sally":"1"}'', u''{"bob":"2"'', u''"sally":"2"}'']}, u''params'': {u''env'': {u''st2_liveaction_id'': u''57d96694c4da5f434d2ccd83'', u''st2_execution_id'': u''57d96694c4da5f434d2ccd84'', u''__actions'': {u''st2.action'': {u''st2_context'': {u''endpoint'': u''http://0.0.0.0:9101/v1/actionexecutions'', u''api_url'': u''http://0.0.0.0:9101/v1'', u''parent'': {u''user'': u''st2admin'', u''execution_id'': u''57d96694c4da5f434d2ccd84''}, u''auth_token'': u''8972baddd5194a01bb76f9f1b870c9a4'', u''notify'': {}, u''skip_notify_tasks'': []}}}, u''st2_action_api_url'': u''http://0.0.0.0:9101/v1''}}, u''id'': u''61be1c76-b8e0-4f8e-b362-9feed766b553'', u''spec'': {u''tasks'': {u''iterate_data'': {u''with-items'': u''item in <% $.iter_data %>'', u''name'': u''iterate_data'', u''workflow'': u''mistral_array_test.mistral_array_mock'', u''version'': u''2.0'', u''input'': {u''bob'': u''<% $.bob %>'', u''sally'': u''<% $.sally %>''}, u''type'': u''direct''}}, u''description'': u''Mistral Array'', u''version'': u''2.0'', u''input'': [u''iter_data''], u''output'': {u''stdout'': u''<% $.stdout %>''}, u''type'': u''direct'', u''name'': u''mistral_array_test.mistral_array''}}}, ''name'': u''iterate_data'', ''workflow_name'': u''mistral_array_test.mistral_array'', ''runtime_context'': {}, ''created_at'': ''2016-09-14 15:02:45.023539'', ''state'': ''ERROR'', ''spec'': {u''with-items'': u''item in <% $.iter_data %>'', u''version'': ''2.0'', u''name'': u''iterate_data'', u''workflow'': u''mistral_array_test.mistral_array_mock'', u''input'': {u''bob'': u''<% $.bob %>'', u''sally'': u''<% $.sally %>''}, u''type'': u''direct''}, ''workflow_execution_id'': u''61be1c76-b8e0-4f8e-b362-9feed766b553'', ''workflow_id'': u''1fa6e7e3-7391-4260-8c35-663de14d446e'', ''processed'': False, ''published'': {}, ''scope'': ''private'', ''project_id'': ''<default-project>'', ''type'': ''task_execution'', ''id'': u''da37a414-1977-47e7-8e9e-356f8fb94ecf''}]: Can not evaluate YAQL expression:  $.bob , data = {u''iter_data'': [u''{"bob":"1"'', u''"sally":"1"}'', u''{"bob":"2"'', u''"sally":"2"}''], u''item'': u''{"bob":"1"'', u''__env'': {u''st2_liveaction_id'': u''57d96694c4da5f434d2ccd83'', u''st2_execution_id'': u''57d96694c4da5f434d2ccd84'', u''__actions'': {u''st2.action'': {u''st2_context'': {u''endpoint'': u''http://0.0.0.0:9101/v1/actionexecutions'', u''api_url'': u''http://0.0.0.0:9101/v1'', u''parent'': {u''user'': u''st2admin'', u''execution_id'': u''57d96694c4da5f434d2ccd84''}, u''auth_token'': u''8972baddd5194a01bb76f9f1b870c9a4'', u''skip_notify_tasks'': [], u''notify'': {}}}}, u''st2_action_api_url'': u''http://0.0.0.0:9101/v1''}, u''__execution'': {u''input'': {u''iter_data'': [u''{"bob":"1"'', u''"sally":"1"}'', u''{"bob":"2"'', u''"sally":"2"}'']}, u''params'': {u''env'': {u''st2_liveaction_id'': u''57d96694c4da5f434d2ccd83'', u''st2_execution_id'': u''57d96694c4da5f434d2ccd84'', u''__actions'': {u''st2.action'': {u''st2_context'': {u''endpoint'': u''http://0.0.0.0:9101/v1/actionexecutions'', u''api_url'': u''http://0.0.0.0:9101/v1'', u''parent'': {u''user'': u''st2admin'', u''execution_id'': u''57d96694c4da5f434d2ccd84''}, u''auth_token'': u''8972baddd5194a01bb76f9f1b870c9a4'', u''notify'': {}, u''skip_notify_tasks'': []}}}, u''st2_action_api_url'': u''http://0.0.0.0:9101/v1''}}, u''id'': u''61be1c76-b8e0-4f8e-b362-9feed766b553'', u''spec'': {u''tasks'': {u''iterate_data'': {u''with-items'': u''item in <% $.iter_data %>'', u''name'': u''iterate_data'', u''workflow'': u''mistral_array_test.mistral_array_mock'', u''version'': u''2.0'', u''input'': {u''bob'': u''<% $.bob %>'', u''sally'': u''<% $.sally %>''}, u''type'': u''direct''}}, u''description'': u''Mistral Array'', u''version'': u''2.0'', u''input'': [u''iter_data''], u''output'': {u''stdout'': u''<% $.stdout %>''}, u''type'': u''direct'', u''name'': u''mistral_array_test.mistral_array''}}}; error: u''bob'''
  tasks:
  - created_at: '2016-09-14 15:02:45.023539'
    id: da37a414-1977-47e7-8e9e-356f8fb94ecf
    input: null
    name: iterate_data
    published: {}
    result: []
    state: ERROR
    state_info: 'Can not evaluate YAQL expression:  $.bob , data = {u''iter_data'': [u''{"bob":"1"'', u''"sally":"1"}'', u''{"bob":"2"'', u''"sally":"2"}''], u''item'': u''{"bob":"1"'', u''__env'': {u''st2_liveaction_id'': u''57d96694c4da5f434d2ccd83'', u''st2_execution_id'': u''57d96694c4da5f434d2ccd84'', u''__actions'': {u''st2.action'': {u''st2_context'': {u''endpoint'': u''http://0.0.0.0:9101/v1/actionexecutions'', u''api_url'': u''http://0.0.0.0:9101/v1'', u''parent'': {u''user'': u''st2admin'', u''execution_id'': u''57d96694c4da5f434d2ccd84''}, u''auth_token'': u''8972baddd5194a01bb76f9f1b870c9a4'', u''skip_notify_tasks'': [], u''notify'': {}}}}, u''st2_action_api_url'': u''http://0.0.0.0:9101/v1''}, u''__execution'': {u''input'': {u''iter_data'': [u''{"bob":"1"'', u''"sally":"1"}'', u''{"bob":"2"'', u''"sally":"2"}'']}, u''params'': {u''env'': {u''st2_liveaction_id'': u''57d96694c4da5f434d2ccd83'', u''st2_execution_id'': u''57d96694c4da5f434d2ccd84'', u''__actions'': {u''st2.action'': {u''st2_context'': {u''endpoint'': u''http://0.0.0.0:9101/v1/actionexecutions'', u''api_url'': u''http://0.0.0.0:9101/v1'', u''parent'': {u''user'': u''st2admin'', u''execution_id'': u''57d96694c4da5f434d2ccd84''}, u''auth_token'': u''8972baddd5194a01bb76f9f1b870c9a4'', u''notify'': {}, u''skip_notify_tasks'': []}}}, u''st2_action_api_url'': u''http://0.0.0.0:9101/v1''}}, u''id'': u''61be1c76-b8e0-4f8e-b362-9feed766b553'', u''spec'': {u''tasks'': {u''iterate_data'': {u''with-items'': u''item in <% $.iter_data %>'', u''name'': u''iterate_data'', u''workflow'': u''mistral_array_test.mistral_array_mock'', u''version'': u''2.0'', u''input'': {u''bob'': u''<% $.bob %>'', u''sally'': u''<% $.sally %>''}, u''type'': u''direct''}}, u''description'': u''Mistral Array'', u''version'': u''2.0'', u''input'': [u''iter_data''], u''output'': {u''stdout'': u''<% $.stdout %>''}, u''type'': u''direct'', u''name'': u''mistral_array_test.mistral_array''}}}; error: u''bob'''
    updated_at: '2016-09-14 15:02:45.031358'
    workflow_execution_id: -b8e0-4f8e-b362-9feed766b553
    workflow_name: mistral_array_test.mistral_array

From mistral log:

2016-09-14 15:02:44.979 17082 INFO mistral.engine.rpc [-] Received RPC request 'start_workflow'[rpc_ctx=MistralContext {u'project_name': None, u'user_id': None, u'roles': [u''], u'auth_token': None, u'is_trust_scoped': False, u'service_catalog': None, u'project_id': None, u'user_name': None}, workflow_identifier=mistral_array_test.mistral_array, workflow_input={u'iter_data': [u'{"bob":"1"', u'"sally":"1"}', u'{"bob":"2"', u'"sally":"2"}']}, description=, params={u'env': {u'st2_liveaction_id': u'57d96694c4da5f434d2ccd83', u'st2_execution_id': u'57d96694c4da5f434d2ccd84', u'__actions': {u'st2.action': {u'st2_context': {u'endpoint': u'http://0.0.0.0:9101/v1/actionexecutions', u'api_url': u'http://0.0.0.0:9101/v1', u'parent': {u'user': u'st2admin', u'execution_id': u'57d96694c4da5f434d2ccd84'}, u'auth_token': u'8972baddd5194a01bb76f9f1b870c9a4', u'notify': {}, u'skip_notify_tasks': []}}}, u'st2_action_api_url': u'http://0.0.0.0:9101/v1'}}]
2016-09-14 15:02:45.008 17082 INFO workflow_trace [-] Starting workflow: 'mistral_array_test.mistral_array' (execution_id=61be1c76-b8e0-4f8e-b362-9feed766b553)
2016-09-14 15:02:45.019 17082 INFO workflow_trace [-] Execution of workflow 'mistral_array_test.mistral_array' [IDLE -> RUNNING] (execution_id=61be1c76-b8e0-4f8e-b362-9feed766b553)
2016-09-14 15:02:45.027 17082 ERROR mistral.engine.task_handler [-] An error while calculating task action inputs [task_execution_id=da37a414-1977-47e7-8e9e-356f8fb94ecf]: Can not evaluate YAQL expression:  $.bob , data = {u'iter_data': [u'{"bob":"1"', u'"sally":"1"}', u'{"bob":"2"', u'"sally":"2"}'], u'item': u'{"bob":"1"', u'__env': {u'st2_liveaction_id': u'57d96694c4da5f434d2ccd83', u'st2_execution_id': u'57d96694c4da5f434d2ccd84', u'__actions': {u'st2.action': {u'st2_context': {u'endpoint': u'http://0.0.0.0:9101/v1/actionexecutions', u'api_url': u'http://0.0.0.0:9101/v1', u'parent': {u'user': u'st2admin', u'execution_id': u'57d96694c4da5f434d2ccd84'}, u'auth_token': u'8972baddd5194a01bb76f9f1b870c9a4', u'skip_notify_tasks': [], u'notify': {}}}}, u'st2_action_api_url': u'http://0.0.0.0:9101/v1'}, u'__execution': {u'input': {u'iter_data': [u'{"bob":"1"', u'"sally":"1"}', u'{"bob":"2"', u'"sally":"2"}']}, u'params': {u'env': {u'st2_liveaction_id': u'57d96694c4da5f434d2ccd83', u'st2_execution_id': u'57d96694c4da5f434d2ccd84', u'__actions': {u'st2.action': {u'st2_context': {u'endpoint': u'http://0.0.0.0:9101/v1/actionexecutions', u'api_url': u'http://0.0.0.0:9101/v1', u'parent': {u'user': u'st2admin', u'execution_id': u'57d96694c4da5f434d2ccd84'}, u'auth_token': u'8972baddd5194a01bb76f9f1b870c9a4', u'notify': {}, u'skip_notify_tasks': []}}}, u'st2_action_api_url': u'http://0.0.0.0:9101/v1'}}, u'id': u'61be1c76-b8e0-4f8e-b362-9feed766b553', u'spec': {u'tasks': {u'iterate_data': {u'with-items': u'item in <% $.iter_data %>', u'name': u'iterate_data', u'workflow': u'mistral_array_test.mistral_array_mock', u'version': u'2.0', u'input': {u'bob': u'<% $.bob %>', u'sally': u'<% $.sally %>'}, u'type': u'direct'}}, u'description': u'Mistral Array', u'version': u'2.0', u'input': [u'iter_data'], u'output': {u'stdout': u'<% $.stdout %>'}, u'type': u'direct', u'name': u'mistral_array_test.mistral_array'}}}; error: u'bob'
2016-09-14 15:02:45.027 17082 INFO workflow_trace [-] Task execution 'iterate_data' [RUNNING -> ERROR] (execution_id=61be1c76-b8e0-4f8e-b362-9feed766b553)
2016-09-14 15:02:45.028 17082 INFO workflow_trace [-] Execution of workflow 'mistral_array_test.mistral_array' [RUNNING -> ERROR] (execution_id=61be1c76-b8e0-4f8e-b362-9feed766b553)
bigmstone commented 8 years ago

Based on this:

parameters:
  iter_data:
  - '{"bob":"1"'
  - '"sally":"1"}'
  - '{"bob":"2"'
  - '"sally":"2"}'

It would appear we're serializing the array strangely in ST2. Will need to dig into that portion to know more.