Service Task response containing "items" key cause TypeError exception on next task

[vtexier]

I created a Spiff Connector to a service returning a dict with the key "items" in it. The corresponding value of the key is a list.

This key cause a TypeError on the next talk in the process in the diagram. Here in the code:

https://github.com/sartography/spiff-arena/blob/8c7061b0402be1bc4b23e9d820c58b85356413c2/SpiffWorkflow/SpiffWorkflow/bpmn/serializer/helpers/registry.py#L49C1-L49C55

task.data is treated as an object (called obj in the code) and when obj.items() is called Python try to call the list value as a function.

Here is the full exception of the event on the next task:

Stacktrace:
TypeError: 'list' object is not callable
                                 ^^^^^^^^^^^
    items = [ (k, v) for k, v in obj.items() ]
  File "/app/venv/lib/python3.11/site-packages/SpiffWorkflow/bpmn/serializer/helpers/registry.py", line 49, in clean
    self.clean(obj)
  File "/app/venv/lib/python3.11/site-packages/SpiffWorkflow/bpmn/serializer/helpers/registry.py", line 42, in convert
                    ^^^^^^^^^^^^^^^
    return dict((k, self.convert(v)) for k, v in obj.items())
  File "/app/venv/lib/python3.11/site-packages/SpiffWorkflow/bpmn/serializer/helpers/dictionary.py", line 97, in <genexpr>
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    return dict((k, self.convert(v)) for k, v in obj.items())
  File "/app/venv/lib/python3.11/site-packages/SpiffWorkflow/bpmn/serializer/helpers/dictionary.py", line 97, in convert
           ^^^^^^^^^^^^^^^^^^^^
    return super().convert(obj)
  File "/app/venv/lib/python3.11/site-packages/SpiffWorkflow/bpmn/serializer/helpers/registry.py", line 43, in convert
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    'data': self.data_converter.convert(task.data),
  File "/app/venv/lib/python3.11/site-packages/SpiffWorkflow/bpmn/serializer/workflow.py", line 233, in task_to_dict
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    new_properties_json = self.serializer.task_to_dict(spiff_task)
  File "/app/src/spiffworkflow_backend/services/task_service.py", line 265, in update_task_model
    self.update_task_model(task_model, spiff_task)
  File "/app/src/spiffworkflow_backend/services/task_service.py", line 199, in update_task_model_with_spiff_task
    self.task_service.update_task_model_with_spiff_task(waiting_spiff_task)
  File "/app/src/spiffworkflow_backend/services/workflow_execution_service.py", line 198, in after_engine_steps
    self.after_engine_steps(bpmn_process_instance)
  File "/app/src/spiffworkflow_backend/services/workflow_execution_service.py", line 234, in on_exception
    self.delegate.on_exception(bpmn_process_instance)
  File "/app/src/spiffworkflow_backend/services/workflow_execution_service.py", line 91, in on_exception
    self.execution_strategy.on_exception(self.bpmn_process_instance)
  File "/app/src/spiffworkflow_backend/services/workflow_execution_service.py", line 422, in run_and_save
    execution_service.run_and_save(exit_at, save)
  File "/app/src/spiffworkflow_backend/services/process_instance_processor.py", line 1418, in _do_engine_steps
    self._do_engine_steps(exit_at, save, execution_strategy_name, execution_strategy)
  File "/app/src/spiffworkflow_backend/services/process_instance_processor.py", line 1383, in do_engine_steps
    yield
  File "/app/src/spiffworkflow_backend/services/process_instance_queue_service.py", line 98, in dequeued
Traceback (most recent call last):

During handling of the above exception, another exception occurred:

SpiffWorkflow.bpmn.exceptions.WorkflowTaskException: TypeError:'list' object is not callable. 
    raise wte
  File "/app/venv/lib/python3.11/site-packages/SpiffWorkflow/bpmn/PythonScriptEngine.py", line 78, in execute
    super().execute(task, script, methods)
  File "/app/src/spiffworkflow_backend/services/process_instance_processor.py", line 362, in execute
    raise e
  File "/app/src/spiffworkflow_backend/services/process_instance_processor.py", line 365, in execute
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    return task.workflow.script_engine.execute(task, self.script)
  File "/app/venv/lib/python3.11/site-packages/SpiffWorkflow/bpmn/specs/mixins/script_task.py", line 46, in _execute
           ^^^^^^^^^^^^^^^^^^^
    return self._execute(task)
  File "/app/venv/lib/python3.11/site-packages/SpiffWorkflow/bpmn/specs/mixins/script_task.py", line 31, in _run_hook
             ^^^^^^^^^^^^^^^^^^^^^^^
    result = self._run_hook(my_task)
  File "/app/venv/lib/python3.11/site-packages/SpiffWorkflow/specs/base.py", line 305, in _run
    raise exc
  File "/app/venv/lib/python3.11/site-packages/SpiffWorkflow/specs/base.py", line 316, in _run
             ^^^^^^^^^^^^^^^^^^^^^^^^^
    retval = self.task_spec._run(self)
  File "/app/venv/lib/python3.11/site-packages/SpiffWorkflow/task.py", line 589, in run
    task.run()
  File "/app/src/spiffworkflow_backend/services/workflow_execution_service.py", line 318, in spiff_run
    self.execution_strategy.spiff_run(self.bpmn_process_instance, exit_at)
  File "/app/src/spiffworkflow_backend/services/workflow_execution_service.py", line 408, in run_and_save
Traceback (most recent call last):

During handling of the above exception, another exception occurred:

TypeError: 'list' object is not callable
                ^^^^^^^^^^^
    for k, v in arg.items():
  File "/app/venv/lib/python3.11/site-packages/SpiffWorkflow/bpmn/PythonScriptEngineEnvironment.py", line 94, in __init__
              ^^^^^^
    self[k] = Box(v)
  File "/app/venv/lib/python3.11/site-packages/SpiffWorkflow/bpmn/PythonScriptEngineEnvironment.py", line 96, in __init__
           ^^^^^^^^^
    return Box(data)
  File "/app/venv/lib/python3.11/site-packages/SpiffWorkflow/bpmn/PythonScriptEngineEnvironment.py", line 149, in convert_to_box
    Box.convert_to_box(context)
  File "/app/venv/lib/python3.11/site-packages/SpiffWorkflow/bpmn/PythonScriptEngineEnvironment.py", line 158, in _prepare_context
    self._prepare_context(context)
  File "/app/venv/lib/python3.11/site-packages/SpiffWorkflow/bpmn/PythonScriptEngineEnvironment.py", line 47, in execute
    super().execute(script, context, external_methods)
  File "/app/src/spiffworkflow_backend/services/process_instance_processor.py", line 135, in execute
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    return self.environment.execute(script, context, external_methods)
  File "/app/venv/lib/python3.11/site-packages/SpiffWorkflow/bpmn/PythonScriptEngine.py", line 118, in _execute
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    return self._execute(script, task.data, external_methods or {})
  File "/app/venv/lib/python3.11/site-packages/SpiffWorkflow/bpmn/PythonScriptEngine.py", line 75, in execute
Traceback (most recent call last):

[vtexier]

I am able to reproduce the problem with this simple code. The problem is in the Box class. It overrides standard dict methods with dict keys.

from SpiffWorkflow.bpmn.PythonScriptEngineEnvironment import Box

# create a standard dict
d = {"items": [1,2], "toto": 'titi'}
# create a Box dict
b = Box(d)

# iterate items() on standard dict is OK
d.items()
# iterate items()  on Box dict is NOT OK
b.items()

Box is deprecated, but the deprecation message does not mention the new class to use... Can you tell me what class to use in replacement of Box ? I want to test this code with the new class replacing Box to see if the problem persist.

[jbirddog]

Thanks for the update, we have been internally debating on Box for a bit. Hopefully the diff that replaces Box is all red lines. I do appreciate following up on this, will add more points to our discussion.

[jbirddog]

Also forgot to address the replacement for Box - at least within arena is based on this line - https://github.com/sartography/spiff-arena/blob/main/spiffworkflow-backend/src/spiffworkflow_backend/services/process_instance_processor.py#L264

You'd want to switch it to NonTaskDataBasedScriptEngineEnvironment as the base class, but this hasn't been tested in a while and looks like it may lead to issues based on the comment above.

[essweine]

I have opened https://github.com/sartography/SpiffWorkflow/issues/359 in the SpiffWorkflow library to remove Box.