[QST] RuntimeError: Error: <class 'AttributeError'> - 'list_iterator' object has no attribute 'names'

[BenHall]

❓ Questions & Help

I'm attempting to train/infer a model using Transformers4rec, following the tutorial on the site - https://nvidia-merlin.github.io/Transformers4Rec/main/examples/tutorial/index.html

I've managed to output some models: 0_transformworkflowtriton 1_predictpytorchtriton executor_model

I'm now attempting to predict recommendations using it. I've used nvcr.io/nvidia/merlin/merlin-pytorch:23.06 to start tritonserver for the hosting.

However, my calls fail with:

RuntimeError: Error: <class 'AttributeError'> - 'list_iterator' object has no attribute 'names'

I'm at a lost as to why. The 4th part of the tutorial seems to have been removed.

Any examples/assistance?

Details

Server: nvcr.io/nvidia/merlin/merlin-pytorch:23.06

Client code:

from merlin.systems.triton.utils import send_triton_request
response = send_triton_request(workflow.input_schema, filtered_batch, ["item_id_scores", "item_ids"], endpoint=ip+":8001")
print(response)

Stack trace:

InferenceServerException: [StatusCode.INTERNAL] Traceback (most recent call last):
  File "/root/models/executor_model/1/model.py", line 100, in execute
    outputs = self.ensemble.transform(inputs, runtime=TritonExecutorRuntime())
  File "/usr/local/lib/python3.8/dist-packages/merlin/systems/dag/ensemble.py", line 78, in transform
    return runtime.transform(self.graph, transformable)
  File "/usr/local/lib/python3.8/dist-packages/merlin/systems/dag/runtimes/base_runtime.py", line 53, in transform
    return self.executor.transform(transformable, [graph.output_node])
  File "/usr/local/lib/python3.8/dist-packages/merlin/dag/executors.py", line 87, in transform
    transformed_data = self._execute_node(node, transformable, capture_dtypes, strict)
  File "/usr/local/lib/python3.8/dist-packages/merlin/dag/executors.py", line 101, in _execute_node
    upstream_outputs = self._run_upstream_transforms(
  File "/usr/local/lib/python3.8/dist-packages/merlin/dag/executors.py", line 114, in _run_upstream_transforms
    node_output = self._execute_node(
  File "/usr/local/lib/python3.8/dist-packages/merlin/dag/executors.py", line 107, in _execute_node
    transform_output = self._run_node_transform(node, transform_input, capture_dtypes, strict)
  File "/usr/local/lib/python3.8/dist-packages/merlin/dag/executors.py", line 227, in _run_node_transform
    raise exc
  File "/usr/local/lib/python3.8/dist-packages/merlin/dag/executors.py", line 214, in _run_node_transform
    transformed_data = node.op.transform(selection, input_data)
  File "/usr/local/lib/python3.8/dist-packages/merlin/systems/dag/runtimes/triton/ops/workflow.py", line 92, in 
transform
    raise RuntimeError(inference_response.error().message())
RuntimeError: Error: <class 'AttributeError'> - 'list_iterator' object has no attribute 'names', Traceback: ['  
File "/root/models/0_transformworkflowtriton/1/model.py", line 117, in execute\n    transformed = 
self.runner.run_workflow(input_tensors)\n', '  File 
"/usr/local/lib/python3.8/dist-packages/merlin/systems/workflow/base.py", line 103, in run_workflow\n    
transformed = LocalExecutor().transform(transformable, self.workflow.graph)\n', '  File 
"/usr/local/lib/python3.8/dist-packages/merlin/dag/executors.py", line 87, in transform\n    transformed_data = 
self._execute_node(node, transformable, capture_dtypes, strict)\n', '  File 
"/usr/local/lib/python3.8/dist-packages/merlin/dag/executors.py", line 101, in _execute_node\n    upstream_outputs 
= self._run_upstream_transforms(\n', '  File "/usr/local/lib/python3.8/dist-packages/merlin/dag/executors.py", line
114, in _run_upstream_transforms\n    node_output = self._execute_node(\n', '  File 
"/usr/local/lib/python3.8/dist-packages/merlin/dag/executors.py", line 101, in _execute_node\n    upstream_outputs 
= self._run_upstream_transforms(\n', '  File "/usr/local/lib/python3.8/dist-packages/merlin/dag/executors.py", line
114, in _run_upstream_transforms\n    node_output = self._execute_node(\n', '  File 
"/usr/local/lib/python3.8/dist-packages/merlin/dag/executors.py", line 101, in _execute_node\n    upstream_outputs 
= self._run_upstream_transforms(\n', '  File "/usr/local/lib/python3.8/dist-packages/merlin/dag/executors.py", line
114, in _run_upstream_transforms\n    node_output = self._execute_node(\n', '  File 
"/usr/local/lib/python3.8/dist-packages/merlin/dag/executors.py", line 101, in _execute_node\n    upstream_outputs 
= self._run_upstream_transforms(\n', '  File "/usr/local/lib/python3.8/dist-packages/merlin/dag/executors.py", line
114, in _run_upstream_transforms\n    node_output = self._execute_node(\n', '  File 
"/usr/local/lib/python3.8/dist-packages/merlin/dag/executors.py", line 101, in _execute_node\n    upstream_outputs 
= self._run_upstream_transforms(\n', '  File "/usr/local/lib/python3.8/dist-packages/merlin/dag/executors.py", line
114, in _run_upstream_transforms\n    node_output = self._execute_node(\n', '  File 
"/usr/local/lib/python3.8/dist-packages/merlin/dag/executors.py", line 101, in _execute_node\n    upstream_outputs 
= self._run_upstream_transforms(\n', '  File "/usr/local/lib/python3.8/dist-packages/merlin/dag/executors.py", line
114, in _run_upstream_transforms\n    node_output = self._execute_node(\n', '  File 
"/usr/local/lib/python3.8/dist-packages/merlin/dag/executors.py", line 101, in _execute_node\n    upstream_outputs 
= self._run_upstream_transforms(\n', '  File "/usr/local/lib/python3.8/dist-packages/merlin/dag/executors.py", line
114, in _run_upstream_transforms\n    node_output = self._execute_node(\n', '  File 
"/usr/local/lib/python3.8/dist-packages/merlin/dag/executors.py", line 101, in _execute_node\n    upstream_outputs 
= self._run_upstream_transforms(\n', '  File "/usr/local/lib/python3.8/dist-packages/merlin/dag/executors.py", line
114, in _run_upstream_transforms\n    node_output = self._execute_node(\n', '  File 
"/usr/local/lib/python3.8/dist-packages/merlin/dag/executors.py", line 101, in _execute_node\n    upstream_outputs 
= self._run_upstream_transforms(\n', '  File "/usr/local/lib/python3.8/dist-packages/merlin/dag/executors.py", line
114, in _run_upstream_transforms\n    node_output = self._execute_node(\n', '  File 
"/usr/local/lib/python3.8/dist-packages/merlin/dag/executors.py", line 107, in _execute_node\n    transform_output 
= self._run_node_transform(node, transform_input, capture_dtypes, strict)\n', '  File 
"/usr/local/lib/python3.8/dist-packages/merlin/dag/executors.py", line 227, in _run_node_transform\n    raise 
exc\n', '  File "/usr/local/lib/python3.8/dist-packages/merlin/dag/executors.py", line 214, in 
_run_node_transform\n    transformed_data = node.op.transform(selection, input_data)\n', '  File 
"<ipython-input-10-c1b836251c55>", line 3, in transform\n']

[rnyak]

@BenHall please follow the getting started or end-to-end examples in this folder. we did not maintain tutorial series for a while. thanks.

[BenHall]

@rnyak Thanks. I was attempting end-to-end-session-based but 02-End-to-end-session-based-with-Yoochoose-PyT.ipynb fails with inference to with:

ValueError                                Traceback (most recent call last)
Cell In[29], line 2
      1 from merlin.systems.triton.utils import send_triton_request
----> 2 response = send_triton_request(workflow.input_schema, filtered_batch, model.output_schema.column_names)
      3 print(response)
File /usr/local/lib/python3.8/dist-packages/pandas/core/arrays/masked.py:471, in BaseMaskedArray.astype(self, dtype, copy)
    469 # to_numpy will also raise, but we get somewhat nicer exception messages here
    470 if is_integer_dtype(dtype) and self._hasna:
--> 471     raise ValueError("cannot convert NA to integer")
    472 if is_bool_dtype(dtype) and self._hasna:
    473     # careful: astype_nansafe converts np.nan to True
    474     raise ValueError("cannot convert float NaN to bool")

ValueError: cannot convert NA to integer

[bbozkaya]

@BenHall I duplicated the error above on my end. The solution is to replace the cell that reads and processes the raw data for Triton request by adding a .dropna to get rid of rows with NA values in the Category columns. Also increase 50 to 100 to have enough rows after .dropna. This would be the new code:

import pandas as pd
interactions_merged_df = pd.read_parquet(os.path.join(INPUT_DATA_DIR, "interactions_merged_df.parquet"))
interactions_merged_df = interactions_merged_df.sort_values('timestamp')
batch = interactions_merged_df[-100:]
batch = batch.dropna()
sessions_to_use = batch.session_id.value_counts()
filtered_batch = batch[batch.session_id.isin(sessions_to_use[sessions_to_use.values>1].index.values)]

[rnyak]

Closing the ticket due to low activity.