iree-org / iree

A retargetable MLIR-based machine learning compiler and runtime toolkit.
http://iree.dev/
Apache License 2.0
2.82k stars 608 forks source link

Nightly 'samples' build of Colab notebooks broken recently #18919

Open ScottTodd opened 4 days ago

ScottTodd commented 4 days ago

What happened?

https://github.com/iree-org/iree/actions/workflows/samples.yml has been failing for two days. Sample logs: https://github.com/iree-org/iree/actions/runs/11547361887/job/32137116277

tail of logs:

---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
ImportError: numpy.core.multiarray failed to import

The above exception was the direct cause of the following exception:

SystemError                               Traceback (most recent call last)
Cell In[9], line 1
----> 1 interpreter = tflite.Interpreter(
      2 model_path=str(ARTIFACTS_DIR.joinpath("text_classification.tflite")))
      3 interpreter.allocate_tensors()
      4 input_details = interpreter.get_input_details()

File ~/work/iree/iree/.notebook.venv/lib/python3.11/site-packages/tflite_runtime/interpreter.py:470, in Interpreter.__init__(self, model_path, model_content, experimental_delegates, num_threads, experimental_op_resolver_type, experimental_preserve_all_tensors, experimental_disable_delegate_clustering, experimental_default_delegate_latest_features)
    464 custom_op_registerers_by_name = [
    465     x for x in self._custom_op_registerers if isinstance(x, str)
    466 ]
    467 custom_op_registerers_by_func = [
    468     x for x in self._custom_op_registerers if not isinstance(x, str)
    469 ]
--> 470 self._interpreter = _interpreter_wrapper.CreateWrapperFromFile(
    471 model_path,
    472 op_resolver_id,
    473 custom_op_registerers_by_name,
    474 custom_op_registerers_by_func,
    475 experimental_preserve_all_tensors,
    476 experimental_disable_delegate_clustering,
    477 int(num_threadsor1),
    478 experimental_default_delegate_latest_features,
    479 )
    480 if not self._interpreter:
    481   raise ValueError('Failed to open {}'.format(model_path))

SystemError: <built-in method CreateWrapperFromFile of PyCapsule object at 0x7fdc9b6ea220> returned a result with an exception set

+ deactivate
F
======================================================================
FAIL: test_tensorflow_hub_import.ipynb (__main__.ColabNotebookTests.test_tensorflow_hub_import.ipynb)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/runner/work/iree/iree/./samples/colab/test_notebooks.py", line 48, in unit_test
    self.assertEqual(completed_process.returncode, 0)
AssertionError: 1 != 0

======================================================================
FAIL: test_tflite_text_classification.ipynb (__main__.ColabNotebookTests.test_tflite_text_classification.ipynb)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/runner/work/iree/iree/./samples/colab/test_notebooks.py", line 48, in unit_test
    self.assertEqual(completed_process.returncode, 0)
AssertionError: 1 != 0

----------------------------------------------------------------------
Ran 10 tests in 547.453s

FAILED (failures=2, skipped=2)

There is also another error higher up:

CompilerToolError: Error invoking IREE compiler tool iree-compile
Error code: 1
Diagnostics:
failed to translate executables
-:709:12: error: 'tensor.extract_slice' op unhandled operation when converting to destination passing style
-:267:3: note: called from
-:709:12: error: failed to run translation of source executable to target executable for backend #hal.executable.target<"vmvx", "vmvx-bytecode-fb", {ukernels = "none"}>
-:267:3: note: called from

Steps to reproduce your issue

  1. Workflow file: https://github.com/iree-org/iree/blob/main/.github/workflows/samples.yml
  2. Test script: https://github.com/iree-org/iree/blob/main/samples/colab/test_notebooks.py

Check if the notebooks also fail on Colab itself: https://github.com/iree-org/iree/tree/main/samples/colab (the test script simulates running in Colab but software package versions can differ easily)

What component(s) does this issue relate to?

Python

Version information

Not sure from the current logs. Check the date ranges against https://github.com/iree-org/iree/releases

Additional context

No response

ScottTodd commented 9 hours ago

As needed, we can skip or xfail these notebook tests to get the workflow passing again: https://github.com/iree-org/iree/blob/8bfcc8546c17530b778e85791f2a407cac1b3f84/samples/colab/test_notebooks.py#L14-L26

Instructions for updating the notebooks themselves are at https://github.com/iree-org/iree/tree/main/samples/colab#working-with-github