pytorch / pytorch

Tensors and Dynamic neural networks in Python with strong GPU acceleration
https://pytorch.org
Other
80.06k stars 21.52k forks source link

torch._dynamo.export raises Unexpected type in sourceless builder <class 'nemo.core.neural_types.elements.VoidType'> for torchaudio model #112745

Open msis opened 8 months ago

msis commented 8 months ago

🐛 Describe the bug

When exporting the following model using the dynamo_export, there's an error about an Unexpected type in sourceless builder:

import torch

from nemo.collections.asr.models import EncDecCTCModelBPE

model = EncDecCTCModelBPE.from_pretrained(model_name="stt_en_conformer_ctc_small")

model.to(device="cpu").freeze()
model = model.eval()

example_input = model.preprocessor.input_example(max_batch=2)

export_output = torch._dynamo.export(model.preprocessor, args=(example_input,))
$ TORCH_LOGS="+dynamo" TORCHDYNAMO_VERBOSE=1 python onnx_nemo_export.py
[NeMo W 2023-11-02 11:17:30 transformer_bpe_models:59] Could not import NeMo NLP collection which is required for speech translation model.
[NeMo I 2023-11-02 11:17:30 cloud:68] Downloading from: https://api.ngc.nvidia.com/v2/models/nvidia/nemo/stt_en_conformer_ctc_small/versions/1.6.0/files/stt_en_conformer_ctc_small.nemo to /Users/msis/.cache/torch/NeMo/NeMo_1.21.0/stt_en_conformer_ctc_small/5d2d8e5b2b5adb8f5091363c6ba19c55/stt_en_conformer_ctc_small.nemo
100% [.............................................................] 49088734 / 49088734[NeMo I 2023-11-02 11:17:38 common:913] Instantiating model from pre-trained checkpoint
[NeMo I 2023-11-02 11:17:39 mixins:170] Tokenizer SentencePieceTokenizer initialized with 1024 tokens
[NeMo W 2023-11-02 11:17:39 modelPT:161] If you intend to do training or fine-tuning, please call the ModelPT.setup_training_data() method and provide a valid configuration file to setup the train data loader.
    Train config : 
    manifest_filepath: /data/NeMo_ASR_SET/English/v2.0/train/tarred_audio_manifest.json
    sample_rate: 16000
    batch_size: 64
    shuffle: true
    num_workers: 8
    pin_memory: true
    use_start_end_token: false
    trim_silence: false
    max_duration: 20.0
    min_duration: 0.1
    shuffle_n: 2048
    is_tarred: true
    tarred_audio_filepaths: /data/NeMo_ASR_SET/English/v2.0/train/audio__OP_0..4095_CL_.tar

[NeMo W 2023-11-02 11:17:39 modelPT:168] If you intend to do validation, please call the ModelPT.setup_validation_data() or ModelPT.setup_multiple_validation_data() method and provide a valid configuration file to setup the validation data loader(s). 
    Validation config : 
    manifest_filepath:
    - /data/ASR/LibriSpeech/librispeech_withsp2/manifests/librivox-dev-other.json
    - /data/ASR/LibriSpeech/librispeech_withsp2/manifests/librivox-dev-clean.json
    - /data/ASR/LibriSpeech/librispeech_withsp2/manifests/librivox-test-other.json
    - /data/ASR/LibriSpeech/librispeech_withsp2/manifests/librivox-test-clean.json
    sample_rate: 16000
    batch_size: 64
    shuffle: false
    num_workers: 8
    pin_memory: true
    use_start_end_token: false
    is_tarred: false
    tarred_audio_filepaths: na

[NeMo W 2023-11-02 11:17:39 modelPT:174] Please call the ModelPT.setup_test_data() or ModelPT.setup_multiple_test_data() method and provide a valid configuration file to setup the test data loader(s).
    Test config : 
    manifest_filepath:
    - /data/ASR/LibriSpeech/librispeech_withsp2/manifests/librivox-test-other.json
    - /data/ASR/LibriSpeech/librispeech_withsp2/manifests/librivox-dev-clean.json
    - /data/ASR/LibriSpeech/librispeech_withsp2/manifests/librivox-dev-other.json
    - /data/ASR/LibriSpeech/librispeech_withsp2/manifests/librivox-test-clean.json
    sample_rate: 16000
    batch_size: 64
    shuffle: false
    num_workers: 8
    pin_memory: true
    use_start_end_token: false
    is_tarred: false
    tarred_audio_filepaths: na

[NeMo I 2023-11-02 11:17:39 features:289] PADDING: 0
[NeMo I 2023-11-02 11:17:39 save_restore_connector:249] Model EncDecCTCModelBPE was successfully restored from /Users/msis/.cache/torch/NeMo/NeMo_1.21.0/stt_en_conformer_ctc_small/5d2d8e5b2b5adb8f5091363c6ba19c55/stt_en_conformer_ctc_small.nemo.
[NeMo W 2023-11-02 11:17:39 nemo_logging:349] /Users/msis/.virtualenvs/onnxdynamo/lib/python3.10/site-packages/torch/onnx/_internal/exporter.py:130: UserWarning: torch.onnx.dynamo_export only implements opset version 18 for now. If you need to use a different opset version, please register them with register_custom_op.
      warnings.warn(

[2023-11-02 11:17:39,902] torch._dynamo.eval_frame: [DEBUG] skipping __init__ /usr/local/Cellar/python@3.10/3.10.13/Frameworks/Python.framework/Versions/3.10/lib/python3.10/contextlib.py
[2023-11-02 11:17:39,902] torch._dynamo.eval_frame: [DEBUG] skipping __enter__ /usr/local/Cellar/python@3.10/3.10.13/Frameworks/Python.framework/Versions/3.10/lib/python3.10/contextlib.py
[2023-11-02 11:17:39,902] torch._dynamo.eval_frame: [DEBUG] skipping helper /usr/local/Cellar/python@3.10/3.10.13/Frameworks/Python.framework/Versions/3.10/lib/python3.10/contextlib.py
[2023-11-02 11:17:39,902] torch._dynamo.eval_frame: [DEBUG] skipping __init__ /usr/local/Cellar/python@3.10/3.10.13/Frameworks/Python.framework/Versions/3.10/lib/python3.10/contextlib.py
[2023-11-02 11:17:39,902] torch._dynamo.eval_frame: [DEBUG] skipping __enter__ /usr/local/Cellar/python@3.10/3.10.13/Frameworks/Python.framework/Versions/3.10/lib/python3.10/contextlib.py
[2023-11-02 11:17:39,902] torch._dynamo.eval_frame: [DEBUG] skipping enable_dynamic /Users/msis/.virtualenvs/onnxdynamo/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py
[2023-11-02 11:17:39,902] torch._dynamo.eval_frame: [DEBUG] skipping wrapped /Users/msis/.virtualenvs/onnxdynamo/lib/python3.10/site-packages/torch/onnx/_internal/fx/dynamo_graph_extractor.py
[2023-11-02 11:17:39,902] torch._dynamo.convert_frame: [DEBUG] skipping because no torch.* is_typecheck_enabled             /Users/msis/.virtualenvs/onnxdynamo/lib/python3.10/site-packages/nemo/core/classes/common.py 50
[2023-11-02 11:17:39,904] [0/0] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo start tracing __call__ /Users/msis/.virtualenvs/onnxdynamo/lib/python3.10/site-packages/nemo/core/classes/common.py:1030
[2023-11-02 11:17:39,906] [0/0] torch._dynamo.symbolic_convert.__trace_source: [DEBUG] TRACE starts_line __call__ /Users/msis/.virtualenvs/onnxdynamo/lib/python3.10/site-packages/nemo/core/classes/common.py:1030
[2023-11-02 11:17:39,906] [0/0] torch._dynamo.symbolic_convert.__trace_source: [DEBUG]         @wrapt.decorator(enabled=is_typecheck_enabled)
[2023-11-02 11:17:39,941] [0/0] torch._dynamo.variables.builder: [DEBUG] wrap_to_fake L['args'][0][0] (1, 29979) [<DimDynamic.STATIC: 2>, <DimDynamic.STATIC: 2>] [None, None]
[2023-11-02 11:17:39,942] [0/0] torch._dynamo.variables.builder: [DEBUG] wrap_to_fake L['args'][0][1] (1,) [<DimDynamic.STATIC: 2>] [None]
[2023-11-02 11:17:39,943] [0/0] torch._dynamo.symbolic_convert.__trace_source: [DEBUG] TRACE starts_line __call__ /Users/msis/.virtualenvs/onnxdynamo/lib/python3.10/site-packages/nemo/core/classes/common.py:1045
[2023-11-02 11:17:39,943] [0/0] torch._dynamo.symbolic_convert.__trace_source: [DEBUG]             if instance is None:
[2023-11-02 11:17:39,943] [0/0] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST instance []
[2023-11-02 11:17:39,943] [0/0] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST None [NNModuleVariable()]
[2023-11-02 11:17:39,943] [0/0] torch._dynamo.symbolic_convert: [DEBUG] TRACE IS_OP 0 [NNModuleVariable(), ConstantVariable(NoneType)]
[2023-11-02 11:17:39,943] [0/0] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_FALSE 16 [ConstantVariable(bool)]
[2023-11-02 11:17:39,944] [0/0] torch._dynamo.symbolic_convert.__trace_source: [DEBUG] TRACE starts_line __call__ /Users/msis/.virtualenvs/onnxdynamo/lib/python3.10/site-packages/nemo/core/classes/common.py:1048
[2023-11-02 11:17:39,944] [0/0] torch._dynamo.symbolic_convert.__trace_source: [DEBUG]             if not isinstance(instance, Typing):
[2023-11-02 11:17:39,944] [0/0] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL isinstance []
[2023-11-02 11:17:39,944] [0/0] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST instance [BuiltinVariable(isinstance)]
[2023-11-02 11:17:39,944] [0/0] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL Typing [BuiltinVariable(isinstance), NNModuleVariable()]
[2023-11-02 11:17:39,944] [0/0] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 2 [BuiltinVariable(isinstance), NNModuleVariable(), UserDefinedClassVariable()]
[2023-11-02 11:17:39,945] [0/0] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 34 [ConstantVariable(bool)]
[2023-11-02 11:17:39,945] [0/0] torch._dynamo.symbolic_convert.__trace_source: [DEBUG] TRACE starts_line __call__ /Users/msis/.virtualenvs/onnxdynamo/lib/python3.10/site-packages/nemo/core/classes/common.py:1051
[2023-11-02 11:17:39,945] [0/0] torch._dynamo.symbolic_convert.__trace_source: [DEBUG]             if hasattr(instance, 'input_ports') or hasattr(instance, 'output_ports'):
[2023-11-02 11:17:39,945] [0/0] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL hasattr []
[2023-11-02 11:17:39,945] [0/0] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST instance [BuiltinVariable(hasattr)]
[2023-11-02 11:17:39,945] [0/0] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST input_ports [BuiltinVariable(hasattr), NNModuleVariable()]
[2023-11-02 11:17:39,945] [0/0] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 2 [BuiltinVariable(hasattr), NNModuleVariable(), ConstantVariable(str)]
[2023-11-02 11:17:39,945] [0/0] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_TRUE 54 [ConstantVariable(bool)]
[2023-11-02 11:17:39,945] [0/0] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL hasattr []
[2023-11-02 11:17:39,945] [0/0] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST instance [BuiltinVariable(hasattr)]
[2023-11-02 11:17:39,945] [0/0] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST output_ports [BuiltinVariable(hasattr), NNModuleVariable()]
[2023-11-02 11:17:39,946] [0/0] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 2 [BuiltinVariable(hasattr), NNModuleVariable(), ConstantVariable(str)]
[2023-11-02 11:17:39,946] [0/0] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_FALSE 62 [ConstantVariable(bool)]
[2023-11-02 11:17:39,946] [0/0] torch._dynamo.symbolic_convert.__trace_source: [DEBUG] TRACE starts_line __call__ /Users/msis/.virtualenvs/onnxdynamo/lib/python3.10/site-packages/nemo/core/classes/common.py:1058
[2023-11-02 11:17:39,946] [0/0] torch._dynamo.symbolic_convert.__trace_source: [DEBUG]             if self.input_types is typecheck.TypeState.UNINITIALIZED:
[2023-11-02 11:17:39,946] [0/0] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self []
[2023-11-02 11:17:39,946] [0/0] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR input_types [UserDefinedObjectVariable(typecheck)]
[2023-11-02 11:17:39,946] [0/0] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL typecheck [EnumVariable(<enum 'TypeState'>)]
[2023-11-02 11:17:39,947] [0/0] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR TypeState [EnumVariable(<enum 'TypeState'>), UserDefinedClassVariable()]
[2023-11-02 11:17:39,947] [0/0] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR UNINITIALIZED [EnumVariable(<enum 'TypeState'>), UserDefinedClassVariable()]
[2023-11-02 11:17:39,948] [0/0] torch._dynamo.symbolic_convert: [DEBUG] TRACE IS_OP 0 [EnumVariable(<enum 'TypeState'>), EnumVariable(<enum 'TypeState'>)]
[2023-11-02 11:17:39,948] [0/0] torch._dynamo.symbolic_convert: [DEBUG] TRACE POP_JUMP_IF_FALSE 84 [ConstantVariable(bool)]
[2023-11-02 11:17:39,948] [0/0] torch._dynamo.symbolic_convert.__trace_source: [DEBUG] TRACE starts_line __call__ /Users/msis/.virtualenvs/onnxdynamo/lib/python3.10/site-packages/nemo/core/classes/common.py:1059
[2023-11-02 11:17:39,948] [0/0] torch._dynamo.symbolic_convert.__trace_source: [DEBUG]                 self.input_types = instance.input_types
[2023-11-02 11:17:39,948] [0/0] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST instance []
[2023-11-02 11:17:39,948] [0/0] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR input_types [NNModuleVariable()]
[2023-11-02 11:17:39,948] [0/0] torch._dynamo.symbolic_convert: [DEBUG] INLINING <code object input_types at 0x17d4b16e0, file "/Users/msis/.virtualenvs/onnxdynamo/lib/python3.10/site-packages/nemo/collections/asr/modules/audio_preprocessing.py", line 178>
[2023-11-02 11:17:39,949] [0/0] torch._dynamo.symbolic_convert.__trace_source: [DEBUG] TRACE starts_line input_types /Users/msis/.virtualenvs/onnxdynamo/lib/python3.10/site-packages/nemo/collections/asr/modules/audio_preprocessing.py:178 (inline depth: 1)
[2023-11-02 11:17:39,949] [0/0] torch._dynamo.symbolic_convert.__trace_source: [DEBUG]         @property
[2023-11-02 11:17:39,949] [0/0] torch._dynamo.symbolic_convert.__trace_source: [DEBUG] TRACE starts_line input_types /Users/msis/.virtualenvs/onnxdynamo/lib/python3.10/site-packages/nemo/collections/asr/modules/audio_preprocessing.py:183 (inline depth: 1)
[2023-11-02 11:17:39,949] [0/0] torch._dynamo.symbolic_convert.__trace_source: [DEBUG]                 "input_signal": NeuralType(('B', 'T'), AudioSignal(freq=self._sample_rate)),
[2023-11-02 11:17:39,949] [0/0] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL NeuralType []
[2023-11-02 11:17:39,949] [0/0] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST ('B', 'T') [UserDefinedClassVariable()]
[2023-11-02 11:17:39,949] [0/0] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL AudioSignal [UserDefinedClassVariable(), ConstantVariable(tuple)]
[2023-11-02 11:17:39,950] [0/0] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self [UserDefinedClassVariable(), ConstantVariable(tuple), UserDefinedClassVariable()]
[2023-11-02 11:17:39,950] [0/0] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR _sample_rate [UserDefinedClassVariable(), ConstantVariable(tuple), UserDefinedClassVariable(), NNModuleVariable()]
[2023-11-02 11:17:39,950] [0/0] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST ('freq',) [UserDefinedClassVariable(), ConstantVariable(tuple), UserDefinedClassVariable(), ConstantVariable(int)]
[2023-11-02 11:17:39,950] [0/0] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION_KW 1 [UserDefinedClassVariable(), ConstantVariable(tuple), UserDefinedClassVariable(), ConstantVariable(int), ConstantVariable(tuple)]
[2023-11-02 11:17:39,951] [0/0] torch._dynamo.symbolic_convert: [DEBUG] INLINING <code object __init__ at 0x14712aad0, file "/Users/msis/.virtualenvs/onnxdynamo/lib/python3.10/site-packages/nemo/core/neural_types/elements.py", line 193>
[2023-11-02 11:17:39,951] [0/0] torch._dynamo.symbolic_convert.__trace_source: [DEBUG] TRACE starts_line __init__ /Users/msis/.virtualenvs/onnxdynamo/lib/python3.10/site-packages/nemo/core/neural_types/elements.py:193 (inline depth: 2)
[2023-11-02 11:17:39,951] [0/0] torch._dynamo.symbolic_convert.__trace_source: [DEBUG]         def __init__(self, freq: int = None):
[2023-11-02 11:17:39,951] [0/0] torch._dynamo.symbolic_convert.__trace_source: [DEBUG] TRACE starts_line __init__ /Users/msis/.virtualenvs/onnxdynamo/lib/python3.10/site-packages/nemo/core/neural_types/elements.py:194 (inline depth: 2)
[2023-11-02 11:17:39,951] [0/0] torch._dynamo.symbolic_convert.__trace_source: [DEBUG]             self._params = {}
[2023-11-02 11:17:39,951] [0/0] torch._dynamo.symbolic_convert: [DEBUG] TRACE BUILD_MAP 0 []
[2023-11-02 11:17:39,951] [0/0] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self [ConstDictVariable()]
[2023-11-02 11:17:39,951] [0/0] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_ATTR _params [ConstDictVariable(), UserDefinedObjectVariable(AudioSignal)]
[2023-11-02 11:17:39,952] [0/0] torch._dynamo.symbolic_convert.__trace_source: [DEBUG] TRACE starts_line __init__ /Users/msis/.virtualenvs/onnxdynamo/lib/python3.10/site-packages/nemo/core/neural_types/elements.py:195 (inline depth: 2)
[2023-11-02 11:17:39,952] [0/0] torch._dynamo.symbolic_convert.__trace_source: [DEBUG]             self._params['freq'] = freq
[2023-11-02 11:17:39,952] [0/0] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST freq []
[2023-11-02 11:17:39,952] [0/0] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self [ConstantVariable(int)]
[2023-11-02 11:17:39,952] [0/0] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR _params [ConstantVariable(int), UserDefinedObjectVariable(AudioSignal)]
[2023-11-02 11:17:39,952] [0/0] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST freq [ConstantVariable(int), ConstDictVariable()]
[2023-11-02 11:17:39,952] [0/0] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_SUBSCR None [ConstantVariable(int), ConstDictVariable(), ConstantVariable(str)]
[2023-11-02 11:17:39,953] [0/0] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST None []
[2023-11-02 11:17:39,953] [0/0] torch._dynamo.symbolic_convert: [DEBUG] TRACE RETURN_VALUE None [ConstantVariable(NoneType)]
[2023-11-02 11:17:39,953] [0/0] torch._dynamo.symbolic_convert: [DEBUG] DONE INLINING <code object __init__ at 0x14712aad0, file "/Users/msis/.virtualenvs/onnxdynamo/lib/python3.10/site-packages/nemo/core/neural_types/elements.py", line 193>
[2023-11-02 11:17:39,953] [0/0] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 2 [UserDefinedClassVariable(), ConstantVariable(tuple), UserDefinedObjectVariable(AudioSignal)]
[2023-11-02 11:17:39,953] [0/0] torch._dynamo.output_graph: [DEBUG] restore_graphstate: removed 0 nodes
[2023-11-02 11:17:39,953] [0/0] torch._dynamo.symbolic_convert: [DEBUG] empty checkpoint
[2023-11-02 11:17:39,953] [0/0] torch._dynamo.symbolic_convert: [DEBUG] FAILED INLINING <code object input_types at 0x17d4b16e0, file "/Users/msis/.virtualenvs/onnxdynamo/lib/python3.10/site-packages/nemo/collections/asr/modules/audio_preprocessing.py", line 178>
[2023-11-02 11:17:39,953] [0/0] torch._dynamo.output_graph: [DEBUG] restore_graphstate: removed 0 nodes
[2023-11-02 11:17:39,954] [0/0] torch._dynamo.symbolic_convert: [DEBUG] empty checkpoint
Traceback (most recent call last):
  File "/Users/msis/.virtualenvs/onnxdynamo/lib/python3.10/site-packages/torch/onnx/_internal/exporter.py", line 1195, in dynamo_export
    ).export()
  File "/Users/msis/.virtualenvs/onnxdynamo/lib/python3.10/site-packages/torch/onnx/_internal/exporter.py", line 941, in export
    graph_module = self.options.fx_tracer.generate_fx(
  File "/Users/msis/.virtualenvs/onnxdynamo/lib/python3.10/site-packages/torch/onnx/_internal/fx/dynamo_graph_extractor.py", line 199, in generate_fx
    graph_module, graph_guard = torch._dynamo.export(
  File "/Users/msis/.virtualenvs/onnxdynamo/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 1140, in inner
    result_traced = opt_f(*args, **kwargs)
  File "/Users/msis/.virtualenvs/onnxdynamo/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 328, in _fn
    return fn(*args, **kwargs)
  File "/Users/msis/.virtualenvs/onnxdynamo/lib/python3.10/site-packages/torch/onnx/_internal/fx/dynamo_graph_extractor.py", line 154, in wrapped
    return output_adapter.apply(model_func(*args, **kwargs))
  File "/Users/msis/.virtualenvs/onnxdynamo/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 490, in catch_errors
    return callback(frame, cache_entry, hooks, frame_state)
  File "/Users/msis/.virtualenvs/onnxdynamo/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 133, in _fn
    return fn(*args, **kwargs)
  File "/Users/msis/.virtualenvs/onnxdynamo/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 389, in _convert_frame_assert
    return _compile(
  File "/Users/msis/.virtualenvs/onnxdynamo/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 569, in _compile
    guarded_code = compile_inner(code, one_graph, hooks, transform)
  File "/Users/msis/.virtualenvs/onnxdynamo/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 189, in time_wrapper
    r = func(*args, **kwargs)
  File "/Users/msis/.virtualenvs/onnxdynamo/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 491, in compile_inner
    out_code = transform_code_object(code, transform)
  File "/Users/msis/.virtualenvs/onnxdynamo/lib/python3.10/site-packages/torch/_dynamo/bytecode_transformation.py", line 1028, in transform_code_object
    transformations(instructions, code_options)
  File "/Users/msis/.virtualenvs/onnxdynamo/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 458, in transform
    tracer.run()
  File "/Users/msis/.virtualenvs/onnxdynamo/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2074, in run
    super().run()
  File "/Users/msis/.virtualenvs/onnxdynamo/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 724, in run
    and self.step()
  File "/Users/msis/.virtualenvs/onnxdynamo/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 688, in step
    getattr(self, inst.opname)(inst)
  File "/Users/msis/.virtualenvs/onnxdynamo/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1191, in LOAD_ATTR
    result = BuiltinVariable(getattr).call_function(
  File "/Users/msis/.virtualenvs/onnxdynamo/lib/python3.10/site-packages/torch/_dynamo/variables/builtin.py", line 618, in call_function
    result = handler(tx, *args, **kwargs)
  File "/Users/msis/.virtualenvs/onnxdynamo/lib/python3.10/site-packages/torch/_dynamo/variables/builtin.py", line 1088, in call_getattr
    return obj.var_getattr(tx, name).add_options(options)
  File "/Users/msis/.virtualenvs/onnxdynamo/lib/python3.10/site-packages/torch/_dynamo/variables/nn_module.py", line 215, in var_getattr
    ).call_function(tx, [(self)], {})
  File "/Users/msis/.virtualenvs/onnxdynamo/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 261, in call_function
    return super().call_function(tx, args, kwargs)
  File "/Users/msis/.virtualenvs/onnxdynamo/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 90, in call_function
    return tx.inline_user_function_return(
  File "/Users/msis/.virtualenvs/onnxdynamo/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 598, in inline_user_function_return
    result = InliningInstructionTranslator.inline_call(self, fn, args, kwargs)
  File "/Users/msis/.virtualenvs/onnxdynamo/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2179, in inline_call
    return cls.inline_call_(parent, func, args, kwargs)
  File "/Users/msis/.virtualenvs/onnxdynamo/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2286, in inline_call_
    tracer.run()
  File "/Users/msis/.virtualenvs/onnxdynamo/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 724, in run
    and self.step()
  File "/Users/msis/.virtualenvs/onnxdynamo/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 688, in step
    getattr(self, inst.opname)(inst)
  File "/Users/msis/.virtualenvs/onnxdynamo/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 392, in wrapper
    return inner_fn(self, inst)
  File "/Users/msis/.virtualenvs/onnxdynamo/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1115, in CALL_FUNCTION
    self.call_function(fn, args, {})
  File "/Users/msis/.virtualenvs/onnxdynamo/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 562, in call_function
    self.push(fn.call_function(self, args, kwargs))
  File "/Users/msis/.virtualenvs/onnxdynamo/lib/python3.10/site-packages/torch/_dynamo/variables/user_defined.py", line 175, in call_function
    return var.add_options(var.call_method(tx, "__init__", args, kwargs))
  File "/Users/msis/.virtualenvs/onnxdynamo/lib/python3.10/site-packages/torch/_dynamo/variables/user_defined.py", line 301, in call_method
    ).call_function(tx, args, kwargs)
  File "/Users/msis/.virtualenvs/onnxdynamo/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 307, in call_function
    return super().call_function(tx, args, kwargs)
  File "/Users/msis/.virtualenvs/onnxdynamo/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 261, in call_function
    return super().call_function(tx, args, kwargs)
  File "/Users/msis/.virtualenvs/onnxdynamo/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 90, in call_function
    return tx.inline_user_function_return(
  File "/Users/msis/.virtualenvs/onnxdynamo/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 598, in inline_user_function_return
    result = InliningInstructionTranslator.inline_call(self, fn, args, kwargs)
  File "/Users/msis/.virtualenvs/onnxdynamo/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2179, in inline_call
    return cls.inline_call_(parent, func, args, kwargs)
  File "/Users/msis/.virtualenvs/onnxdynamo/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2234, in inline_call_
    sub_locals, closure_cells = func.bind_args(parent, args, kwargs)
  File "/Users/msis/.virtualenvs/onnxdynamo/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 157, in bind_args
    [
  File "/Users/msis/.virtualenvs/onnxdynamo/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 158, in <listcomp>
    wrap(val=arg, source=source)
  File "/Users/msis/.virtualenvs/onnxdynamo/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 33, in wrap_bound_arg
    return SourcelessBuilder()(tx, val).add_options(options)
  File "/Users/msis/.virtualenvs/onnxdynamo/lib/python3.10/site-packages/torch/_dynamo/variables/builder.py", line 1646, in __call__
    unimplemented(f"Unexpected type in sourceless builder {type(value)}")
  File "/Users/msis/.virtualenvs/onnxdynamo/lib/python3.10/site-packages/torch/_dynamo/exc.py", line 172, in unimplemented
    raise Unsupported(msg)
torch._dynamo.exc.Unsupported: Unexpected type in sourceless builder <class 'nemo.core.neural_types.elements.VoidType'>

from user code:
   File "/Users/msis/.virtualenvs/onnxdynamo/lib/python3.10/site-packages/nemo/core/classes/common.py", line 1059, in __call__
    self.input_types = instance.input_types
  File "/Users/msis/.virtualenvs/onnxdynamo/lib/python3.10/site-packages/nemo/collections/asr/modules/audio_preprocessing.py", line 183, in input_types
    "input_signal": NeuralType(('B', 'T'), AudioSignal(freq=self._sample_rate)),

[2023-11-02 11:17:39,960] torch._dynamo.utils: [INFO] TorchDynamo compilation metrics:
[2023-11-02 11:17:39,960] torch._dynamo.utils: [INFO] Function                           Runtimes (s)
[2023-11-02 11:17:39,960] torch._dynamo.utils: [INFO] -------------------------------  --------------
[2023-11-02 11:17:39,960] torch._dynamo.utils: [INFO] _compile.<locals>.compile_inner  

report_dynamo_export.sarif:

{
 "runs":[
  {
   "tool":{
    "driver":{
     "name":"torch.onnx.dynamo_export",
     "contents":[
      "localizedData",
      "nonLocalizedData"
     ],
     "language":"en-US",
     "rules":[],
     "version":"2.1.0"
    }
   },
   "language":"en-US",
   "newlineSequences":[
    "\r\n",
    "\n"
   ],
   "results":[]
  }
 ],
 "version":"2.1.0",
 "schemaUri":"https://docs.oasis-open.org/sarif/sarif/v2.1.0/cs01/schemas/sarif-schema-2.1.0.json"
}

Versions

PyTorch version: 2.1.0 Is debug build: False CUDA used to build PyTorch: None ROCM used to build PyTorch: N/A

OS: macOS 14.1 (x86_64) GCC version: Could not collect Clang version: 15.0.0 (clang-1500.0.40.1) CMake version: version 3.27.7 Libc version: N/A

Python version: 3.10.13 (main, Aug 24 2023, 12:59:26) [Clang 15.0.0 (clang-1500.0.40.1)] (64-bit runtime) Python platform: macOS-14.1-x86_64-i386-64bit Is CUDA available: False CUDA runtime version: No CUDA CUDA_MODULE_LOADING set to: N/A GPU models and configuration: No CUDA Nvidia driver version: No CUDA cuDNN version: No CUDA HIP runtime version: N/A MIOpen runtime version: N/A Is XNNPACK available: True

CPU: Intel(R) Core(TM) i9-10910 CPU @ 3.60GHz

Versions of relevant libraries: [pip3] numpy==1.23.5 [pip3] onnx==1.15.0 [pip3] onnxscript==0.1.0.dev20231102 [pip3] pytorch-lightning==2.0.7 [pip3] torch==2.1.0 [pip3] torchaudio==2.1.0 [pip3] torchmetrics==1.2.0 [pip3] torchvision==0.16.0 [conda] Could not collect

cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @wenzhe-nrv @jiayisunx @chenyang78 @kadeng @chauhang @aakhundov

justinchuby commented 8 months ago

I got

Traceback (most recent call last):
  File "/home/justinchu/dev/pytorch/test.py", line 3, in <module>
    from nemo.collections.asr.models import EncDecCTCModelBPE
  File "/home/justinchu/anaconda3/envs/pytorch/lib/python3.10/site-packages/nemo/collections/asr/__init__.py", line 15, in <module>
    from nemo.collections.asr import data, losses, models, modules
  File "/home/justinchu/anaconda3/envs/pytorch/lib/python3.10/site-packages/nemo/collections/asr/losses/__init__.py", line 15, in <module>
    from nemo.collections.asr.losses.angularloss import AngularSoftmaxLoss
  File "/home/justinchu/anaconda3/envs/pytorch/lib/python3.10/site-packages/nemo/collections/asr/losses/angularloss.py", line 18, in <module>
    from nemo.core.classes import Loss, Typing, typecheck
  File "/home/justinchu/anaconda3/envs/pytorch/lib/python3.10/site-packages/nemo/core/__init__.py", line 16, in <module>
    from nemo.core.classes import *
  File "/home/justinchu/anaconda3/envs/pytorch/lib/python3.10/site-packages/nemo/core/classes/__init__.py", line 18, in <module>
    import pytorch_lightning
  File "/home/justinchu/anaconda3/envs/pytorch/lib/python3.10/site-packages/pytorch_lightning/__init__.py", line 27, in <module>
    from pytorch_lightning.callbacks import Callback  # noqa: E402
  File "/home/justinchu/anaconda3/envs/pytorch/lib/python3.10/site-packages/pytorch_lightning/callbacks/__init__.py", line 14, in <module>
    from pytorch_lightning.callbacks.batch_size_finder import BatchSizeFinder
  File "/home/justinchu/anaconda3/envs/pytorch/lib/python3.10/site-packages/pytorch_lightning/callbacks/batch_size_finder.py", line 24, in <module>
    from pytorch_lightning.callbacks.callback import Callback
  File "/home/justinchu/anaconda3/envs/pytorch/lib/python3.10/site-packages/pytorch_lightning/callbacks/callback.py", line 22, in <module>
    from pytorch_lightning.utilities.types import STEP_OUTPUT
  File "/home/justinchu/anaconda3/envs/pytorch/lib/python3.10/site-packages/pytorch_lightning/utilities/types.py", line 40, in <module>
    from torchmetrics import Metric
  File "/home/justinchu/anaconda3/envs/pytorch/lib/python3.10/site-packages/torchmetrics/__init__.py", line 14, in <module>
    from torchmetrics import functional  # noqa: E402
  File "/home/justinchu/anaconda3/envs/pytorch/lib/python3.10/site-packages/torchmetrics/functional/__init__.py", line 14, in <module>
    from torchmetrics.functional.audio._deprecated import _permutation_invariant_training as permutation_invariant_training
  File "/home/justinchu/anaconda3/envs/pytorch/lib/python3.10/site-packages/torchmetrics/functional/audio/__init__.py", line 14, in <module>
    from torchmetrics.functional.audio.pit import permutation_invariant_training, pit_permutate
  File "/home/justinchu/anaconda3/envs/pytorch/lib/python3.10/site-packages/torchmetrics/functional/audio/pit.py", line 22, in <module>
    from torchmetrics.utilities import rank_zero_warn
  File "/home/justinchu/anaconda3/envs/pytorch/lib/python3.10/site-packages/torchmetrics/utilities/__init__.py", line 14, in <module>
    from torchmetrics.utilities.checks import check_forward_full_state_property
  File "/home/justinchu/anaconda3/envs/pytorch/lib/python3.10/site-packages/torchmetrics/utilities/checks.py", line 25, in <module>
    from torchmetrics.metric import Metric
  File "/home/justinchu/anaconda3/envs/pytorch/lib/python3.10/site-packages/torchmetrics/metric.py", line 30, in <module>
    from torchmetrics.utilities.data import (
  File "/home/justinchu/anaconda3/envs/pytorch/lib/python3.10/site-packages/torchmetrics/utilities/data.py", line 22, in <module>
    from torchmetrics.utilities.imports import _TORCH_GREATER_EQUAL_1_12, _XLA_AVAILABLE
  File "/home/justinchu/anaconda3/envs/pytorch/lib/python3.10/site-packages/torchmetrics/utilities/imports.py", line 41, in <module>
    _TORCHVISION_GREATER_EQUAL_0_8: Optional[bool] = compare_version("torchvision", operator.ge, "0.8.0")
  File "/home/justinchu/anaconda3/envs/pytorch/lib/python3.10/site-packages/lightning_utilities/core/imports.py", line 73, in compare_version
    pkg = importlib.import_module(package)
  File "/home/justinchu/anaconda3/envs/pytorch/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "/home/justinchu/anaconda3/envs/pytorch/lib/python3.10/site-packages/torchvision/__init__.py", line 6, in <module>
    from torchvision import _meta_registrations, datasets, io, models, ops, transforms, utils
  File "/home/justinchu/anaconda3/envs/pytorch/lib/python3.10/site-packages/torchvision/_meta_registrations.py", line 164, in <module>
    def meta_nms(dets, scores, iou_threshold):
  File "/home/justinchu/dev/pytorch/torch/library.py", line 417, in inner
    handle = entry.abstract_impl.register(func, source)
  File "/home/justinchu/dev/pytorch/torch/_library/abstract_impl.py", line 30, in register
    if torch._C._dispatch_has_kernel_for_dispatch_key(self.qualname, "Meta"):
RuntimeError: operator torchvision::nms does not exist

not sure what is wrong yet

Collecting environment information...
PyTorch version: 2.2.0a0+gitbb14805
Is debug build: True
CUDA used to build PyTorch: None
ROCM used to build PyTorch: N/A

OS: Ubuntu 22.04.3 LTS (x86_64)
GCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
Clang version: 14.0.0-1ubuntu1.1
CMake version: version 3.22.1
Libc version: glibc-2.35

Python version: 3.10.9 (main, Jan 11 2023, 15:21:40) [GCC 11.2.0] (64-bit runtime)
Python platform: Linux-6.2.0-1016-azure-x86_64-with-glibc2.35
Is CUDA available: False
CUDA runtime version: No CUDA
CUDA_MODULE_LOADING set to: N/A
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Architecture:                       x86_64
CPU op-mode(s):                     32-bit, 64-bit
Address sizes:                      46 bits physical, 48 bits virtual
Byte Order:                         Little Endian
CPU(s):                             64
On-line CPU(s) list:                0-63
Vendor ID:                          GenuineIntel
Model name:                         Intel(R) Xeon(R) Platinum 8370C CPU @ 2.80GHz
CPU family:                         6
Model:                              106
Thread(s) per core:                 2
Core(s) per socket:                 32
Socket(s):                          1
Stepping:                           6
BogoMIPS:                           5586.87
Flags:                              fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology cpuid pni pclmulqdq vmx ssse3 fma cx16 pcid sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single pti tpr_shadow vnmi ept vpid fsgsbase bmi1 hle avx2 smep bmi2 erms invpcid rtm avx512f avx512dq rdseed adx smap clflushopt avx512cd avx512bw avx512vl xsaveopt xsavec xsaves md_clear
Virtualization:                     VT-x
Hypervisor vendor:                  Microsoft
Virtualization type:                full
L1d cache:                          1.5 MiB (32 instances)
L1i cache:                          1 MiB (32 instances)
L2 cache:                           40 MiB (32 instances)
L3 cache:                           48 MiB (1 instance)
NUMA node(s):                       1
NUMA node0 CPU(s):                  0-63
Vulnerability Gather data sampling: Unknown: Dependent on hypervisor status
Vulnerability Itlb multihit:        KVM: Mitigation: VMX disabled
Vulnerability L1tf:                 Mitigation; PTE Inversion; VMX conditional cache flushes, SMT vulnerable
Vulnerability Mds:                  Mitigation; Clear CPU buffers; SMT Host state unknown
Vulnerability Meltdown:             Mitigation; PTI
Vulnerability Mmio stale data:      Vulnerable: Clear CPU buffers attempted, no microcode; SMT Host state unknown
Vulnerability Retbleed:             Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass:    Vulnerable
Vulnerability Spectre v1:           Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:           Mitigation; Retpolines, STIBP disabled, RSB filling, PBRSB-eIBRS Not affected
Vulnerability Srbds:                Not affected
Vulnerability Tsx async abort:      Mitigation; Clear CPU buffers; SMT Host state unknown

Versions of relevant libraries:
[pip3] flake8==6.0.0
[pip3] flake8-bugbear==23.3.23
[pip3] flake8-comprehensions==3.12.0
[pip3] flake8-executable==2.1.3
[pip3] flake8-logging-format==0.9.0
[pip3] flake8-pyi==23.3.1
[pip3] flake8-simplify==0.19.3
[pip3] mypy==1.4.1
[pip3] mypy-extensions==1.0.0
[pip3] numpy==1.23.5
[pip3] onnx==1.15.0
[pip3] onnx-script==0.1.0
[pip3] onnxruntime==1.14.1
[pip3] onnxscript==0.1.0.dev20231103
[pip3] pytorch==2.0.0a0+git965c7d6
[pip3] pytorch-lightning==2.1.0
[pip3] torch==2.2.0.dev20231103+cpu
[pip3] torchaudio==2.2.0.dev20231103+cpu
[pip3] torchmetrics==1.2.0
[pip3] torchvision==0.17.0.dev20231103+cpu
[conda] Could not collect
thiagocrepaldi commented 7 months ago

I got

Traceback (most recent call last):
  File "/home/justinchu/dev/pytorch/test.py", line 3, in <module>
    from nemo.collections.asr.models import EncDecCTCModelBPE
  File "/home/justinchu/anaconda3/envs/pytorch/lib/python3.10/site-packages/nemo/collections/asr/__init__.py", line 15, in <module>
    from nemo.collections.asr import data, losses, models, modules
  File "/home/justinchu/anaconda3/envs/pytorch/lib/python3.10/site-packages/nemo/collections/asr/losses/__init__.py", line 15, in <module>
    from nemo.collections.asr.losses.angularloss import AngularSoftmaxLoss
  File "/home/justinchu/anaconda3/envs/pytorch/lib/python3.10/site-packages/nemo/collections/asr/losses/angularloss.py", line 18, in <module>
    from nemo.core.classes import Loss, Typing, typecheck
  File "/home/justinchu/anaconda3/envs/pytorch/lib/python3.10/site-packages/nemo/core/__init__.py", line 16, in <module>
    from nemo.core.classes import *
  File "/home/justinchu/anaconda3/envs/pytorch/lib/python3.10/site-packages/nemo/core/classes/__init__.py", line 18, in <module>
    import pytorch_lightning
  File "/home/justinchu/anaconda3/envs/pytorch/lib/python3.10/site-packages/pytorch_lightning/__init__.py", line 27, in <module>
    from pytorch_lightning.callbacks import Callback  # noqa: E402
  File "/home/justinchu/anaconda3/envs/pytorch/lib/python3.10/site-packages/pytorch_lightning/callbacks/__init__.py", line 14, in <module>
    from pytorch_lightning.callbacks.batch_size_finder import BatchSizeFinder
  File "/home/justinchu/anaconda3/envs/pytorch/lib/python3.10/site-packages/pytorch_lightning/callbacks/batch_size_finder.py", line 24, in <module>
    from pytorch_lightning.callbacks.callback import Callback
  File "/home/justinchu/anaconda3/envs/pytorch/lib/python3.10/site-packages/pytorch_lightning/callbacks/callback.py", line 22, in <module>
    from pytorch_lightning.utilities.types import STEP_OUTPUT
  File "/home/justinchu/anaconda3/envs/pytorch/lib/python3.10/site-packages/pytorch_lightning/utilities/types.py", line 40, in <module>
    from torchmetrics import Metric
  File "/home/justinchu/anaconda3/envs/pytorch/lib/python3.10/site-packages/torchmetrics/__init__.py", line 14, in <module>
    from torchmetrics import functional  # noqa: E402
  File "/home/justinchu/anaconda3/envs/pytorch/lib/python3.10/site-packages/torchmetrics/functional/__init__.py", line 14, in <module>
    from torchmetrics.functional.audio._deprecated import _permutation_invariant_training as permutation_invariant_training
  File "/home/justinchu/anaconda3/envs/pytorch/lib/python3.10/site-packages/torchmetrics/functional/audio/__init__.py", line 14, in <module>
    from torchmetrics.functional.audio.pit import permutation_invariant_training, pit_permutate
  File "/home/justinchu/anaconda3/envs/pytorch/lib/python3.10/site-packages/torchmetrics/functional/audio/pit.py", line 22, in <module>
    from torchmetrics.utilities import rank_zero_warn
  File "/home/justinchu/anaconda3/envs/pytorch/lib/python3.10/site-packages/torchmetrics/utilities/__init__.py", line 14, in <module>
    from torchmetrics.utilities.checks import check_forward_full_state_property
  File "/home/justinchu/anaconda3/envs/pytorch/lib/python3.10/site-packages/torchmetrics/utilities/checks.py", line 25, in <module>
    from torchmetrics.metric import Metric
  File "/home/justinchu/anaconda3/envs/pytorch/lib/python3.10/site-packages/torchmetrics/metric.py", line 30, in <module>
    from torchmetrics.utilities.data import (
  File "/home/justinchu/anaconda3/envs/pytorch/lib/python3.10/site-packages/torchmetrics/utilities/data.py", line 22, in <module>
    from torchmetrics.utilities.imports import _TORCH_GREATER_EQUAL_1_12, _XLA_AVAILABLE
  File "/home/justinchu/anaconda3/envs/pytorch/lib/python3.10/site-packages/torchmetrics/utilities/imports.py", line 41, in <module>
    _TORCHVISION_GREATER_EQUAL_0_8: Optional[bool] = compare_version("torchvision", operator.ge, "0.8.0")
  File "/home/justinchu/anaconda3/envs/pytorch/lib/python3.10/site-packages/lightning_utilities/core/imports.py", line 73, in compare_version
    pkg = importlib.import_module(package)
  File "/home/justinchu/anaconda3/envs/pytorch/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "/home/justinchu/anaconda3/envs/pytorch/lib/python3.10/site-packages/torchvision/__init__.py", line 6, in <module>
    from torchvision import _meta_registrations, datasets, io, models, ops, transforms, utils
  File "/home/justinchu/anaconda3/envs/pytorch/lib/python3.10/site-packages/torchvision/_meta_registrations.py", line 164, in <module>
    def meta_nms(dets, scores, iou_threshold):
  File "/home/justinchu/dev/pytorch/torch/library.py", line 417, in inner
    handle = entry.abstract_impl.register(func, source)
  File "/home/justinchu/dev/pytorch/torch/_library/abstract_impl.py", line 30, in register
    if torch._C._dispatch_has_kernel_for_dispatch_key(self.qualname, "Meta"):
RuntimeError: operator torchvision::nms does not exist

not sure what is wrong yet

Collecting environment information...
PyTorch version: 2.2.0a0+gitbb14805
Is debug build: True
CUDA used to build PyTorch: None
ROCM used to build PyTorch: N/A

OS: Ubuntu 22.04.3 LTS (x86_64)
GCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
Clang version: 14.0.0-1ubuntu1.1
CMake version: version 3.22.1
Libc version: glibc-2.35

Python version: 3.10.9 (main, Jan 11 2023, 15:21:40) [GCC 11.2.0] (64-bit runtime)
Python platform: Linux-6.2.0-1016-azure-x86_64-with-glibc2.35
Is CUDA available: False
CUDA runtime version: No CUDA
CUDA_MODULE_LOADING set to: N/A
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Architecture:                       x86_64
CPU op-mode(s):                     32-bit, 64-bit
Address sizes:                      46 bits physical, 48 bits virtual
Byte Order:                         Little Endian
CPU(s):                             64
On-line CPU(s) list:                0-63
Vendor ID:                          GenuineIntel
Model name:                         Intel(R) Xeon(R) Platinum 8370C CPU @ 2.80GHz
CPU family:                         6
Model:                              106
Thread(s) per core:                 2
Core(s) per socket:                 32
Socket(s):                          1
Stepping:                           6
BogoMIPS:                           5586.87
Flags:                              fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology cpuid pni pclmulqdq vmx ssse3 fma cx16 pcid sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single pti tpr_shadow vnmi ept vpid fsgsbase bmi1 hle avx2 smep bmi2 erms invpcid rtm avx512f avx512dq rdseed adx smap clflushopt avx512cd avx512bw avx512vl xsaveopt xsavec xsaves md_clear
Virtualization:                     VT-x
Hypervisor vendor:                  Microsoft
Virtualization type:                full
L1d cache:                          1.5 MiB (32 instances)
L1i cache:                          1 MiB (32 instances)
L2 cache:                           40 MiB (32 instances)
L3 cache:                           48 MiB (1 instance)
NUMA node(s):                       1
NUMA node0 CPU(s):                  0-63
Vulnerability Gather data sampling: Unknown: Dependent on hypervisor status
Vulnerability Itlb multihit:        KVM: Mitigation: VMX disabled
Vulnerability L1tf:                 Mitigation; PTE Inversion; VMX conditional cache flushes, SMT vulnerable
Vulnerability Mds:                  Mitigation; Clear CPU buffers; SMT Host state unknown
Vulnerability Meltdown:             Mitigation; PTI
Vulnerability Mmio stale data:      Vulnerable: Clear CPU buffers attempted, no microcode; SMT Host state unknown
Vulnerability Retbleed:             Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass:    Vulnerable
Vulnerability Spectre v1:           Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:           Mitigation; Retpolines, STIBP disabled, RSB filling, PBRSB-eIBRS Not affected
Vulnerability Srbds:                Not affected
Vulnerability Tsx async abort:      Mitigation; Clear CPU buffers; SMT Host state unknown

Versions of relevant libraries:
[pip3] flake8==6.0.0
[pip3] flake8-bugbear==23.3.23
[pip3] flake8-comprehensions==3.12.0
[pip3] flake8-executable==2.1.3
[pip3] flake8-logging-format==0.9.0
[pip3] flake8-pyi==23.3.1
[pip3] flake8-simplify==0.19.3
[pip3] mypy==1.4.1
[pip3] mypy-extensions==1.0.0
[pip3] numpy==1.23.5
[pip3] onnx==1.15.0
[pip3] onnx-script==0.1.0
[pip3] onnxruntime==1.14.1
[pip3] onnxscript==0.1.0.dev20231103
[pip3] pytorch==2.0.0a0+git965c7d6
[pip3] pytorch-lightning==2.1.0
[pip3] torch==2.2.0.dev20231103+cpu
[pip3] torchaudio==2.2.0.dev20231103+cpu
[pip3] torchmetrics==1.2.0
[pip3] torchvision==0.17.0.dev20231103+cpu
[conda] Could not collect

that happens when your pytorch and torchvision versions do not match.

I managed to repro this error, and it comes from the dynamo evaluator. The repro can be rewritten as

import torch

from nemo.collections.asr.models import EncDecCTCModelBPE

model = EncDecCTCModelBPE.from_pretrained(model_name="stt_en_conformer_ctc_small")

model.to(device="cpu").freeze()
model = model.eval()

example_input = model.preprocessor.input_example(max_batch=2)

_ = torch._dynamo.export(model.preprocessor)(example_input)
thiagocrepaldi commented 7 months ago

I will move this back to the pytorch core triage queue so that they can help us here