Open msis opened 8 months ago
I got
Traceback (most recent call last):
File "/home/justinchu/dev/pytorch/test.py", line 3, in <module>
from nemo.collections.asr.models import EncDecCTCModelBPE
File "/home/justinchu/anaconda3/envs/pytorch/lib/python3.10/site-packages/nemo/collections/asr/__init__.py", line 15, in <module>
from nemo.collections.asr import data, losses, models, modules
File "/home/justinchu/anaconda3/envs/pytorch/lib/python3.10/site-packages/nemo/collections/asr/losses/__init__.py", line 15, in <module>
from nemo.collections.asr.losses.angularloss import AngularSoftmaxLoss
File "/home/justinchu/anaconda3/envs/pytorch/lib/python3.10/site-packages/nemo/collections/asr/losses/angularloss.py", line 18, in <module>
from nemo.core.classes import Loss, Typing, typecheck
File "/home/justinchu/anaconda3/envs/pytorch/lib/python3.10/site-packages/nemo/core/__init__.py", line 16, in <module>
from nemo.core.classes import *
File "/home/justinchu/anaconda3/envs/pytorch/lib/python3.10/site-packages/nemo/core/classes/__init__.py", line 18, in <module>
import pytorch_lightning
File "/home/justinchu/anaconda3/envs/pytorch/lib/python3.10/site-packages/pytorch_lightning/__init__.py", line 27, in <module>
from pytorch_lightning.callbacks import Callback # noqa: E402
File "/home/justinchu/anaconda3/envs/pytorch/lib/python3.10/site-packages/pytorch_lightning/callbacks/__init__.py", line 14, in <module>
from pytorch_lightning.callbacks.batch_size_finder import BatchSizeFinder
File "/home/justinchu/anaconda3/envs/pytorch/lib/python3.10/site-packages/pytorch_lightning/callbacks/batch_size_finder.py", line 24, in <module>
from pytorch_lightning.callbacks.callback import Callback
File "/home/justinchu/anaconda3/envs/pytorch/lib/python3.10/site-packages/pytorch_lightning/callbacks/callback.py", line 22, in <module>
from pytorch_lightning.utilities.types import STEP_OUTPUT
File "/home/justinchu/anaconda3/envs/pytorch/lib/python3.10/site-packages/pytorch_lightning/utilities/types.py", line 40, in <module>
from torchmetrics import Metric
File "/home/justinchu/anaconda3/envs/pytorch/lib/python3.10/site-packages/torchmetrics/__init__.py", line 14, in <module>
from torchmetrics import functional # noqa: E402
File "/home/justinchu/anaconda3/envs/pytorch/lib/python3.10/site-packages/torchmetrics/functional/__init__.py", line 14, in <module>
from torchmetrics.functional.audio._deprecated import _permutation_invariant_training as permutation_invariant_training
File "/home/justinchu/anaconda3/envs/pytorch/lib/python3.10/site-packages/torchmetrics/functional/audio/__init__.py", line 14, in <module>
from torchmetrics.functional.audio.pit import permutation_invariant_training, pit_permutate
File "/home/justinchu/anaconda3/envs/pytorch/lib/python3.10/site-packages/torchmetrics/functional/audio/pit.py", line 22, in <module>
from torchmetrics.utilities import rank_zero_warn
File "/home/justinchu/anaconda3/envs/pytorch/lib/python3.10/site-packages/torchmetrics/utilities/__init__.py", line 14, in <module>
from torchmetrics.utilities.checks import check_forward_full_state_property
File "/home/justinchu/anaconda3/envs/pytorch/lib/python3.10/site-packages/torchmetrics/utilities/checks.py", line 25, in <module>
from torchmetrics.metric import Metric
File "/home/justinchu/anaconda3/envs/pytorch/lib/python3.10/site-packages/torchmetrics/metric.py", line 30, in <module>
from torchmetrics.utilities.data import (
File "/home/justinchu/anaconda3/envs/pytorch/lib/python3.10/site-packages/torchmetrics/utilities/data.py", line 22, in <module>
from torchmetrics.utilities.imports import _TORCH_GREATER_EQUAL_1_12, _XLA_AVAILABLE
File "/home/justinchu/anaconda3/envs/pytorch/lib/python3.10/site-packages/torchmetrics/utilities/imports.py", line 41, in <module>
_TORCHVISION_GREATER_EQUAL_0_8: Optional[bool] = compare_version("torchvision", operator.ge, "0.8.0")
File "/home/justinchu/anaconda3/envs/pytorch/lib/python3.10/site-packages/lightning_utilities/core/imports.py", line 73, in compare_version
pkg = importlib.import_module(package)
File "/home/justinchu/anaconda3/envs/pytorch/lib/python3.10/importlib/__init__.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "/home/justinchu/anaconda3/envs/pytorch/lib/python3.10/site-packages/torchvision/__init__.py", line 6, in <module>
from torchvision import _meta_registrations, datasets, io, models, ops, transforms, utils
File "/home/justinchu/anaconda3/envs/pytorch/lib/python3.10/site-packages/torchvision/_meta_registrations.py", line 164, in <module>
def meta_nms(dets, scores, iou_threshold):
File "/home/justinchu/dev/pytorch/torch/library.py", line 417, in inner
handle = entry.abstract_impl.register(func, source)
File "/home/justinchu/dev/pytorch/torch/_library/abstract_impl.py", line 30, in register
if torch._C._dispatch_has_kernel_for_dispatch_key(self.qualname, "Meta"):
RuntimeError: operator torchvision::nms does not exist
not sure what is wrong yet
Collecting environment information...
PyTorch version: 2.2.0a0+gitbb14805
Is debug build: True
CUDA used to build PyTorch: None
ROCM used to build PyTorch: N/A
OS: Ubuntu 22.04.3 LTS (x86_64)
GCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
Clang version: 14.0.0-1ubuntu1.1
CMake version: version 3.22.1
Libc version: glibc-2.35
Python version: 3.10.9 (main, Jan 11 2023, 15:21:40) [GCC 11.2.0] (64-bit runtime)
Python platform: Linux-6.2.0-1016-azure-x86_64-with-glibc2.35
Is CUDA available: False
CUDA runtime version: No CUDA
CUDA_MODULE_LOADING set to: N/A
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True
CPU:
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Address sizes: 46 bits physical, 48 bits virtual
Byte Order: Little Endian
CPU(s): 64
On-line CPU(s) list: 0-63
Vendor ID: GenuineIntel
Model name: Intel(R) Xeon(R) Platinum 8370C CPU @ 2.80GHz
CPU family: 6
Model: 106
Thread(s) per core: 2
Core(s) per socket: 32
Socket(s): 1
Stepping: 6
BogoMIPS: 5586.87
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology cpuid pni pclmulqdq vmx ssse3 fma cx16 pcid sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single pti tpr_shadow vnmi ept vpid fsgsbase bmi1 hle avx2 smep bmi2 erms invpcid rtm avx512f avx512dq rdseed adx smap clflushopt avx512cd avx512bw avx512vl xsaveopt xsavec xsaves md_clear
Virtualization: VT-x
Hypervisor vendor: Microsoft
Virtualization type: full
L1d cache: 1.5 MiB (32 instances)
L1i cache: 1 MiB (32 instances)
L2 cache: 40 MiB (32 instances)
L3 cache: 48 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-63
Vulnerability Gather data sampling: Unknown: Dependent on hypervisor status
Vulnerability Itlb multihit: KVM: Mitigation: VMX disabled
Vulnerability L1tf: Mitigation; PTE Inversion; VMX conditional cache flushes, SMT vulnerable
Vulnerability Mds: Mitigation; Clear CPU buffers; SMT Host state unknown
Vulnerability Meltdown: Mitigation; PTI
Vulnerability Mmio stale data: Vulnerable: Clear CPU buffers attempted, no microcode; SMT Host state unknown
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Vulnerable
Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2: Mitigation; Retpolines, STIBP disabled, RSB filling, PBRSB-eIBRS Not affected
Vulnerability Srbds: Not affected
Vulnerability Tsx async abort: Mitigation; Clear CPU buffers; SMT Host state unknown
Versions of relevant libraries:
[pip3] flake8==6.0.0
[pip3] flake8-bugbear==23.3.23
[pip3] flake8-comprehensions==3.12.0
[pip3] flake8-executable==2.1.3
[pip3] flake8-logging-format==0.9.0
[pip3] flake8-pyi==23.3.1
[pip3] flake8-simplify==0.19.3
[pip3] mypy==1.4.1
[pip3] mypy-extensions==1.0.0
[pip3] numpy==1.23.5
[pip3] onnx==1.15.0
[pip3] onnx-script==0.1.0
[pip3] onnxruntime==1.14.1
[pip3] onnxscript==0.1.0.dev20231103
[pip3] pytorch==2.0.0a0+git965c7d6
[pip3] pytorch-lightning==2.1.0
[pip3] torch==2.2.0.dev20231103+cpu
[pip3] torchaudio==2.2.0.dev20231103+cpu
[pip3] torchmetrics==1.2.0
[pip3] torchvision==0.17.0.dev20231103+cpu
[conda] Could not collect
I got
Traceback (most recent call last): File "/home/justinchu/dev/pytorch/test.py", line 3, in <module> from nemo.collections.asr.models import EncDecCTCModelBPE File "/home/justinchu/anaconda3/envs/pytorch/lib/python3.10/site-packages/nemo/collections/asr/__init__.py", line 15, in <module> from nemo.collections.asr import data, losses, models, modules File "/home/justinchu/anaconda3/envs/pytorch/lib/python3.10/site-packages/nemo/collections/asr/losses/__init__.py", line 15, in <module> from nemo.collections.asr.losses.angularloss import AngularSoftmaxLoss File "/home/justinchu/anaconda3/envs/pytorch/lib/python3.10/site-packages/nemo/collections/asr/losses/angularloss.py", line 18, in <module> from nemo.core.classes import Loss, Typing, typecheck File "/home/justinchu/anaconda3/envs/pytorch/lib/python3.10/site-packages/nemo/core/__init__.py", line 16, in <module> from nemo.core.classes import * File "/home/justinchu/anaconda3/envs/pytorch/lib/python3.10/site-packages/nemo/core/classes/__init__.py", line 18, in <module> import pytorch_lightning File "/home/justinchu/anaconda3/envs/pytorch/lib/python3.10/site-packages/pytorch_lightning/__init__.py", line 27, in <module> from pytorch_lightning.callbacks import Callback # noqa: E402 File "/home/justinchu/anaconda3/envs/pytorch/lib/python3.10/site-packages/pytorch_lightning/callbacks/__init__.py", line 14, in <module> from pytorch_lightning.callbacks.batch_size_finder import BatchSizeFinder File "/home/justinchu/anaconda3/envs/pytorch/lib/python3.10/site-packages/pytorch_lightning/callbacks/batch_size_finder.py", line 24, in <module> from pytorch_lightning.callbacks.callback import Callback File "/home/justinchu/anaconda3/envs/pytorch/lib/python3.10/site-packages/pytorch_lightning/callbacks/callback.py", line 22, in <module> from pytorch_lightning.utilities.types import STEP_OUTPUT File "/home/justinchu/anaconda3/envs/pytorch/lib/python3.10/site-packages/pytorch_lightning/utilities/types.py", line 40, in <module> from torchmetrics import Metric File "/home/justinchu/anaconda3/envs/pytorch/lib/python3.10/site-packages/torchmetrics/__init__.py", line 14, in <module> from torchmetrics import functional # noqa: E402 File "/home/justinchu/anaconda3/envs/pytorch/lib/python3.10/site-packages/torchmetrics/functional/__init__.py", line 14, in <module> from torchmetrics.functional.audio._deprecated import _permutation_invariant_training as permutation_invariant_training File "/home/justinchu/anaconda3/envs/pytorch/lib/python3.10/site-packages/torchmetrics/functional/audio/__init__.py", line 14, in <module> from torchmetrics.functional.audio.pit import permutation_invariant_training, pit_permutate File "/home/justinchu/anaconda3/envs/pytorch/lib/python3.10/site-packages/torchmetrics/functional/audio/pit.py", line 22, in <module> from torchmetrics.utilities import rank_zero_warn File "/home/justinchu/anaconda3/envs/pytorch/lib/python3.10/site-packages/torchmetrics/utilities/__init__.py", line 14, in <module> from torchmetrics.utilities.checks import check_forward_full_state_property File "/home/justinchu/anaconda3/envs/pytorch/lib/python3.10/site-packages/torchmetrics/utilities/checks.py", line 25, in <module> from torchmetrics.metric import Metric File "/home/justinchu/anaconda3/envs/pytorch/lib/python3.10/site-packages/torchmetrics/metric.py", line 30, in <module> from torchmetrics.utilities.data import ( File "/home/justinchu/anaconda3/envs/pytorch/lib/python3.10/site-packages/torchmetrics/utilities/data.py", line 22, in <module> from torchmetrics.utilities.imports import _TORCH_GREATER_EQUAL_1_12, _XLA_AVAILABLE File "/home/justinchu/anaconda3/envs/pytorch/lib/python3.10/site-packages/torchmetrics/utilities/imports.py", line 41, in <module> _TORCHVISION_GREATER_EQUAL_0_8: Optional[bool] = compare_version("torchvision", operator.ge, "0.8.0") File "/home/justinchu/anaconda3/envs/pytorch/lib/python3.10/site-packages/lightning_utilities/core/imports.py", line 73, in compare_version pkg = importlib.import_module(package) File "/home/justinchu/anaconda3/envs/pytorch/lib/python3.10/importlib/__init__.py", line 126, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "/home/justinchu/anaconda3/envs/pytorch/lib/python3.10/site-packages/torchvision/__init__.py", line 6, in <module> from torchvision import _meta_registrations, datasets, io, models, ops, transforms, utils File "/home/justinchu/anaconda3/envs/pytorch/lib/python3.10/site-packages/torchvision/_meta_registrations.py", line 164, in <module> def meta_nms(dets, scores, iou_threshold): File "/home/justinchu/dev/pytorch/torch/library.py", line 417, in inner handle = entry.abstract_impl.register(func, source) File "/home/justinchu/dev/pytorch/torch/_library/abstract_impl.py", line 30, in register if torch._C._dispatch_has_kernel_for_dispatch_key(self.qualname, "Meta"): RuntimeError: operator torchvision::nms does not exist
not sure what is wrong yet
Collecting environment information... PyTorch version: 2.2.0a0+gitbb14805 Is debug build: True CUDA used to build PyTorch: None ROCM used to build PyTorch: N/A OS: Ubuntu 22.04.3 LTS (x86_64) GCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0 Clang version: 14.0.0-1ubuntu1.1 CMake version: version 3.22.1 Libc version: glibc-2.35 Python version: 3.10.9 (main, Jan 11 2023, 15:21:40) [GCC 11.2.0] (64-bit runtime) Python platform: Linux-6.2.0-1016-azure-x86_64-with-glibc2.35 Is CUDA available: False CUDA runtime version: No CUDA CUDA_MODULE_LOADING set to: N/A GPU models and configuration: No CUDA Nvidia driver version: No CUDA cuDNN version: No CUDA HIP runtime version: N/A MIOpen runtime version: N/A Is XNNPACK available: True CPU: Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Address sizes: 46 bits physical, 48 bits virtual Byte Order: Little Endian CPU(s): 64 On-line CPU(s) list: 0-63 Vendor ID: GenuineIntel Model name: Intel(R) Xeon(R) Platinum 8370C CPU @ 2.80GHz CPU family: 6 Model: 106 Thread(s) per core: 2 Core(s) per socket: 32 Socket(s): 1 Stepping: 6 BogoMIPS: 5586.87 Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology cpuid pni pclmulqdq vmx ssse3 fma cx16 pcid sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single pti tpr_shadow vnmi ept vpid fsgsbase bmi1 hle avx2 smep bmi2 erms invpcid rtm avx512f avx512dq rdseed adx smap clflushopt avx512cd avx512bw avx512vl xsaveopt xsavec xsaves md_clear Virtualization: VT-x Hypervisor vendor: Microsoft Virtualization type: full L1d cache: 1.5 MiB (32 instances) L1i cache: 1 MiB (32 instances) L2 cache: 40 MiB (32 instances) L3 cache: 48 MiB (1 instance) NUMA node(s): 1 NUMA node0 CPU(s): 0-63 Vulnerability Gather data sampling: Unknown: Dependent on hypervisor status Vulnerability Itlb multihit: KVM: Mitigation: VMX disabled Vulnerability L1tf: Mitigation; PTE Inversion; VMX conditional cache flushes, SMT vulnerable Vulnerability Mds: Mitigation; Clear CPU buffers; SMT Host state unknown Vulnerability Meltdown: Mitigation; PTI Vulnerability Mmio stale data: Vulnerable: Clear CPU buffers attempted, no microcode; SMT Host state unknown Vulnerability Retbleed: Not affected Vulnerability Spec rstack overflow: Not affected Vulnerability Spec store bypass: Vulnerable Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization Vulnerability Spectre v2: Mitigation; Retpolines, STIBP disabled, RSB filling, PBRSB-eIBRS Not affected Vulnerability Srbds: Not affected Vulnerability Tsx async abort: Mitigation; Clear CPU buffers; SMT Host state unknown Versions of relevant libraries: [pip3] flake8==6.0.0 [pip3] flake8-bugbear==23.3.23 [pip3] flake8-comprehensions==3.12.0 [pip3] flake8-executable==2.1.3 [pip3] flake8-logging-format==0.9.0 [pip3] flake8-pyi==23.3.1 [pip3] flake8-simplify==0.19.3 [pip3] mypy==1.4.1 [pip3] mypy-extensions==1.0.0 [pip3] numpy==1.23.5 [pip3] onnx==1.15.0 [pip3] onnx-script==0.1.0 [pip3] onnxruntime==1.14.1 [pip3] onnxscript==0.1.0.dev20231103 [pip3] pytorch==2.0.0a0+git965c7d6 [pip3] pytorch-lightning==2.1.0 [pip3] torch==2.2.0.dev20231103+cpu [pip3] torchaudio==2.2.0.dev20231103+cpu [pip3] torchmetrics==1.2.0 [pip3] torchvision==0.17.0.dev20231103+cpu [conda] Could not collect
that happens when your pytorch and torchvision versions do not match.
I managed to repro this error, and it comes from the dynamo evaluator. The repro can be rewritten as
import torch
from nemo.collections.asr.models import EncDecCTCModelBPE
model = EncDecCTCModelBPE.from_pretrained(model_name="stt_en_conformer_ctc_small")
model.to(device="cpu").freeze()
model = model.eval()
example_input = model.preprocessor.input_example(max_batch=2)
_ = torch._dynamo.export(model.preprocessor)(example_input)
I will move this back to the pytorch core triage queue so that they can help us here
🐛 Describe the bug
When exporting the following model using the
dynamo_export
, there's an error about anUnexpected type in sourceless builder
:report_dynamo_export.sarif
:Versions
PyTorch version: 2.1.0 Is debug build: False CUDA used to build PyTorch: None ROCM used to build PyTorch: N/A
OS: macOS 14.1 (x86_64) GCC version: Could not collect Clang version: 15.0.0 (clang-1500.0.40.1) CMake version: version 3.27.7 Libc version: N/A
Python version: 3.10.13 (main, Aug 24 2023, 12:59:26) [Clang 15.0.0 (clang-1500.0.40.1)] (64-bit runtime) Python platform: macOS-14.1-x86_64-i386-64bit Is CUDA available: False CUDA runtime version: No CUDA CUDA_MODULE_LOADING set to: N/A GPU models and configuration: No CUDA Nvidia driver version: No CUDA cuDNN version: No CUDA HIP runtime version: N/A MIOpen runtime version: N/A Is XNNPACK available: True
CPU: Intel(R) Core(TM) i9-10910 CPU @ 3.60GHz
Versions of relevant libraries: [pip3] numpy==1.23.5 [pip3] onnx==1.15.0 [pip3] onnxscript==0.1.0.dev20231102 [pip3] pytorch-lightning==2.0.7 [pip3] torch==2.1.0 [pip3] torchaudio==2.1.0 [pip3] torchmetrics==1.2.0 [pip3] torchvision==0.16.0 [conda] Could not collect
cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @wenzhe-nrv @jiayisunx @chenyang78 @kadeng @chauhang @aakhundov