search

mlcommons / cm4mlops

A collection of portable, reusable and cross-platform automation recipes (CM scripts) with a human-friendly interface and minimal dependencies to make it easier to build, run, benchmark and optimize AI, ML and other applications and systems across diverse and continuously changing models, data sets, software and hardware (cloud/edge)
http://docs.mlcommons.org/cm4mlops/
Apache License 2.0
7 stars 12 forks source link

CM fails to build RNNT #98

Open WarrenSchultz opened 6 days ago

WarrenSchultz commented 6 days ago

Tried using the container build for RNNT, as well as running within an existing RN50 container.

[2024-07-02 14:20:09,097 preprocess_data.py:59 INFO] Converting wav files to npy files for test set...
Dataset loaded with 4.36 hours. Filtered 1.02 hours.
sort_by_duration False
Traceback (most recent call last):
  File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/home/cmuser/CM/repos/local/cache/2ad39391d05c435f/repo/closed/NVIDIA/code/rnnt/tensorrt/preprocess_data.py", line 189, in <module>
    main()
  File "/home/cmuser/CM/repos/local/cache/2ad39391d05c435f/repo/closed/NVIDIA/code/rnnt/tensorrt/preprocess_data.py", line 183, in main
    preprocess_rnnt(data_dir, preprocessed_data_dir)
  File "/home/cmuser/CM/repos/local/cache/2ad39391d05c435f/repo/closed/NVIDIA/code/rnnt/tensorrt/preprocess_data.py", line 76, in preprocess_rnnt
    convert_rnnt_data_main(args)
  File "/home/cmuser/CM/repos/local/cache/2ad39391d05c435f/repo/closed/NVIDIA/code/rnnt/tensorrt/preprocessing/convert_rnnt_data.py", line 222, in main
    audio_preprocessor = AudioPreprocessing(**featurizer_config)
  File "/home/cmuser/CM/repos/local/cache/2ad39391d05c435f/repo/closed/NVIDIA/code/rnnt/tensorrt/preprocessing/preprocessing.py", line 113, in __init__
    self.featurizer = FeatureFactory.from_config(kwargs)
  File "/home/cmuser/CM/repos/local/cache/2ad39391d05c435f/repo/closed/NVIDIA/code/rnnt/tensorrt/preprocessing/parts/features.py", line 356, in from_config
    return featurizer.from_config(cfg, log="log" in feat_type)
  File "/home/cmuser/CM/repos/local/cache/2ad39391d05c435f/repo/closed/NVIDIA/code/rnnt/tensorrt/preprocessing/parts/features.py", line 330, in from_config
    return cls(sample_rate=cfg['sample_rate'], window_size=cfg['window_size'],
  File "/home/cmuser/CM/repos/local/cache/2ad39391d05c435f/repo/closed/NVIDIA/code/rnnt/tensorrt/preprocessing/parts/features.py", line 248, in __init__
    librosa.filters.mel(sample_rate, self.n_fft, n_mels=nfilt, fmin=lowfreq,
TypeError: mel() takes 0 positional arguments but 2 positional arguments (and 3 keyword-only arguments) were given
make: *** [/home/cmuser/CM/repos/local/cache/2ad39391d05c435f/repo/closed/NVIDIA/Makefile.data:36: preprocess_data] Error 1

CM error: Portable CM script failed (name = app-mlperf-inference-nvidia, return code = 256)
arjunsuresh commented 5 days ago

As RNNT is removed from the official models, we are no longer supporting it in the default CM workflow. But you should be able to run it using "_r4.0" variation in the run command where the old Nvidia code will be used (by default we use _r4.1). If you feel the retired benchmarks are important to be maintained please raise an issue here

WarrenSchultz commented 5 days ago

Thanks! We'll discuss internally.