Unable to Run Lava Inference Despite Trying Different Framework Versions

haoxiangsnr commented 1 year ago

Hello, thank you for organizing and hosting this challenge.

According to FAQ, the Intel Neuromorphic DNS Challenge requires participants to run their solutions on the test set using the Lava framework for inference. However, we are encountering issues running the lava_inference.ipynb file. We have attempted to combine different versions of the lava-dl and lava frameworks but still have issues running the lava_inference.ipynb file.

Environment

OS: Ubuntu 20.04.3 LTS (x86_64)
CUDA used to build PyTorch: 11.6
PyTorch version: 1.12.1
Libc version: glibc-2.31
Nvidia driver version: 525.116.04
GCC version: (Ubuntu 9.4.0-1ubuntu 1~20.04.1) 9.4.0
GPU 0-9: NVIDIA GeForce RTX 3090

Steps to Reproduce

We followed the instructions in the official documentation to install lava-dl=0.3.2, but the installation process became stuck.

$ conda create -n lava python=3.8
$ conda activate lava
$ pip install https://github.com/lava-nc/lava-dl/releases/download/v0.3.2/lava_dl-0.3.2.tar.gz

Collecting https://github.com/lava-nc/lava-dl/releases/download/v0.3.2/lava_dl-0.3.2.tar.gz
  Downloading https://github.com/lava-nc/lava-dl/releases/download/v0.3.2/lava_dl-0.3.2.tar.gz (40.5 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 40.5/40.5 MB 26.3 MB/s eta 0:00:00
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Collecting lava-nc@ git+https://github.com/lava-nc/lava.git@v0.5.1 (from lava-dl==0.3.2)
  Cloning https://github.com/lava-nc/lava.git (to revision v0.5.1) to /tmp/pip-install-erg3309r/lava-nc_952dc0a7067c4c60a3e1c53e57009a9b
  Running command git clone --filter=blob:none --quiet https://github.com/lava-nc/lava.git /tmp/pip-install-erg3309r/lava-nc_952dc0a7067c4c60a3e1c53e57009a9b
  Running command git checkout -q d321a8b01b7cac017c52f196329c4e486558b8b7

Based on this information, it seems that lava-dl=0.3.2 corresponds to lava=0.5.1. I attempted to install both Lava and Lava-dl using Conda.

conda install lava-dl=0.3.3 -c conda-forge
conda install lava=0.5.1 -c conda-forge

At this point, the installation was successful. However, when running the net = netx.hdf5.Network(trained_folder + '/network.net') cell, an error was encountered.

IndexError: tuple index out of range

This error appears to be due to an issue with resolving the HDF5 format. I attempted to upgrade the lava-dl version to the latest version, 0.3.3, but encountered this error:

ModuleNotFoundError: No module named 'lava.proc.rf'

I attempted to update lava and lava-dl to the latest versions, lava-dl=0.3.3 and lava=0.7.0, respectively. However, when running the second to last cell, an error was encountered.

num_samples = 5
num_steps = 18801
/home/xianghao/mambaforge/envs/lava/lib/python3.8/site-packages/lava/magma/compiler/compiler_graphs.py:869: UserWarning: Cannot import module '<module 'train_sdnn' from '/home/xianghao/proj/IntelNeuromorphicDNSChallenge/baseline_solution/sdnn_delays/train_sdnn.py'>' when searching ProcessModels for Process 'DelayBuffer'.
  warnings.warn(
/home/xianghao/mambaforge/envs/lava/lib/python3.8/site-packages/lava/magma/compiler/compiler_graphs.py:869: UserWarning: Cannot import module '<module 'train_sdnn' from '/home/xianghao/proj/IntelNeuromorphicDNSChallenge/baseline_solution/sdnn_delays/train_sdnn.py'>' when searching ProcessModels for Process 'AmplitudeMixer'.
  warnings.warn(
/home/xianghao/mambaforge/envs/lava/lib/python3.8/site-packages/lava/magma/compiler/compiler_graphs.py:869: UserWarning: Cannot import module '<module 'train_sdnn' from '/home/xianghao/proj/IntelNeuromorphicDNSChallenge/baseline_solution/sdnn_delays/train_sdnn.py'>' when searching ProcessModels for Process 'FixedPtAmp'.
  warnings.warn(
/home/xianghao/mambaforge/envs/lava/lib/python3.8/site-packages/lava/magma/compiler/compiler_graphs.py:869: UserWarning: Cannot import module '<module 'train_sdnn' from '/home/xianghao/proj/IntelNeuromorphicDNSChallenge/baseline_solution/sdnn_delays/train_sdnn.py'>' when searching ProcessModels for Process 'STFT'.
  warnings.warn(
/home/xianghao/mambaforge/envs/lava/lib/python3.8/site-packages/lava/magma/compiler/compiler_graphs.py:869: UserWarning: Cannot import module '<module 'train_sdnn' from '/home/xianghao/proj/IntelNeuromorphicDNSChallenge/baseline_solution/sdnn_delays/train_sdnn.py'>' when searching ProcessModels for Process 'FloatPtAmp'.
  warnings.warn(
/home/xianghao/mambaforge/envs/lava/lib/python3.8/site-packages/lava/magma/compiler/compiler_graphs.py:869: UserWarning: Cannot import module '<module 'train_sdnn' from '/home/xianghao/proj/IntelNeuromorphicDNSChallenge/baseline_solution/sdnn_delays/train_sdnn.py'>' when searching ProcessModels for Process 'AudioSource'.
  warnings.warn(
/home/xianghao/mambaforge/envs/lava/lib/python3.8/site-packages/lava/magma/compiler/compiler_graphs.py:869: UserWarning: Cannot import module '<module 'train_sdnn' from '/home/xianghao/proj/IntelNeuromorphicDNSChallenge/baseline_solution/sdnn_delays/train_sdnn.py'>' when searching ProcessModels for Process 'AudioReceiver'.
  warnings.warn(
/home/xianghao/mambaforge/envs/lava/lib/python3.8/site-packages/lava/magma/compiler/compiler_graphs.py:869: UserWarning: Cannot import module '<module 'train_sdnn' from '/home/xianghao/proj/IntelNeuromorphicDNSChallenge/baseline_solution/sdnn_delays/train_sdnn.py'>' when searching ProcessModels for Process 'ISTFT'.
  warnings.warn(
/home/xianghao/mambaforge/envs/lava/lib/python3.8/site-packages/lava/magma/compiler/compiler_graphs.py:869: UserWarning: Cannot import module '<module 'train_sdnn' from '/home/xianghao/proj/IntelNeuromorphicDNSChallenge/baseline_solution/sdnn_delays/train_sdnn.py'>' when searching ProcessModels for Process 'Bias'.
  warnings.warn(
Encountered Fatal Exception: get_del_wgts() takes 2 positional arguments but 3 were givenEncountered Fatal Exception: get_del_wgts() takes 2 positional arguments but 3 were given

Traceback: Traceback: 

Traceback (most recent call last):
  File "/home/xianghao/mambaforge/envs/lava/lib/python3.8/site-packages/lava/magma/runtime/runtime.py", line 96, in target_fn
    actor.start(*args, **kwargs)
  File "/home/xianghao/mambaforge/envs/lava/lib/python3.8/site-packages/lava/magma/core/model/py/model.py", line 91, in start
    self.run()
  File "/home/xianghao/mambaforge/envs/lava/lib/python3.8/site-packages/lava/magma/core/model/py/model.py", line 212, in run
    raise inst
  File "/home/xianghao/mambaforge/envs/lava/lib/python3.8/site-packages/lava/magma/core/model/py/model.py", line 198, in run
    self._cmd_handlers[cmd]()
  File "/home/xianghao/mambaforge/envs/lava/lib/python3.8/site-packages/lava/magma/core/model/py/model.py", line 581, in _run_async
    self.run_async()
  File "/home/xianghao/mambaforge/envs/lava/lib/python3.8/site-packages/lava/magma/core/model/py/model.py", line 731, in run_async
    py_loihi_model.run_spk(self)
  File "/home/xianghao/mambaforge/envs/lava/lib/python3.8/site-packages/lava/proc/dense/models.py", line 332, in run_spk
    a_accum = self.calc_act(s_in)
  File "/home/xianghao/mambaforge/envs/lava/lib/python3.8/site-packages/lava/proc/dense/models.py", line 226, in calc_act
    np.sum(self.get_del_wgts(self.weights,
TypeError: get_del_wgts() takes 2 positional arguments but 3 were given
Traceback (most recent call last):
  File "/home/xianghao/mambaforge/envs/lava/lib/python3.8/site-packages/lava/magma/runtime/runtime.py", line 96, in target_fn
    actor.start(*args, **kwargs)
...
    np.sum(self.get_del_wgts(self.weights,
TypeError: get_del_wgts() takes 2 positional arguments but 3 were given

Questions

Some questions that I would really appreciate your help with.

Can you please provide detailed environment information in the form of a dependency file, such as a poetry.lock file for Poetry or an environment.yml file for Conda? This would help ensure we have the correct versions of all dependencies.
We have also noticed that all evaluation metrics can be obtained through the evaluation file. Could you please clarify whether the final audio metric for the challenge will be based on the results from the lava_inference.ipynb notebook or the evaluate_network.ipynb notebook?
Regarding the lava_inference.ipynb notebook, we noticed that you mentioned that axonal delay is currently not supported on the Lava platform, which could lead to some performance degradation. Could you kindly confirm if the official baseline result for the challenge is 12.50 dB or 11.47 dB in light of this limitation?

I would greatly appreciate any guidance or support you can provide. Thank you in advance for your help!

bamsumit commented 1 year ago

@haoxiangsnr It is unfortunate that you had installation issues with lava and lava-dl, but it seems you have it working. It looks like your network uses a dense layer with delays which seems to have issues with AsyncRunCfg. Please try using Loihi2SimCfg to move ahead:

run_config = Loihi2SimCfg(select_tag='fixed_pt')
# run_config = AsyncSimCfg(select_tag='fixed_pt')

For Track 1, the evaluation metrics will be based on evaluate_network.ipynb. lava_inference.ipynb needs to demonstrate that the network produces desired results at least for a few samples. This is because CPU inference can be slow, especially using delays.

The baseline is 12.50 dB.

bamsumit commented 11 months ago

@haoxiangsnr I am not sure if you have already seen it. We have updated the lava inference using delay in the network: https://github.com/IntelLabs/IntelNeuromorphicDNSChallenge/blob/main/baseline_solution/sdnn_delays/lava_inference.ipynb

IntelLabs / IntelNeuromorphicDNSChallenge