lava-nc / lava

A Software Framework for Neuromorphic Computing
https://lava-nc.org
Other
555 stars 143 forks source link

Reading Var() causes 'NoneType' object has no attribute 'get' #712

Closed ZoltanV-V closed 1 year ago

ZoltanV-V commented 1 year ago

Describe the bug .get() method for Var doesn't seem to be reading the value correctly and causes a 'NoneType' error.

To reproduce current behavior When I try to run any of the testintegration*.py scripts on a fresh install of the latest version of Lava, it ends in a NoneType error. This function from test_integration_hc_lif_with_c_ref_ports.py is one that fails:

def run_post_mgmt(self):
        self.var1 = self.in_port1.recv()
        self.var2 = self.in_port2.recv()

I get this error that just prints repetitively:

:ERROR: lava.magma.runtime.runtime_services.nxsdk_runtime_service - 'NoneType' object has no attribute 'get'
:ERROR: lava.magma.runtime.runtime_services.nxsdk_runtime_service - 'NoneType' object has no attribute 'get'
:ERROR: lava.magma.runtime.runtime_services.nxsdk_runtime_service - 'NoneType' object has no attribute 'get'

Trying to run a simple inference on a pretrained model that is loaded with NetX also yields the same error:

INFO:DRV:      Mapping chipIds.............Done 0.02ms
INFO:DRV:      Mapping coreIds.............Done 0.67ms
INFO:DRV:      Partitioning neuron groups..Done 0.01s
INFO:DRV:      Mapping axons...............Done 2.14ms
INFO:DRV:      Partitioning MPDS...........Done 4.83ms
INFO:DRV:      Creating Embedded Snips and ChannelsError 0.67ms
06/08/2023 10:08:27PM:ERROR: lava.magma.runtime.runtime_services.nxsdk_runtime_service - 'NoneType' object has no attribute 'get'
06/08/2023 10:08:27PM:ERROR: lava.magma.runtime.runtime_services.nxsdk_runtime_service - 'NoneType' object has no attribute 'get'
06/08/2023 10:08:27PM:ERROR: lava.magma.runtime.runtime_services.nxsdk_runtime_service - 'NoneType' object has no attribute 'get'
06/08/2023 10:08:27PM:ERROR: lava.magma.runtime.runtime_services.nxsdk_runtime_service - 'NoneType' object has no attribute 'get'

I saw a similar issue that was posted last year (https://github.com/lava-nc/lava/issues/226) but having the latest version of Lava, I verified that get() from variable.py has already been changed as per the recommendation from that post so I'm not sure what else could be causing the error.

Environment (please complete the following information):

Additional context Add any other context about the problem here. Remove section otherwise.

PhilippPlank commented 1 year ago

Thank you for reporting this issue. We will look into it!

bamsumit commented 1 year ago

@ZoltanV-V This seems to be Loihi execution related issue. Can you reach out to support here (accessible only to INRC members) with an example workload.

PhilippPlank commented 1 year ago

Could you please let us know which lava-loihi and nxcore version you are using? This issue should be fixed with the current release of lava-loihi/nxcore available on our cloud systems for INRC members.

ZoltanV-V commented 1 year ago

Could you please let us know which lava-loihi and nxcore version you are using? This issue should be fixed with the current release of lava-loihi/nxcore available on our cloud systems for INRC members.

We are currently using lava_loihi-0.4.0 and nxcore-2.20

PhilippPlank commented 1 year ago

This problem should be solved with newer lava-loihi versions. I will close this now, if anyone else experiences similar problems, please open a new issue.