ERROR error in DLRModel instantiation TVMError: key 0 is not supported

[gillouche]

Hello,

I am building a FastAI model with SageMaker Neo for my Raspberry Pi 4 model B. I am trying to deploy it with AWS Greengrass. DLR is directly installed on my RPI4 using this link from this page.

If I connect to my RPI4 with SSH and load the model with the python interpreter, it works.

pi@raspberrypi:~/tmp $ python3
Python 3.7.3 (default, Dec 20 2019, 18:57:59) 
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import dlr
>>> model = dlr.DLRModel('./cpu')
>>> model
<dlr.api.DLRModel object at 0xb66b1490>
>>> exit()

Unfortunately, the same model downloaded from S3 in the Lambda deployed by AWS greengrass gives me that error.

[2020-05-26T19:09:49.82+02:00][INFO]-main.py:48,Starting main. [2020-05-26T19:09:52.2+02:00][INFO]-main.py:55,Webcam successfully initialized, triggering the processing. [2020-05-26T19:09:52.2+02:00][INFO]-main.py:26,Starting webcam processing and inference. [2020-05-26T19:09:52.2+02:00][INFO]-main.py:30,Initializing the process with image sampling number (15), face min confidence (0.9), mask min confidence (0.9). [2020-05-26T19:09:52.201+02:00][INFO]-inference_wrapper.py:24,The model archive does not exist at /tmp/model.zip. Downloading from S3 [2020-05-26T19:09:58.352+02:00][INFO]-inference_wrapper.py:30,Unzipping done in /tmp [2020-05-26T19:09:58.352+02:00][INFO]-inference_wrapper.py:32,Loading the face detection model. [2020-05-26T19:09:58.435+02:00][INFO]-inference_wrapper.py:38,Loading the mask detection model. [2020-05-26T19:09:58.981+02:00][ERROR]-init.py:1037,2020-05-26 19:09:58,831 ERROR error in DLRModel instantiation TVMError: key 0 is not supported [2020-05-26T19:09:58.981+02:00][ERROR]-Stack trace: [2020-05-26T19:09:58.981+02:00][ERROR]- File "/home/pi/workplace/neo-ai-dlr/3rdparty/tvm/src/runtime/graph/graph_runtime.h", line 405 [2020-05-26T19:09:58.981+02:00][ERROR]- [bt] (0) /usr/local/dlr/libdlr.so(dmlc::LogMessageFatal::~LogMessageFatal()+0x38) [0xaa4f8338] [2020-05-26T19:09:58.981+02:00][ERROR]- [bt] (1) /usr/local/dlr/libdlr.so(tvm::runtime::GraphRuntime::Load(dmlc::JSONReader*)+0x1c78) [0xaa5a3d6c] [2020-05-26T19:09:58.981+02:00][ERROR]- [bt] (2) /usr/local/dlr/libdlr.so(tvm::runtime::GraphRuntime::Init(std::cxx11::basic_string<char, std::char_traits, std::allocator > const&, tvm::runtime::Module, std::vector<DLContext, std::allocator > const&)+0x190) [0xaa591be0] [2020-05-26T19:09:58.981+02:00][ERROR]- [bt] (3) /usr/local/dlr/libdlr.so(dlr::TVMModel::SetupTVMModule(std::vector<std::cxx11::basic_string<char, std::char_traits, std::allocator >, std::allocator<std::cxx11::basic_string<char, std::char_traits, std::allocator > > >)+0xc88) [0xaa50bb64] [2020-05-26T19:09:58.981+02:00][ERROR]- [bt] (4) /usr/local/dlr/libdlr.so(CreateDLRModel+0x1bd4) [0xaa4f4dc0] [2020-05-26T19:09:58.981+02:00][ERROR]-Traceback (most recent call last): [2020-05-26T19:09:58.981+02:00][ERROR]- File "/usr/local/lib/python3.7/dist-packages/dlr/api.py", line 82, in init__ [2020-05-26T19:09:58.981+02:00][ERROR]- self._impl = DLRModelImpl(model_path, dev_type, dev_id) [2020-05-26T19:09:58.981+02:00][ERROR]- File "/usr/local/lib/python3.7/dist-packages/dlr/dlr_model.py", line 111, in init [2020-05-26T19:09:58.981+02:00][ERROR]- c_int(dev_id))) [2020-05-26T19:09:58.981+02:00][ERROR]- File "/usr/local/lib/python3.7/dist-packages/dlr/dlr_model.py", line 26, in _check_call [2020-05-26T19:09:58.981+02:00][ERROR]- raise DLRError(_LIB.DLRGetLastError().decode('ascii')) [2020-05-26T19:09:58.981+02:00][ERROR]-dlr.dlr_model.DLRError: TVMError: key 0 is not supported

I am trying to find what this error means to see if I can fix it myself. Would you have any idea of the reason I get this error in the Greengrass Lambda and not the python interpreter ?

Thank you.

[samskalicky]

@gillouche it seems to be failing when trying to read the json from your compiled model:

ERROR error in DLRModel instantiation TVMError: key 0 is not supported
[bt] (1) /usr/local/dlr/libdlr.so(tvm::runtime::GraphRuntime::Load(dmlc::JSONReader*)+0x1c78) [0xaa5a3d6c]

Can you share the graph.json for your compile model?

[gillouche]

Hi @samskalicky,

In the tar.gz created I only have compiled_model.json. If that's what you meant, I attached it compiled_model.json.zip

If not, how can I get it ?

Thanks.

[samskalicky]

@gillouche can you confirm that you're giving greengrass the s3 location of your compiled model and not the original uncompiled model?

[gillouche]

Hi @samskalicky

sorry for the slow response times. I am currently downloading the models from S3, unzip them in /tmp and load them manually. If I remember correctly, I tried both methods (manually download zip file with models from S3 and this procedure.

The project is moving fast and we are not using DLR anymore for the moment so I don't have much time to test again so I am going to close the comment. I will open a new one or reopen this one if I encounter the problem again in the future when we need DLR.

Thanks for your help.