search

cornell-zhang / heterocl

HeteroCL: A Multi-Paradigm Programming Infrastructure for Software-Defined Heterogeneous Computing
https://cornell-zhang.github.io/heterocl/
Apache License 2.0
326 stars 92 forks source link

munmap_chunk(): invalid pointer if importing keras and heterocl together #367

Closed hecmay closed 3 years ago

hecmay commented 3 years ago

Super weird error. If I import heterocl and than tensorflow.keras, I will get an error from libhcl.so. To reproduce the error

(ubuntu) [sx233@brg-zhang-xcel convnet]$ python
Python 3.8.8 (default, Feb 24 2021, 21:46:12) 
[GCC 7.3.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import heterocl as hcl
>>> import numpy as np
>>> from tensorflow import keras
*** Error in `python': munmap_chunk(): invalid pointer: 0x00007ffd256be760 ***
======= Backtrace: =========
/lib64/libc.so.6(+0x7f474)[0x7fd757f94474]
/work/shared/users/phd/sx233/heterocl/tvm/lib/libhcl.so(_ZNSt10_HashtableISsSsSaISsENSt8__detail9_IdentityESt8equal_toISsESt4hashISsENS1_18_Mod_range_hashingENS1_20_Default_ranged_hashENS1_20_Prime_rehash_policyENS1_17_Hashtable_traitsILb1ELb1ELb1EEEE21_M_insert_unique_nodeEmmPNS1_10_Hash_nodeISsLb1EEE+0xc9)[0x7fd6d3615929]
/home/sx233/.local/lib/python3.8/site-packages/tensorflow-2.2.1-py3.8-linux-x86_64.egg/tensorflow/python/../libtensorflow_framework.so.2(+0xd64567)[0x7fd61e829567]
/home/sx233/.local/lib/python3.8/site-packages/tensorflow-2.2.1-py3.8-linux-x86_64.egg/tensorflow/python/../libtensorflow_framework.so.2(_ZNK10tensorflow10OpRegistry21RegisterAlreadyLockedERKSt8functionIFNS_6StatusEPNS_18OpRegistrationDataEEE+0xe6)[0x7fd61e965ce6]
/home/sx233/.local/lib/python3.8/site-packages/tensorflow-2.2.1-py3.8-linux-x86_64.egg/tensorflow/python/../libtensorflow_framework.so.2(_ZNK10tensorflow10OpRegistry12CallDeferredEv+0x80)[0x7fd61e9663b0]
/home/sx233/.local/lib/python3.8/site-packages/tensorflow-2.2.1-py3.8-linux-x86_64.egg/tensorflow/python/../libtensorflow_framework.so.2(_ZNK10tensorflow10OpRegistry20ProcessRegistrationsEv+0x2a)[0x7fd61e96645a]
/home/sx233/.local/lib/python3.8/site-packages/tensorflow-2.2.1-py3.8-linux-x86_64.egg/tensorflow/python/../libtensorflow_framework.so.2(_ZN10tensorflow11LoadLibraryEPKcPPvPPKvPm+0x879)[0x7fd61e829e09]
/home/sx233/.local/lib/python3.8/site-packages/tensorflow-2.2.1-py3.8-linux-x86_64.egg/tensorflow/python/_pywrap_tensorflow_internal.so(TF_LoadLibrary+0x37)[0x7fd622be3dd7]
/home/sx233/.local/lib/python3.8/site-packages/tensorflow-2.2.1-py3.8-linux-x86_64.egg/tensorflow/python/_pywrap_tf_session.so(+0x32c14)[0x7fd61c6acc14]
/home/sx233/.local/lib/python3.8/site-packages/tensorflow-2.2.1-py3.8-linux-x86_64.egg/tensorflow/python/_pywrap_tf_session.so(+0x29525)[0x7fd61c6a3525]
python(PyCFunction_Call+0x58)[0x56220f9e72d8]
python(_PyObject_MakeTpCall+0x23c)[0x56220f9d6edc]
python(_PyEval_EvalFrameDefault+0x45a9)[0x56220fa62879]
python(_PyFunction_Vectorcall+0x10b)[0x56220fa2d4bb]
python(+0x1039bd)[0x56220f9969bd]
python(_PyEval_EvalCodeWithName+0x300)[0x56220fa2c760]
python(PyEval_EvalCode+0x23)[0x56220fac14e3]
python(+0x2563a3)[0x56220fae93a3]
python(+0x1470d9)[0x56220f9da0d9]

It works fine if we import keras and then heterocl

hecmay commented 3 years ago

I am closing this issue since it should have nothing to do with heterocl. I re-tried the same steps on another server and it worked totally fine.

(test) [sx233@zhang-21 issues]$ python
Python 3.6.10 |Anaconda, Inc.| (default, May  8 2020, 02:54:21)
[GCC 7.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import heterocl as hcl
>>> import numpy as np
>>> from tensorflow import keras
>>>