AttributeError: 'Delegate' object has no attribute '_library'

BotScutters commented 4 years ago

Hey @Namburger, I've got a new issue here I'm hoping you can help me out with...

My setup:

Using a Coral M.2 Accelerator A+E key in a device running Ubuntu 16.04 The model file is a MobileNet v2 SSD object detection model compiled for edgetpu, and I have successfully run this model file without issue on other similar devices. Running dpkg -l | grep libedgetpu yields: ii libedgetpu1-std:amd64 14.1 amd64 Support library for Edge TPU I'm pretty sure that the only change since my code was working is that it used to be running 14.0 and has just now updated to 14.1.

Running:

import tflite_runtime.interpreter as tflite

model = tflite.Interpreter(
    "/src/object_detection/model/model_edgetpu.tflite", 
    experimental_delegates=[tflite.load_delegate('libedgetpu.so.1')]
)
model.allocate_tensors()

The model.allocate_tensors() call results in the following RuntimeError:

RuntimeError: Encountered unresolved custom op: edgetpu-custom-op.Node number 0 (edgetpu-custom-op) failed to prepare.
Stack (most recent call last):
  File "OID_Launcher.py", line 203, in <module>
    ObjectIdentification(config=config).run()
  File "OID_Launcher.py", line 34, in __init__
    models = self.__load_models(cameras_dict)
  File "OID_Launcher.py", line 75, in __load_models
    logger.critical('Error: Unable to load models. {}'.format(e), exc_info=True, stack_info=True)
Exception ignored in: <bound method Delegate.__del__ of <tflite_runtime.interpreter.Delegate object at 0x7f2a16f79160>>
Traceback (most recent call last):
  File "/home/oid/Envs/object_identification/lib/python3.5/site-packages/tflite_runtime/interpreter.py", line 125, in __del__
    if self._library is not None:
AttributeError: 'Delegate' object has no attribute '_library'
terminate called without an active exception

Do you know what might be causing this new error? Thanks!

Namburger commented 4 years ago

@BotScutters Humn, libedgetpu14.1 should be the same as 14 with added supports for pcie in window, we only added it to linux also to make sure versioning is the same across the board. Can you try reinstalling the tflite_runtime package and then run this simple script to see if inference can be run at all?

import numpy as np 
import sys
from tflite_runtime.interpreter import Interpreter
from tflite_runtime.interpreter import load_delegate

if len(sys.argv) < 2:
    print('Usage:', sys.argv[0], 'model_path')
    exit()

def main():
    """Runs inference with an input tflite model.""" 
    model_path = str(sys.argv[1])
    if model_path.endswith('edgetpu.tflite'):
        print('initialized for edgetpu')
        delegates = [load_delegate('libedgetpu.so.1.0')]                             
        interpreter = Interpreter(model_path, experimental_delegates=delegates)
    else: 
        print('initialized for cpu')
        interpreter = Interpreter(model_path)

    interpreter.allocate_tensors() 
    input_details = interpreter.get_input_details() 
    images = np.zeros(input_details[0]['shape'], input_details[0]['dtype'])
    #print(images)
    interpreter.set_tensor(input_details[0]['index'], images) 
    interpreter.invoke() 
    output_details = interpreter.get_output_details() 
    outputs = interpreter.get_tensor(output_details[0]['index']) 
    print(outputs)
    print('Success.') 

if __name__== '__main__':
    main()

Also, I just realized something (and this may applies to your other system also), but make sure your linux user is in the plugdev group:

sudo usermod -aG plugdev $USER
sudo reboot now

BotScutters commented 4 years ago

@Namburger Unfortunately no progress here, and I'm starting to see this occurring on every device that undergoes the update from 14.0 to 14.1. Since receiving this message:

I've confirmed that my dev device was already a member of the plugdev group.
I ran your test script and received the same error message as before
I ran the above commands to add the user to plugdev anyway (this produced no change) and rebooted
I reinstalled the tflite-runtime library
Both my code and your test script still yield the same error

I've also confirmed on a separate device that the error does not start occurring until I install v14.1 of libedgetpu. The script I'm running for installation is:

# Install Edge TPU Libraries
echo "deb https://packages.cloud.google.com/apt coral-edgetpu-stable main" | sudo tee /etc/apt/sources.list.d/coral-edgetpu.list
curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -
sudo apt-get update

# Libraries required for Mini PCIe Coral Chip
sudo apt-get install -y gasket-dkms libedgetpu1-std

Namburger commented 4 years ago

@BotScutters I see, let me confirm with the team lead who wrote the pcie supports for 14.1, for now, as a workaround maybe you down grade to runtime version 14? k8.tar.gz

You can just untar it and replace those 2 files with what's in /usr/lib/x86_64-linux-gnu. I'm contacting our lead and will try to get this sorted out ASAP

BotScutters commented 4 years ago

Thanks @Namburger, confirmed that downgrading back to v14 works. Please let me know once you've got a fix for this.

Namburger commented 4 years ago

Will do, sorry for this issue

Namburger commented 4 years ago

@BotScutters if you still happens to have a machine that experiencing this issue, could you possibly share the outputs of this?

ldd /usr/lib/x86_64-linux-gnu/libedgetpu.so.1.0

BotScutters commented 4 years ago

➜ ldd /usr/lib/x86_64-linux-gnu/libedgetpu.so.1.0
        linux-vdso.so.1 =>  (0x00007ffe63939000)
        libusb-1.0.so.0 => /lib/x86_64-linux-gnu/libusb-1.0.so.0 (0x00007f78052d5000)
        libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f7804f53000)
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f7804c4a000)
        libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f7804a34000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f7804817000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f780444d000)
        libudev.so.1 => /lib/x86_64-linux-gnu/libudev.so.1 (0x00007f78055e4000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f78054ed000)
        librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f7804245000)

Namburger commented 4 years ago

@BotScutters we don't see how this is possible but we can't really reproduce this issue, can you also share the outputs of these commands?

python3 -c 'print(__import__("tflite_runtime").__git_version__)' 
md5sum /usr/lib/x86_64-linux-gnu/libedgetpu.so.1.0

FYI, usually this error only occurs if the library is not installed at all. Actually, if you can also attach the strace logs too, that would be super

BotScutters commented 4 years ago

Hmm...well, this issue is pretty rapidly spreading to more devices that I've got out in the field so I'm hoping we can resolve it soon...

➜ python3 -c 'print(__import__("tflite_runtime").__git_version__)' 
0.6.0-76902-gd855adf

➜ md5sum /usr/lib/x86_64-linux-gnu/libedgetpu.so.1.0
980ef8d9db4585893d2280c1e2ab61cf  /usr/lib/x86_64-linux-gnu/libedgetpu.so.1.0

I'll send you the strace log separately shortly.

Namburger commented 4 years ago

@BotScutters Okay, so both the md5sum and the ~git_version string of the tflite_runtime package~ are not what I expected, so there must be something wrong during installation.

[Edit] sorry, tflite_runtime is okay, the one you sent didn't include the whole commit string.

Can you check this again:

$ dpkg -l | grep edgetpu 
$ dpkg -L libedgetpu1-std

BotScutters commented 4 years ago

➜ dpkg -l | grep edgetpu
ii  edgetpu                                    13.0                                            all          Edge TPU compiler and runtime
ii  edgetpu-compiler                           14.1                                            amd64        Edge TPU model compiler
ii  libedgetpu1-std:amd64                      14.1                                            amd64        Support library for Edge TPU
ii  python3-edgetpu                            13.0                                            amd64        Edge TPU Python API

➜ dpkg -L libedgetpu1-std
/.
/lib
/lib/udev
/lib/udev/rules.d
/lib/udev/rules.d/60-libedgetpu1-std.rules
/usr
/usr/lib
/usr/lib/x86_64-linux-gnu
/usr/lib/x86_64-linux-gnu/libedgetpu.so.1.0
/usr/share
/usr/share/doc
/usr/share/doc/libedgetpu1-std
/usr/share/doc/libedgetpu1-std/changelog.gz
/usr/share/doc/libedgetpu1-std/copyright
/usr/share/lintian
/usr/share/lintian/overrides
/usr/share/lintian/overrides/libedgetpu1-std
/usr/lib/x86_64-linux-gnu/libedgetpu.so.1

BotScutters commented 4 years ago

Ah, this is actually the same device where you had me manually downgrade to v14, so it's not displaying the error now. Want me to reinstall back up to v14.1 and repost those outputs?

Namburger commented 4 years ago

@BotScutters so I feel that there are some inconsistencies in what is reported. 1) Your outputs of dpkg is showing that you've installed libedgetpu1-std:amd64 14.1, and you mentioned that this is the version that is causing issue for you. 2) However, the md5sum that you gave matches the md5sum for version 14 which really shouldn't have any problem.

I guess what I want is those outputs from a device that is actually having issue, that's important because that's the one that we need to debug, so not the one that's working, it'll cause a lot of confusion

Here is my suggestion, from a machine that is failing:

1) uninstall everything:

sudo apt purge libedgetpu-std

2) Check that this returns nothing:

dpkg -l | grep libedgetpu

3) Then make sure this fails (because there should be nothing loaded):

python3 -c "from ctypes import *; cdll.LoadLibrary('libedgetpu.so.1.0'); print('success')" 
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/lib/python3.7/ctypes/__init__.py", line 434, in LoadLibrary
    return self._dlltype(name)
  File "/usr/lib/python3.7/ctypes/__init__.py", line 356, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: libedgetpu.so.1.0: cannot open shared object file: No such file or directory

4) Then reinstall libedgetpu (suggest using the max frequency because it's faster :) ):

sudo apt install libedgetpu1-max:amd64

5) Then reinstall the tflite_runtime package and ensure that the git version string matches this one:

python3 -c 'print(__import__("tflite_runtime").__git_version__)'                              
0.6.0-76902-gd855adfc5a

BotScutters commented 4 years ago

Hmm. So I upgraded to back to 14.1 and now it's working fine... so what did we do that addressed the issue I was previously seeing?

Namburger commented 4 years ago

Hmm. So I upgraded to back to 14.1 and now it's working fine... so what did we do that addressed the issue I was previously seeing?

I'm not sure, but as of now we all can only suspects that there was a mismatched in tflite_runtime and libedgetpu.so version

BotScutters commented 4 years ago

Okay, trying to work through your above steps and the uninstall isn't quite working right...

➜ sudo apt purge libedgetpu-std
Reading package lists... Done
Building dependency tree       
Reading state information... Done
E: Unable to locate package libedgetpu-std

➜ dpkg -l | grep libedgetpu
ii  libedgetpu1-std:amd64                      14.1                                            amd64        Support library for Edge TPU

Namburger commented 4 years ago

Oh sorry,

sudo apt-purge libedgetpu1-std

BotScutters commented 4 years ago

Okay, it was

sudo apt purge libedgetpu1-std

That works.

For reinstalling, does it matter whether it's max vs std if it's on the M.2 accelerator? I thought I remember reading that for M.2 it dynamically selects frequency and doesn't matter which version it is

Namburger commented 4 years ago

For reinstalling, does it matter whether it's max vs std if it's on the M.2 accelerator? I thought I remember reading that for M.2 it dynamically selects frequency and doesn't matter which version it is

Oh sorry, you're correct, it's generally for usb devices :) https://coral.ai/docs/m2/get-started/#2b-on-windows

BotScutters commented 4 years ago

Ran all of the above to uninstall and reinstall. Still seeing the error. Here are some of the other outputs:

Tflite runtime version string appears slightly different than yours.

➜ python3 -c 'print(__import__("tflite_runtime").__git_version__)'                              
0.6.0-76902-gd855adf

➜ dpkg -l | grep libedgetpu
ii  libedgetpu1-max:amd64                      14.1                                            amd64        Support library for Edge TPU

➜ ldd /usr/lib/x86_64-linux-gnu/libedgetpu.so.1.0
/usr/lib/x86_64-linux-gnu/libedgetpu.so.1.0: /usr/lib/x86_64-linux-gnu/libstdc++.so.6: version `GLIBCXX_3.4.22' not found (required by /usr/lib/x86_64-linux-gnu/libedgetpu.so.1.0)
        linux-vdso.so.1 =>  (0x00007ffd77993000)
        libusb-1.0.so.0 => /lib/x86_64-linux-gnu/libusb-1.0.so.0 (0x00007f4a05896000)
        libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f4a05514000)
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f4a0520b000)
        libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f4a04ff5000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f4a04dd8000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f4a04a0e000)
        libudev.so.1 => /lib/x86_64-linux-gnu/libudev.so.1 (0x00007f4a05ba3000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f4a05aae000)
        librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f4a04806000)

➜ md5sum /usr/lib/x86_64-linux-gnu/libedgetpu.so.1.0
a538e60c77edbefd4a7640631890a0f8  /usr/lib/x86_64-linux-gnu/libedgetpu.so.1.0

Namburger commented 4 years ago

@BotScutters so sorry, I'm really baffled by this, what about this?

python3 -c 'print(__import__("tflite_runtime").__version__)'

and what python version are you using?

BotScutters commented 4 years ago

➜ python3 -c 'print(__import__("tflite_runtime").__version__)'
2.1.0.post1

I'm running Python version 3.5.2. To install the tflite_runtime, I'm using:

➜ pip3 install https://dl.google.com/coral/python/tflite_runtime-2.1.0.post1-cp35-cp35m-linux_x86_64.whl

Namburger commented 4 years ago

@BotScutters I'm really sorry about this issue, I'm trying to get more eyes on this and make sure this doesn't happens again

My suggestion for now is to downgrade to version 14 as per https://github.com/google-coral/edgetpu/issues/167#issuecomment-656224627

BotScutters commented 4 years ago

@Namburger do you have a clean way to specify which version gets installed (i.e. that v14 gets installed) when I'm running the installation command like:

# Libraries required for Mini PCIe Coral Chip
sudo apt-get install -y gasket-dkms libedgetpu1-std

Namburger commented 4 years ago

@BotScutters Unfortunately, our apt packages will always install the latest version and we always expected that the latest will be the most stable one. I'm suspecting that python3.5 could be an issue, going to check now

Namburger commented 4 years ago

@BotScutters Sorry for keep bothering, can you also show this outputs?

uname -a
cat /etc/os-release

We are trying to run from the same exact environment, thanks

[edit] Okay I reproduced with Ubuntu16.04 docker image

BotScutters commented 4 years ago

➜ cat /etc/os-release
NAME="Ubuntu"
VERSION="16.04.6 LTS (Xenial Xerus)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 16.04.6 LTS"
VERSION_ID="16.04"
HOME_URL="http://www.ubuntu.com/"
SUPPORT_URL="http://help.ubuntu.com/"
BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/"
VERSION_CODENAME=xenial
UBUNTU_CODENAME=xenial

Namburger commented 4 years ago

@BotScutters Okay, we pinpointed the issue, The library was built on a newer OS so it required newer glibc version. Your ldd outputs didn't show it the first time (most likely because you downgraded) but the second time:

ldd /usr/lib/x86_64-linux-gnu/libedgetpu.so.1.0
/usr/lib/x86_64-linux-gnu/libedgetpu.so.1.0: /usr/lib/x86_64-linux-gnu/libstdc++.so.6: version `GLIBCXX_3.4.22' not found (required by /usr/lib/x86_64-linux-gnu/libedgetpu.so.1.0)

I don't know if there is a way to fix this, unfortunately, unless you upgrade your OS, but that is not ideal. My suggestion for now is version 14 since it does not makes any differences, but we'll make sure this doesn't repeat for future release

Our apologies for this issue

BotScutters commented 4 years ago

Hmm... upgrading the OS is definitely not ideal or simple to change, since this is just one component of a larger system on many devices...

If I'm reading this correctly:

libedgetpu version 14 works fine on Ubuntu 16.04
libedgetpu version 14.1 does not work with Ubuntu 16.04
Only the latest version can be installed via the apt install command, so specifying that devices should only install version 14 is not an option

I can work with ensuring all of my existing devices only use version 14 per #167 (comment), but I'm wondering about the state going forward. Do you expect the next release to address this and support Ubuntu 16.04 again?

Namburger commented 4 years ago

@BotScutters

upgrading the OS is definitely not ideal or simple to change

I totally understand, from what I've seen you have multiple systems with too many component, this is a bug change that could break other components.

Only the latest version can be installed via the apt install command, so specifying that devices should only install version 14 is not an option

That statement is true for using the our apt package, but you can always install using our zip package from here:

wget https://dl.google.com/coral/edgetpu_api/edgetpu_runtime_20200331.zip
unzip edgetpu_runtime_20200331.zip
sudo bash install.sh

I would also keep a backup of the tflite_runtime 2.1.0.post1 pip package for your Ubuntu16 system.

Do you expect the next release to address this and support Ubuntu 16.04 again?

I can't comment on this at the moment, unfortunately :/ We honestly didn't expect older systems to still be using our libraries, but this issue will make a really strong case for us to keep the supports. I'll keep you updated as soon as I can.

Namburger commented 4 years ago

@BotScutters Okay we've fixed the apt package for now :)

You can simply uninstall it, update apt package and reinstall it and it should works now:

sudo apt purge libedgetpu1-std
sudo apt update
sudo apt get libedgetpu1-std

BotScutters commented 4 years ago

@Namburger awesome, great to hear! Thanks for all your support getting this working!

BotScutters commented 4 years ago

Finally did this update and confirmed the new version is running stably. Thanks again, @Namburger !

BasavaG commented 2 years ago

Hi all, I am facing the same issue windows10. Any work around? Please help me,

Moosems commented 8 months ago

Happening to me as well on my MacBook.

google-coral / edgetpu

AttributeError: 'Delegate' object has no attribute '_library' #167

My setup:

Running: