Closed sonack closed 5 years ago
May you provide an example code sample of using the interpolate
? Thanks
Hi sonack,
Thanks for reaching out! If installed and imported correctly, interpolate should appear in the list of registered converters. Could you share the following information
Which torch / TensorRT version are you using?
What steps did you take to install torch2trt with plugins (ie. did you previously install without plugins?)
For a full example, the image segmentation notebook converts a model that contains interpolate layers.
https://github.com/NVIDIA-AI-IOT/torch2trt/blob/master/notebooks/image_segmentation/conversion.ipynb
However, in interpolate.py we also register several module tests that would probably be the easiest thing to try first. You can execute them by calling
python3 -m torch2trt.test --name=interpolate
This will print information like the throughput, latency, as well as max absolute error (compared to PyTorch).
Please let me know if you have any other questions!
Best, John
Hi John,
Hi sonack,
Thanks for reaching out! If installed and imported correctly, interpolate should appear in the list of registered converters. Could you share the following information
- Which torch / TensorRT version are you using?
The used python version:
The used torch information:
The used TensorRT install package:
TensorRT-5.1.5.0.Ubuntu-14.04.5.x86_64-gnu.cuda-10.0.cudnn7.5.tar.gz
- What steps did you take to install torch2trt with plugins (ie. did you previously install without plugins?)
In order to compile torch2trt without error, I have made some fix in build.py
Below is the diff info:
diff --git a/build.py b/build.py
index 68b103b..024166f 100644
--- a/build.py
+++ b/build.py
@@ -7,23 +7,32 @@ PLUGINS = [
'interpolate',
]
-BASE_FOLDER = 'torch2trt/converters'
+def get_torch_dir():
+ return '/home/xxx/bin/libtorch/libtorch'
+
+def get_cuda_dir(version = '10.0'):
+ return '/usr/local/cuda-{}'.format(version)
+
+BASE_FOLDER = 'torch2trt/converters'
+# -lnvinfer
NINJA_STR = Template(
"""
rule link
- command = g++ -shared -o $$out $$in -L$torch_dir/lib -L$cuda_dir/lib64 -lc10 -lc10_cuda -ltorch -lcudart -lcaffe2 -lcaffe2_gpu -lprotobuf -lprotobuf-lite -pthread -lpthread -lnvinfer
+ command = g++ -shared -o $$out $$in -L$torch_dir/lib -L$cuda_dir/lib64 -lc10 -lc10_cuda -ltorch -lcudart -lcaffe2 -lcaffe2_gpu -lprotobuf -lprotobuf-lite -pthread -lpthread -std=c++11
rule protoc
command = protoc $$in --cpp_out=. --python_out=.
rule cxx
- command = g++ -c -fPIC $$in -I$cuda_dir/include -I$torch_dir/include -I$torch_dir/include/torch/csrc/api/include -I.
+ command = g++ -c -fPIC $$in -I$cuda_dir/include -I$torch_dir/include -I$torch_dir/include/torch/csrc/api/include -I. -std=c++11
"""
).substitute({
- 'torch_dir': imp.find_module('torch')[1],
- 'cuda_dir': '/usr/local/cuda'
+ # 'torch_dir': imp.find_module('torch')[1],
+ 'torch_dir': get_torch_dir(),
+ # 'cuda_dir': '/usr/local/cuda'
+ 'cuda_dir': get_cuda_dir(),
})
PLUGIN_TEMPLATE = Template(
(END)
Besides, I have installed protobuf-3.9.0, and libtorch(libtorch-shared-with-deps-latest.zip
) which is below:
The full compilation and install log is below without any complain:
(pytorch_py36) > python setup.py install --plugins
[4/4] g++ -shared -o torch2trt/libtorch2trt.so interpolate.pb.o interpolate.o -L/home/xxx/bin/lib... -lc10_cuda -ltorch -lcudart -lcaffe2 -lcaffe2_gpu -lprotobuf -lprotobuf-lite -pthread -lpthread -std=c++11
running install
running bdist_egg
running egg_info
creating torch2trt.egg-info
writing torch2trt.egg-info/PKG-INFO
writing dependency_links to torch2trt.egg-info/dependency_links.txt
writing top-level names to torch2trt.egg-info/top_level.txt
writing manifest file 'torch2trt.egg-info/SOURCES.txt'
reading manifest file 'torch2trt.egg-info/SOURCES.txt'
writing manifest file 'torch2trt.egg-info/SOURCES.txt'
installing library code to build/bdist.linux-x86_64/egg
running install_lib
running build_py
creating build
creating build/lib
creating build/lib/torch2trt
copying torch2trt/module_test.py -> build/lib/torch2trt
copying torch2trt/__init__.py -> build/lib/torch2trt
copying torch2trt/torch2trt.py -> build/lib/torch2trt
copying torch2trt/test.py -> build/lib/torch2trt
creating build/lib/torch2trt/converters
copying torch2trt/converters/relu.py -> build/lib/torch2trt/converters
copying torch2trt/converters/transpose.py -> build/lib/torch2trt/converters
copying torch2trt/converters/__init__.py -> build/lib/torch2trt/converters
copying torch2trt/converters/adaptive_avg_pool2d.py -> build/lib/torch2trt/converters
copying torch2trt/converters/cat.py -> build/lib/torch2trt/converters
copying torch2trt/converters/AdaptiveAvgPool2d.py -> build/lib/torch2trt/converters
copying torch2trt/converters/ReLU.py -> build/lib/torch2trt/converters
copying torch2trt/converters/identity.py -> build/lib/torch2trt/converters
copying torch2trt/converters/ConvTranspose2d.py -> build/lib/torch2trt/converters
copying torch2trt/converters/Identity.py -> build/lib/torch2trt/converters
copying torch2trt/converters/LogSoftmax.py -> build/lib/torch2trt/converters
copying torch2trt/converters/MaxPool2d.py -> build/lib/torch2trt/converters
copying torch2trt/converters/add.py -> build/lib/torch2trt/converters
copying torch2trt/converters/AvgPool2d.py -> build/lib/torch2trt/converters
copying torch2trt/converters/view.py -> build/lib/torch2trt/converters
copying torch2trt/converters/Linear.py -> build/lib/torch2trt/converters
copying torch2trt/converters/Conv2d.py -> build/lib/torch2trt/converters
copying torch2trt/converters/ReLU6.py -> build/lib/torch2trt/converters
copying torch2trt/converters/iadd.py -> build/lib/torch2trt/converters
copying torch2trt/converters/mean.py -> build/lib/torch2trt/converters
copying torch2trt/converters/relu6.py -> build/lib/torch2trt/converters
copying torch2trt/converters/BatchNorm2d.py -> build/lib/torch2trt/converters
creating build/lib/torch2trt/converters/interpolate
copying torch2trt/converters/interpolate/__init__.py -> build/lib/torch2trt/converters/interpolate
copying torch2trt/converters/interpolate/interpolate.py -> build/lib/torch2trt/converters/interpolate
copying torch2trt/converters/interpolate/interpolate_pb2.py -> build/lib/torch2trt/converters/interpolate
copying torch2trt/libtorch2trt.so -> build/lib/torch2trt
creating build/bdist.linux-x86_64
creating build/bdist.linux-x86_64/egg
creating build/bdist.linux-x86_64/egg/torch2trt
copying build/lib/torch2trt/module_test.py -> build/bdist.linux-x86_64/egg/torch2trt
copying build/lib/torch2trt/__init__.py -> build/bdist.linux-x86_64/egg/torch2trt
copying build/lib/torch2trt/torch2trt.py -> build/bdist.linux-x86_64/egg/torch2trt
creating build/bdist.linux-x86_64/egg/torch2trt/converters
copying build/lib/torch2trt/converters/relu.py -> build/bdist.linux-x86_64/egg/torch2trt/converters
copying build/lib/torch2trt/converters/transpose.py -> build/bdist.linux-x86_64/egg/torch2trt/converters
copying build/lib/torch2trt/converters/__init__.py -> build/bdist.linux-x86_64/egg/torch2trt/converters
copying build/lib/torch2trt/converters/adaptive_avg_pool2d.py -> build/bdist.linux-x86_64/egg/torch2trt/converters
copying build/lib/torch2trt/converters/cat.py -> build/bdist.linux-x86_64/egg/torch2trt/converters
copying build/lib/torch2trt/converters/AdaptiveAvgPool2d.py -> build/bdist.linux-x86_64/egg/torch2trt/converters
copying build/lib/torch2trt/converters/ReLU.py -> build/bdist.linux-x86_64/egg/torch2trt/converters
copying build/lib/torch2trt/converters/identity.py -> build/bdist.linux-x86_64/egg/torch2trt/converters
copying build/lib/torch2trt/converters/ConvTranspose2d.py -> build/bdist.linux-x86_64/egg/torch2trt/converters
copying build/lib/torch2trt/converters/Identity.py -> build/bdist.linux-x86_64/egg/torch2trt/converters
copying build/lib/torch2trt/converters/LogSoftmax.py -> build/bdist.linux-x86_64/egg/torch2trt/converters
copying build/lib/torch2trt/converters/MaxPool2d.py -> build/bdist.linux-x86_64/egg/torch2trt/converters
creating build/bdist.linux-x86_64/egg/torch2trt/converters/interpolate
copying build/lib/torch2trt/converters/interpolate/__init__.py -> build/bdist.linux-x86_64/egg/torch2trt/converters/interpolate
copying build/lib/torch2trt/converters/interpolate/interpolate.py -> build/bdist.linux-x86_64/egg/torch2trt/converters/interpolate
copying build/lib/torch2trt/converters/interpolate/interpolate_pb2.py -> build/bdist.linux-x86_64/egg/torch2trt/converters/interpolate
copying build/lib/torch2trt/converters/add.py -> build/bdist.linux-x86_64/egg/torch2trt/converters
copying build/lib/torch2trt/converters/AvgPool2d.py -> build/bdist.linux-x86_64/egg/torch2trt/converters
copying build/lib/torch2trt/converters/view.py -> build/bdist.linux-x86_64/egg/torch2trt/converters
copying build/lib/torch2trt/converters/Linear.py -> build/bdist.linux-x86_64/egg/torch2trt/converters
copying build/lib/torch2trt/converters/Conv2d.py -> build/bdist.linux-x86_64/egg/torch2trt/converters
copying build/lib/torch2trt/converters/ReLU6.py -> build/bdist.linux-x86_64/egg/torch2trt/converters
copying build/lib/torch2trt/converters/iadd.py -> build/bdist.linux-x86_64/egg/torch2trt/converters
copying build/lib/torch2trt/converters/mean.py -> build/bdist.linux-x86_64/egg/torch2trt/converters
copying build/lib/torch2trt/converters/relu6.py -> build/bdist.linux-x86_64/egg/torch2trt/converters
copying build/lib/torch2trt/converters/BatchNorm2d.py -> build/bdist.linux-x86_64/egg/torch2trt/converters
copying build/lib/torch2trt/libtorch2trt.so -> build/bdist.linux-x86_64/egg/torch2trt
copying build/lib/torch2trt/test.py -> build/bdist.linux-x86_64/egg/torch2trt
byte-compiling build/bdist.linux-x86_64/egg/torch2trt/module_test.py to module_test.cpython-36.pyc
byte-compiling build/bdist.linux-x86_64/egg/torch2trt/__init__.py to __init__.cpython-36.pyc
byte-compiling build/bdist.linux-x86_64/egg/torch2trt/torch2trt.py to torch2trt.cpython-36.pyc
byte-compiling build/bdist.linux-x86_64/egg/torch2trt/converters/relu.py to relu.cpython-36.pyc
byte-compiling build/bdist.linux-x86_64/egg/torch2trt/converters/transpose.py to transpose.cpython-36.pyc
byte-compiling build/bdist.linux-x86_64/egg/torch2trt/converters/__init__.py to __init__.cpython-36.pyc
byte-compiling build/bdist.linux-x86_64/egg/torch2trt/converters/adaptive_avg_pool2d.py to adaptive_avg_pool2d.cpython-36.pyc
byte-compiling build/bdist.linux-x86_64/egg/torch2trt/converters/cat.py to cat.cpython-36.pyc
byte-compiling build/bdist.linux-x86_64/egg/torch2trt/converters/AdaptiveAvgPool2d.py to AdaptiveAvgPool2d.cpython-36.pyc
byte-compiling build/bdist.linux-x86_64/egg/torch2trt/converters/ReLU.py to ReLU.cpython-36.pyc
byte-compiling build/bdist.linux-x86_64/egg/torch2trt/converters/identity.py to identity.cpython-36.pyc
byte-compiling build/bdist.linux-x86_64/egg/torch2trt/converters/ConvTranspose2d.py to ConvTranspose2d.cpython-36.pyc
byte-compiling build/bdist.linux-x86_64/egg/torch2trt/converters/Identity.py to Identity.cpython-36.pyc
byte-compiling build/bdist.linux-x86_64/egg/torch2trt/converters/LogSoftmax.py to LogSoftmax.cpython-36.pyc
byte-compiling build/bdist.linux-x86_64/egg/torch2trt/converters/MaxPool2d.py to MaxPool2d.cpython-36.pyc
byte-compiling build/bdist.linux-x86_64/egg/torch2trt/converters/interpolate/__init__.py to __init__.cpython-36.pyc
byte-compiling build/bdist.linux-x86_64/egg/torch2trt/converters/interpolate/interpolate.py to interpolate.cpython-36.pyc
byte-compiling build/bdist.linux-x86_64/egg/torch2trt/converters/interpolate/interpolate_pb2.py to interpolate_pb2.cpython-36.pyc
byte-compiling build/bdist.linux-x86_64/egg/torch2trt/converters/add.py to add.cpython-36.pyc
byte-compiling build/bdist.linux-x86_64/egg/torch2trt/converters/AvgPool2d.py to AvgPool2d.cpython-36.pyc
byte-compiling build/bdist.linux-x86_64/egg/torch2trt/converters/view.py to view.cpython-36.pyc
byte-compiling build/bdist.linux-x86_64/egg/torch2trt/converters/Linear.py to Linear.cpython-36.pyc
byte-compiling build/bdist.linux-x86_64/egg/torch2trt/converters/Conv2d.py to Conv2d.cpython-36.pyc
byte-compiling build/bdist.linux-x86_64/egg/torch2trt/converters/ReLU6.py to ReLU6.cpython-36.pyc
byte-compiling build/bdist.linux-x86_64/egg/torch2trt/converters/iadd.py to iadd.cpython-36.pyc
byte-compiling build/bdist.linux-x86_64/egg/torch2trt/converters/mean.py to mean.cpython-36.pyc
byte-compiling build/bdist.linux-x86_64/egg/torch2trt/converters/relu6.py to relu6.cpython-36.pyc
byte-compiling build/bdist.linux-x86_64/egg/torch2trt/converters/BatchNorm2d.py to BatchNorm2d.cpython-36.pyc
byte-compiling build/bdist.linux-x86_64/egg/torch2trt/test.py to test.cpython-36.pyc
creating build/bdist.linux-x86_64/egg/EGG-INFO
copying torch2trt.egg-info/PKG-INFO -> build/bdist.linux-x86_64/egg/EGG-INFO
copying torch2trt.egg-info/SOURCES.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying torch2trt.egg-info/dependency_links.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying torch2trt.egg-info/top_level.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
writing build/bdist.linux-x86_64/egg/EGG-INFO/native_libs.txt
zip_safe flag not set; analyzing archive contents...
torch2trt.__pycache__.__init__.cpython-36: module references __file__
creating dist
creating 'dist/torch2trt-0.0.0-py3.6.egg' and adding 'build/bdist.linux-x86_64/egg' to it
removing 'build/bdist.linux-x86_64/egg' (and everything under it)
Processing torch2trt-0.0.0-py3.6.egg
removing '/home/xxx/anaconda2/envs/pytorch_py36/lib/python3.6/site-packages/torch2trt-0.0.0-py3.6.egg' (and everything under it)
creating /home/xxx/anaconda2/envs/pytorch_py36/lib/python3.6/site-packages/torch2trt-0.0.0-py3.6.egg
Extracting torch2trt-0.0.0-py3.6.egg to /home/xxx/anaconda2/envs/pytorch_py36/lib/python3.6/site-packages
torch2trt 0.0.0 is already the active version in easy-install.pth
Installed /home/xxx/anaconda2/envs/pytorch_py36/lib/python3.6/site-packages/torch2trt-0.0.0-py3.6.egg
Processing dependencies for torch2trt==0.0.0
Finished processing dependencies for torch2trt==0.0.0
and after installation, the directory would be:
There is no wrong when running the alexnet demo:
import torch
from torch2trt import torch2trt
from torchvision.models.alexnet import alexnet
# create some regular pytorch model...
model = alexnet(pretrained=True).eval().cuda()
# create example data
x = torch.ones((1, 3, 224, 224)).cuda()
# convert to TensorRT feeding sample data as input
model_trt = torch2trt(model, [x])
y = model(x)
y_trt = model_trt(x)
# check the output against PyTorch
print(torch.max(torch.abs(y - y_trt)))
it gives
As for whether I have installed torch2trt without plugins before, I think I might installed the torch2trt without plugins at a very first try, but I had uninstalled it before installing the with-plugins one. Does it matters?
But when I run my resnet50-based segmentation model, it gives
/home/xxx/anaconda2/envs/pytorch_py36/lib/python3.6/site-packages/torch/nn/functional.py:2539: UserWarning: Default upsampling behavior when mode=bilinear is changed to align_corners=False since 0.4.0. Please specify align_corners=True if the old behavior is desired. See the documentation of nn.Upsample for details.
"See the documentation of nn.Upsample for details.".format(mode))
Traceback (most recent call last):
File "convert_trt.py", line 29, in <module>
model_trt = torch2trt(model, [x])
File "/home/xxx/anaconda2/envs/pytorch_py36/lib/python3.6/site-packages/torch2trt-0.0.0-py3.6.egg/torch2trt/torch2trt.py", line 252, in torch2trt
outputs = module(*inputs)
File "/home/xxx/anaconda2/envs/pytorch_py36/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in __call__
result = self.forward(*input, **kwargs)
File "/home/xxx/Deploy/ocr_rec_internal/server/python/models/fpn_resnet.py", line 239, in forward
p4 = self._upsample_add(p5, c4)
File "/home/xxx/Deploy/ocr_rec_internal/server/python/models/fpn_resnet.py", line 214, in _upsample_add
return F.interpolate(x, size=(H, W), mode='bilinear') + y
File "/home/xxx/anaconda2/envs/pytorch_py36/lib/python3.6/site-packages/torch2trt-0.0.0-py3.6.egg/torch2trt/torch2trt.py", line 97, in wrapper
converter(ctx)
File "/home/xxx/anaconda2/envs/pytorch_py36/lib/python3.6/site-packages/torch2trt-0.0.0-py3.6.egg/torch2trt/converters/add.py", line 10, in convert_add
layer = ctx.network.add_elementwise(input_a._trt, input_b._trt, trt.ElementWiseOperation.SUM)
AttributeError: 'Tensor' object has no attribute '_trt'
One more word here, the model was trained using F.Upsample
instead of F.interpolate
, I substitute it with interpolate
when deploying.
Below is the network definition:
def conv3x3(in_planes, out_planes, stride=1):
"""3x3 convolution with padding"""
return nn.Conv2d(in_planes, out_planes, kernel_size=3, stride=stride,
padding=1, bias=False)
class convolution(nn.Module):
def __init__(self, k, inp_dim, out_dim, stride=1, with_bn=True):
super(convolution, self).__init__()
pad = (k - 1) // 2
self.conv = nn.Conv2d(inp_dim, out_dim, (k, k), padding=(pad, pad), stride=(stride, stride), bias=not with_bn)
self.bn = nn.BatchNorm2d(out_dim) if with_bn else nn.Sequential()
self.relu = nn.ReLU(inplace=True)
def forward(self, x):
conv = self.conv(x)
bn = self.bn(conv)
relu = self.relu(bn)
return relu
def make_cnv_layer(inp_dim, out_dim):
return convolution(3, inp_dim, out_dim)
def make_kp_layer(cnv_dim, curr_dim, out_dim):
return nn.Sequential(
convolution(3, cnv_dim, curr_dim, with_bn=False),
nn.Conv2d(curr_dim, out_dim, (1, 1))
)
class BasicBlock(nn.Module):
expansion = 1
def __init__(self, inplanes, planes, stride=1, downsample=None):
super(BasicBlock, self).__init__()
self.conv1 = conv3x3(inplanes, planes, stride)
self.bn1 = nn.BatchNorm2d(planes)
self.relu = nn.ReLU(inplace=True)
self.conv2 = conv3x3(planes, planes)
self.bn2 = nn.BatchNorm2d(planes)
self.downsample = downsample
self.stride = stride
def forward(self, x):
residual = x
out = self.conv1(x)
out = self.bn1(out)
out = self.relu(out)
out = self.conv2(out)
out = self.bn2(out)
if self.downsample is not None:
residual = self.downsample(x)
out += residual
out = self.relu(out)
return out
class Bottleneck(nn.Module):
expansion = 4
def __init__(self, inplanes, planes, stride=1, downsample=None):
super(Bottleneck, self).__init__()
self.conv1 = nn.Conv2d(inplanes, planes, kernel_size=1, bias=False)
self.bn1 = nn.BatchNorm2d(planes)
self.conv2 = nn.Conv2d(planes, planes, kernel_size=3, stride=stride,
padding=1, bias=False)
self.bn2 = nn.BatchNorm2d(planes)
self.conv3 = nn.Conv2d(planes, planes * 4, kernel_size=1, bias=False)
self.bn3 = nn.BatchNorm2d(planes * 4)
self.relu = nn.ReLU(inplace=True)
self.downsample = downsample
self.stride = stride
def forward(self, x):
residual = x
out = self.conv1(x)
out = self.bn1(out)
out = self.relu(out)
out = self.conv2(out)
out = self.bn2(out)
out = self.relu(out)
out = self.conv3(out)
out = self.bn3(out)
if self.downsample is not None:
residual = self.downsample(x)
out += residual
out = self.relu(out)
return out
class ResNet(nn.Module):
def __init__(self, block, layers, scale=1):
self.inplanes = 48
super(ResNet, self).__init__()
self.conv1 = nn.Conv2d(3, 48, kernel_size=7, stride=2, padding=3,
bias=False)
self.bn1 = nn.BatchNorm2d(48)
self.relu1 = nn.ReLU(inplace=True)
self.maxpool = nn.MaxPool2d(kernel_size=3, stride=2, padding=1)
self.layer1 = self._make_layer(block, 48, layers[0])
self.layer2 = self._make_layer(block, 64, layers[1], stride=2)
self.layer3 = self._make_layer(block, 128, layers[2], stride=2)
self.layer4 = self._make_layer(block, 256, layers[3], stride=2)
# self.avgpool = nn.AvgPool2d(7, stride=1)
# self.fc = nn.Linear(512 * block.expansion, num_classes)
# Top layer
self.toplayer = nn.Conv2d(1024, 128, kernel_size=1, stride=1, padding=0) # Reduce channels
self.toplayer_bn = nn.BatchNorm2d(128)
self.toplayer_relu = nn.ReLU(inplace=True)
# Smooth layers
self.smooth1 = nn.Conv2d(128, 128, kernel_size=3, stride=1, padding=1)
self.smooth1_bn = nn.BatchNorm2d(128)
self.smooth1_relu = nn.ReLU(inplace=True)
self.smooth2 = nn.Conv2d(128, 128, kernel_size=3, stride=1, padding=1)
self.smooth2_bn = nn.BatchNorm2d(128)
self.smooth2_relu = nn.ReLU(inplace=True)
self.smooth3 = nn.Conv2d(128, 128, kernel_size=3, stride=1, padding=1)
self.smooth3_bn = nn.BatchNorm2d(128)
self.smooth3_relu = nn.ReLU(inplace=True)
# Lateral layers
self.latlayer1 = nn.Conv2d(512, 128, kernel_size=1, stride=1, padding=0)
self.latlayer1_bn = nn.BatchNorm2d(128)
self.latlayer1_relu = nn.ReLU(inplace=True)
self.latlayer2 = nn.Conv2d(256, 128, kernel_size=1, stride=1, padding=0)
self.latlayer2_bn = nn.BatchNorm2d(128)
self.latlayer2_relu = nn.ReLU(inplace=True)
##layer1*blockexpansion
self.latlayer3 = nn.Conv2d(48*4, 128, kernel_size=1, stride=1, padding=0)
self.latlayer3_bn = nn.BatchNorm2d(128)
self.latlayer3_relu = nn.ReLU(inplace=True)
self.conv2 = nn.Conv2d(512, 128, kernel_size=3, stride=1, padding=1)
self.bn2 = nn.BatchNorm2d(128)
self.relu2 = nn.ReLU(inplace=True)
self.conv_label = make_cnv_layer(128, 64)
self.conv_center = make_cnv_layer(128, 64)
self.conv_up = make_cnv_layer(128, 64)
self.conv_down = make_cnv_layer(128, 64)
self.conv_label_last = make_kp_layer(64, 16, 3)
self.heat_center = make_kp_layer(64, 16, 1)
self.heat_up = make_kp_layer(64, 16, 1)
self.heat_down = make_kp_layer(64, 16, 1)
self.scale = scale
for m in self.modules():
if isinstance(m, nn.Conv2d):
n = m.kernel_size[0] * m.kernel_size[1] * m.out_channels
m.weight.data.normal_(0, math.sqrt(2. / n))
elif isinstance(m, nn.BatchNorm2d):
m.weight.data.fill_(1)
m.bias.data.zero_()
def _make_layer(self, block, planes, blocks, stride=1):
downsample = None
if stride != 1 or self.inplanes != planes * block.expansion:
downsample = nn.Sequential(
nn.Conv2d(self.inplanes, planes * block.expansion,
kernel_size=1, stride=stride, bias=False),
nn.BatchNorm2d(planes * block.expansion),
)
layers = []
layers.append(block(self.inplanes, planes, stride, downsample))
self.inplanes = planes * block.expansion
for i in range(1, blocks):
layers.append(block(self.inplanes, planes))
return nn.Sequential(*layers)
def _upsample(self, x, y, scale=1):
_, _, H, W = y.size()
# return F.upsample(x, size=(H // scale, W // scale), mode='bilinear')
# print('interpolate')
return F.interpolate(x, size=(H // scale, W // scale), mode='bilinear')
def _upsample_add(self, x, y):
_, _, H, W = y.size()
# return F.upsample(x, size=(H, W), mode='bilinear') + y
# print('interpolate add')
return F.interpolate(x, size=(H, W), mode='bilinear') + y
def forward(self, x):
h = x
h = self.conv1(h)
h = self.bn1(h)
h = self.relu1(h)
h = self.maxpool(h)
h = self.layer1(h)
c2 = h
h = self.layer2(h)
c3 = h
h = self.layer3(h)
c4 = h
h = self.layer4(h)
c5 = h
# Top-down
p5 = self.toplayer(c5)
p5 = self.toplayer_relu(self.toplayer_bn(p5))
c4 = self.latlayer1(c4)
c4 = self.latlayer1_relu(self.latlayer1_bn(c4))
p4 = self._upsample_add(p5, c4)
p4 = self.smooth1(p4)
p4 = self.smooth1_relu(self.smooth1_bn(p4))
c3 = self.latlayer2(c3)
c3 = self.latlayer2_relu(self.latlayer2_bn(c3))
p3 = self._upsample_add(p4, c3)
p3 = self.smooth2(p3)
p3 = self.smooth2_relu(self.smooth2_bn(p3))
c2 = self.latlayer3(c2)
c2 = self.latlayer3_relu(self.latlayer3_bn(c2))
p2 = self._upsample_add(p3, c2)
p2 = self.smooth3(p2)
p2 = self.smooth3_relu(self.smooth3_bn(p2))
p3 = self._upsample(p3, p2)
p4 = self._upsample(p4, p2)
p5 = self._upsample(p5, p2)
out = torch.cat((p2, p3, p4, p5), 1)
out = self.conv2(out)
out = self.relu2(self.bn2(out))
out_label = self.conv_label(out)
out_label = self.conv_label_last(out_label)
out_label = self._upsample(out_label, x, scale=self.scale)
out_label = F.softmax(out_label, dim=1)
out_center = self.conv_center(out)
out_center = self.heat_center(out_center)
out_center = self._upsample(out_center, x, scale=self.scale)
out_up = self.conv_up(out)
out_up = self.heat_up(out_up)
out_up = self._upsample(out_up, x, scale=self.scale)
out_down = self.conv_down(out)
out_down = self.heat_down(out_down)
out_down = self._upsample(out_down, x, scale=self.scale)
# pdb.set_trace()
# return torch.cat([out_label, out_center, out_up, out_down], 1)
return out_label, out_center, out_up, out_down
def resnet18(**kwargs):
model = ResNet(BasicBlock, [2, 2, 2, 2], **kwargs)
return model
def resnet34(**kwargs):
model = ResNet(BasicBlock, [3, 4, 6, 3], **kwargs)
return model
# I used this
def resnet50(**kwargs):
model = ResNet(Bottleneck, [3, 4, 6, 3], **kwargs)
return model
For a full example, the image segmentation notebook converts a model that contains interpolate layers.
https://github.com/NVIDIA-AI-IOT/torch2trt/blob/master/notebooks/image_segmentation/conversion.ipynb
Thanks, I will dive into it.
However, in interpolate.py we also register several module tests that would probably be the easiest thing to try first. You can execute them by calling
python3 -m torch2trt.test --name=interpolate
This will print information like the throughput, latency, as well as max absolute error (compared to PyTorch).
It give nothing here:
Please let me know if you have any other questions!
Best, John
I really want to use this awesome repository, thanks for your helping!!! If you have something unclear, please contact me, thanks!
Best, sonack
Do you have any idea? Thanks! @jaybdub
when I test the 'interpolate' plugin and run notebooks/image_segmentation/conversion.ipynb ,I got this error
my tensorrt version: TensorRT-5.1.5.0.Ubuntu-16.04.5.x86_64-gnu.cuda-9.0.cudnn7.5
python3.5
When I run :
python3 -m torch2trt.test --name=interpolate
I got nothing info.
@jaybdub
Hi All,
@sonack @binzh93 Sorry for the delay on getting back. It is unusual that the build would pass, yet the interpolate test cases would not run. If interpolate.py is imported, we should at least see the test cases fail. Could you try removing the try/except statement around where interpolate is included and reinstalling
https://github.com/NVIDIA-AI-IOT/torch2trt/blob/master/torch2trt/converters/__init__.py#L31
To see if /what error is thrown? This should probably be made more verbose.
@binzh93 Could you share which version of torchvision you're using? The repository is under active development, it's possible they've made a breaking change since I tested. Ie. view
was recently replaced by flatten
in many of the models, which required a small update to torch2trt to parse.
Thanks for your patience all.
Best, John
Hi All,
@sonack @binzh93 Sorry for the delay on getting back. It is unusual that the build would pass, yet the interpolate test cases would not run. If interpolate.py is imported, we should at least see the test cases fail. Could you try removing the try/except statement around where interpolate is included and reinstalling
https://github.com/NVIDIA-AI-IOT/torch2trt/blob/master/torch2trt/converters/__init__.py#L31
To see if /what error is thrown? This should probably be made more verbose.
@binzh93 Could you share which version of torchvision you're using? The repository is under active development, it's possible they've made a breaking change since I tested. Ie.
view
was recently replaced byflatten
in many of the models, which required a small update to torch2trt to parse.Thanks for your patience all.
Best, John
OK! Thanks very much! I have solved the old issue, but new problem appeared.
I pulled the latest repository, and commented the try .... except
statement of interpolate importing, and recompiled it, but nothing happened:
Compilation Log:
(pytorch_py36) > python setup.py install --plugins
[4/4] g++ -shared -o torch2trt/libtorch2trt.so i...uf -lprotobuf-lite -pthread -lpthread -std=c++11
running install
running bdist_egg
running egg_info
creating torch2trt.egg-info
writing torch2trt.egg-info/PKG-INFO
writing dependency_links to torch2trt.egg-info/dependency_links.txt
writing top-level names to torch2trt.egg-info/top_level.txt
writing manifest file 'torch2trt.egg-info/SOURCES.txt'
reading manifest file 'torch2trt.egg-info/SOURCES.txt'
writing manifest file 'torch2trt.egg-info/SOURCES.txt'
installing library code to build/bdist.linux-x86_64/egg
running install_lib
running build_py
creating build
creating build/lib
creating build/lib/torch2trt
copying torch2trt/module_test.py -> build/lib/torch2trt
copying torch2trt/__init__.py -> build/lib/torch2trt
copying torch2trt/torch2trt.py -> build/lib/torch2trt
copying torch2trt/test.py -> build/lib/torch2trt
creating build/lib/torch2trt/converters
copying torch2trt/converters/relu.py -> build/lib/torch2trt/converters
copying torch2trt/converters/transpose.py -> build/lib/torch2trt/converters
copying torch2trt/converters/__init__.py -> build/lib/torch2trt/converters
copying torch2trt/converters/adaptive_avg_pool2d.py -> build/lib/torch2trt/converters
copying torch2trt/converters/pad.py -> build/lib/torch2trt/converters
copying torch2trt/converters/cat.py -> build/lib/torch2trt/converters
copying torch2trt/converters/AdaptiveAvgPool2d.py -> build/lib/torch2trt/converters
copying torch2trt/converters/softmax.py -> build/lib/torch2trt/converters
copying torch2trt/converters/ReLU.py -> build/lib/torch2trt/converters
copying torch2trt/converters/identity.py -> build/lib/torch2trt/converters
copying torch2trt/converters/ConvTranspose2d.py -> build/lib/torch2trt/converters
copying torch2trt/converters/Identity.py -> build/lib/torch2trt/converters
copying torch2trt/converters/BatchNorm1d.py -> build/lib/torch2trt/converters
copying torch2trt/converters/sigmoid.py -> build/lib/torch2trt/converters
copying torch2trt/converters/LogSoftmax.py -> build/lib/torch2trt/converters
copying torch2trt/converters/MaxPool2d.py -> build/lib/torch2trt/converters
copying torch2trt/converters/tanh.py -> build/lib/torch2trt/converters
copying torch2trt/converters/add.py -> build/lib/torch2trt/converters
copying torch2trt/converters/AvgPool2d.py -> build/lib/torch2trt/converters
copying torch2trt/converters/div.py -> build/lib/torch2trt/converters
copying torch2trt/converters/view.py -> build/lib/torch2trt/converters
copying torch2trt/converters/Linear.py -> build/lib/torch2trt/converters
copying torch2trt/converters/Conv2d.py -> build/lib/torch2trt/converters
copying torch2trt/converters/mul.py -> build/lib/torch2trt/converters
copying torch2trt/converters/Conv1d.py -> build/lib/torch2trt/converters
copying torch2trt/converters/ReLU6.py -> build/lib/torch2trt/converters
copying torch2trt/converters/iadd.py -> build/lib/torch2trt/converters
copying torch2trt/converters/mean.py -> build/lib/torch2trt/converters
copying torch2trt/converters/relu6.py -> build/lib/torch2trt/converters
copying torch2trt/converters/BatchNorm2d.py -> build/lib/torch2trt/converters
creating build/lib/torch2trt/converters/interpolate
copying torch2trt/converters/interpolate/__init__.py -> build/lib/torch2trt/converters/interpolate
copying torch2trt/converters/interpolate/interpolate.py -> build/lib/torch2trt/converters/interpolate
copying torch2trt/converters/interpolate/interpolate_pb2.py -> build/lib/torch2trt/converters/interpolate
copying torch2trt/libtorch2trt.so -> build/lib/torch2trt
creating build/bdist.linux-x86_64
creating build/bdist.linux-x86_64/egg
creating build/bdist.linux-x86_64/egg/torch2trt
copying build/lib/torch2trt/module_test.py -> build/bdist.linux-x86_64/egg/torch2trt
copying build/lib/torch2trt/__init__.py -> build/bdist.linux-x86_64/egg/torch2trt
copying build/lib/torch2trt/torch2trt.py -> build/bdist.linux-x86_64/egg/torch2trt
creating build/bdist.linux-x86_64/egg/torch2trt/converters
copying build/lib/torch2trt/converters/relu.py -> build/bdist.linux-x86_64/egg/torch2trt/converters
copying build/lib/torch2trt/converters/transpose.py -> build/bdist.linux-x86_64/egg/torch2trt/converters
copying build/lib/torch2trt/converters/__init__.py -> build/bdist.linux-x86_64/egg/torch2trt/converters
copying build/lib/torch2trt/converters/adaptive_avg_pool2d.py -> build/bdist.linux-x86_64/egg/torch2trt/converters
copying build/lib/torch2trt/converters/pad.py -> build/bdist.linux-x86_64/egg/torch2trt/converters
copying build/lib/torch2trt/converters/cat.py -> build/bdist.linux-x86_64/egg/torch2trt/converters
copying build/lib/torch2trt/converters/AdaptiveAvgPool2d.py -> build/bdist.linux-x86_64/egg/torch2trt/converters
copying build/lib/torch2trt/converters/softmax.py -> build/bdist.linux-x86_64/egg/torch2trt/converters
copying build/lib/torch2trt/converters/ReLU.py -> build/bdist.linux-x86_64/egg/torch2trt/converters
copying build/lib/torch2trt/converters/identity.py -> build/bdist.linux-x86_64/egg/torch2trt/converters
copying build/lib/torch2trt/converters/ConvTranspose2d.py -> build/bdist.linux-x86_64/egg/torch2trt/converters
copying build/lib/torch2trt/converters/Identity.py -> build/bdist.linux-x86_64/egg/torch2trt/converters
copying build/lib/torch2trt/converters/BatchNorm1d.py -> build/bdist.linux-x86_64/egg/torch2trt/converters
copying build/lib/torch2trt/converters/sigmoid.py -> build/bdist.linux-x86_64/egg/torch2trt/converters
copying build/lib/torch2trt/converters/LogSoftmax.py -> build/bdist.linux-x86_64/egg/torch2trt/converters
copying build/lib/torch2trt/converters/MaxPool2d.py -> build/bdist.linux-x86_64/egg/torch2trt/converters
creating build/bdist.linux-x86_64/egg/torch2trt/converters/interpolate
copying build/lib/torch2trt/converters/interpolate/__init__.py -> build/bdist.linux-x86_64/egg/torch2trt/converters/interpolate
copying build/lib/torch2trt/converters/interpolate/interpolate.py -> build/bdist.linux-x86_64/egg/torch2trt/converters/interpolate
copying build/lib/torch2trt/converters/interpolate/interpolate_pb2.py -> build/bdist.linux-x86_64/egg/torch2trt/converters/interpolate
copying build/lib/torch2trt/converters/tanh.py -> build/bdist.linux-x86_64/egg/torch2trt/converters
copying build/lib/torch2trt/converters/add.py -> build/bdist.linux-x86_64/egg/torch2trt/converters
copying build/lib/torch2trt/converters/AvgPool2d.py -> build/bdist.linux-x86_64/egg/torch2trt/converters
copying build/lib/torch2trt/converters/div.py -> build/bdist.linux-x86_64/egg/torch2trt/converters
copying build/lib/torch2trt/converters/view.py -> build/bdist.linux-x86_64/egg/torch2trt/converters
copying build/lib/torch2trt/converters/Linear.py -> build/bdist.linux-x86_64/egg/torch2trt/converters
copying build/lib/torch2trt/converters/Conv2d.py -> build/bdist.linux-x86_64/egg/torch2trt/converters
copying build/lib/torch2trt/converters/mul.py -> build/bdist.linux-x86_64/egg/torch2trt/converters
copying build/lib/torch2trt/converters/Conv1d.py -> build/bdist.linux-x86_64/egg/torch2trt/converters
copying build/lib/torch2trt/converters/ReLU6.py -> build/bdist.linux-x86_64/egg/torch2trt/converters
copying build/lib/torch2trt/converters/iadd.py -> build/bdist.linux-x86_64/egg/torch2trt/converters
copying build/lib/torch2trt/converters/mean.py -> build/bdist.linux-x86_64/egg/torch2trt/converters
copying build/lib/torch2trt/converters/relu6.py -> build/bdist.linux-x86_64/egg/torch2trt/converters
copying build/lib/torch2trt/converters/BatchNorm2d.py -> build/bdist.linux-x86_64/egg/torch2trt/converters
copying build/lib/torch2trt/libtorch2trt.so -> build/bdist.linux-x86_64/egg/torch2trt
copying build/lib/torch2trt/test.py -> build/bdist.linux-x86_64/egg/torch2trt
byte-compiling build/bdist.linux-x86_64/egg/torch2trt/module_test.py to module_test.cpython-36.pyc
byte-compiling build/bdist.linux-x86_64/egg/torch2trt/__init__.py to __init__.cpython-36.pyc
byte-compiling build/bdist.linux-x86_64/egg/torch2trt/torch2trt.py to torch2trt.cpython-36.pyc
byte-compiling build/bdist.linux-x86_64/egg/torch2trt/converters/relu.py to relu.cpython-36.pyc
byte-compiling build/bdist.linux-x86_64/egg/torch2trt/converters/transpose.py to transpose.cpython-36.pyc
byte-compiling build/bdist.linux-x86_64/egg/torch2trt/converters/__init__.py to __init__.cpython-36.pyc
byte-compiling build/bdist.linux-x86_64/egg/torch2trt/converters/adaptive_avg_pool2d.py to adaptive_avg_pool2d.cpython-36.pyc
byte-compiling build/bdist.linux-x86_64/egg/torch2trt/converters/pad.py to pad.cpython-36.pyc
byte-compiling build/bdist.linux-x86_64/egg/torch2trt/converters/cat.py to cat.cpython-36.pyc
byte-compiling build/bdist.linux-x86_64/egg/torch2trt/converters/AdaptiveAvgPool2d.py to AdaptiveAvgPool2d.cpython-36.pyc
byte-compiling build/bdist.linux-x86_64/egg/torch2trt/converters/softmax.py to softmax.cpython-36.pyc
byte-compiling build/bdist.linux-x86_64/egg/torch2trt/converters/ReLU.py to ReLU.cpython-36.pyc
byte-compiling build/bdist.linux-x86_64/egg/torch2trt/converters/identity.py to identity.cpython-36.pyc
byte-compiling build/bdist.linux-x86_64/egg/torch2trt/converters/ConvTranspose2d.py to ConvTranspose2d.cpython-36.pyc
byte-compiling build/bdist.linux-x86_64/egg/torch2trt/converters/Identity.py to Identity.cpython-36.pyc
byte-compiling build/bdist.linux-x86_64/egg/torch2trt/converters/BatchNorm1d.py to BatchNorm1d.cpython-36.pyc
byte-compiling build/bdist.linux-x86_64/egg/torch2trt/converters/sigmoid.py to sigmoid.cpython-36.pyc
byte-compiling build/bdist.linux-x86_64/egg/torch2trt/converters/LogSoftmax.py to LogSoftmax.cpython-36.pyc
byte-compiling build/bdist.linux-x86_64/egg/torch2trt/converters/MaxPool2d.py to MaxPool2d.cpython-36.pyc
byte-compiling build/bdist.linux-x86_64/egg/torch2trt/converters/interpolate/__init__.py to __init__.cpython-36.pyc
byte-compiling build/bdist.linux-x86_64/egg/torch2trt/converters/interpolate/interpolate.py to interpolate.cpython-36.pyc
byte-compiling build/bdist.linux-x86_64/egg/torch2trt/converters/interpolate/interpolate_pb2.py to interpolate_pb2.cpython-36.pyc
byte-compiling build/bdist.linux-x86_64/egg/torch2trt/converters/tanh.py to tanh.cpython-36.pyc
byte-compiling build/bdist.linux-x86_64/egg/torch2trt/converters/add.py to add.cpython-36.pyc
byte-compiling build/bdist.linux-x86_64/egg/torch2trt/converters/AvgPool2d.py to AvgPool2d.cpython-36.pyc
byte-compiling build/bdist.linux-x86_64/egg/torch2trt/converters/div.py to div.cpython-36.pyc
byte-compiling build/bdist.linux-x86_64/egg/torch2trt/converters/view.py to view.cpython-36.pyc
byte-compiling build/bdist.linux-x86_64/egg/torch2trt/converters/Linear.py to Linear.cpython-36.pyc
byte-compiling build/bdist.linux-x86_64/egg/torch2trt/converters/Conv2d.py to Conv2d.cpython-36.pyc
byte-compiling build/bdist.linux-x86_64/egg/torch2trt/converters/mul.py to mul.cpython-36.pyc
byte-compiling build/bdist.linux-x86_64/egg/torch2trt/converters/Conv1d.py to Conv1d.cpython-36.pyc
byte-compiling build/bdist.linux-x86_64/egg/torch2trt/converters/ReLU6.py to ReLU6.cpython-36.pyc
byte-compiling build/bdist.linux-x86_64/egg/torch2trt/converters/iadd.py to iadd.cpython-36.pyc
byte-compiling build/bdist.linux-x86_64/egg/torch2trt/converters/mean.py to mean.cpython-36.pyc
byte-compiling build/bdist.linux-x86_64/egg/torch2trt/converters/relu6.py to relu6.cpython-36.pyc
byte-compiling build/bdist.linux-x86_64/egg/torch2trt/converters/BatchNorm2d.py to BatchNorm2d.cpython-36.pyc
byte-compiling build/bdist.linux-x86_64/egg/torch2trt/test.py to test.cpython-36.pyc
creating build/bdist.linux-x86_64/egg/EGG-INFO
copying torch2trt.egg-info/PKG-INFO -> build/bdist.linux-x86_64/egg/EGG-INFO
copying torch2trt.egg-info/SOURCES.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying torch2trt.egg-info/dependency_links.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying torch2trt.egg-info/top_level.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
writing build/bdist.linux-x86_64/egg/EGG-INFO/native_libs.txt
zip_safe flag not set; analyzing archive contents...
torch2trt.__pycache__.__init__.cpython-36: module references __file__
creating dist
creating 'dist/torch2trt-0.0.0-py3.6.egg' and adding 'build/bdist.linux-x86_64/egg' to it
removing 'build/bdist.linux-x86_64/egg' (and everything under it)
Processing torch2trt-0.0.0-py3.6.egg
creating /home/xxx/anaconda2/envs/pytorch_py36/lib/python3.6/site-packages/torch2trt-0.0.0-py3.6.egg
Extracting torch2trt-0.0.0-py3.6.egg to /home/xxx/anaconda2/envs/pytorch_py36/lib/python3.6/site-packages
Adding torch2trt 0.0.0 to easy-install.pth file
Installed /home/xxx/anaconda2/envs/pytorch_py36/lib/python3.6/site-packages/torch2trt-0.0.0-py3.6.egg
Processing dependencies for torch2trt==0.0.0
Finished processing dependencies for torch2trt==0.0.0
But when I import torch2trt
in python, it complains
(pytorch_py36) > python
Python 3.6.8 |Anaconda, Inc.| (default, Dec 30 2018, 01:22:34)
[GCC 7.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch2trt
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/home/xxx/bin/torch2trt_clean_3/torch2trt/__init__.py", line 2, in <module>
from .converters import *
File "/home/xxx/bin/torch2trt_clean_3/torch2trt/converters/__init__.py", line 32, in <module>
from .interpolate import *
File "/home/xxx/bin/torch2trt_clean_3/torch2trt/converters/interpolate/__init__.py", line 1, in <module>
from .interpolate import *
File "/home/xxx/bin/torch2trt_clean_3/torch2trt/converters/interpolate/interpolate.py", line 5, in <module>
from .interpolate_pb2 import interpolate_Message
File "/home/xxx/bin/torch2trt_clean_3/torch2trt/converters/interpolate/interpolate_pb2.py", line 7, in <module>
from google.protobuf import descriptor as _descriptor
ModuleNotFoundError: No module named 'google'
so I installed with pip install protobuf
, and recompile and reimport, it seems good:
However, when I test this installation on the (notebook)[https://github.com/NVIDIA-AI-IOT/torch2trt/blob/master/notebooks/image_segmentation/conversion.ipynb], it complains:
The full error log trace is
IndexError Traceback (most recent call last)
<ipython-input-9-7f1400d9157d> in <module>
----> 1 model_trt = torch2trt.torch2trt(model_w, [data], fp16_mode=True)
~/anaconda2/envs/pytorch_py36/lib/python3.6/site-packages/torch2trt-0.0.0-py3.6.egg/torch2trt/torch2trt.py in torch2trt(module, inputs, input_names, output_names, log_level, max_batch_size, fp16_mode, max_workspace_size, strict_type_constraints)
250 ctx.add_inputs(inputs, input_names)
251
--> 252 outputs = module(*inputs)
253
254 if not isinstance(outputs, tuple):
~/anaconda2/envs/pytorch_py36/lib/python3.6/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
491 result = self._slow_forward(*input, **kwargs)
492 else:
--> 493 result = self.forward(*input, **kwargs)
494 for hook in self._forward_hooks.values():
495 hook_result = hook(self, input, result)
<ipython-input-6-8d085fb1ce64> in forward(self, x)
4 self.model = model
5 def forward(self, x):
----> 6 return self.model(x)['out']
~/anaconda2/envs/pytorch_py36/lib/python3.6/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
491 result = self._slow_forward(*input, **kwargs)
492 else:
--> 493 result = self.forward(*input, **kwargs)
494 for hook in self._forward_hooks.values():
495 hook_result = hook(self, input, result)
~/anaconda2/envs/pytorch_py36/lib/python3.6/site-packages/torchvision/models/segmentation/_utils.py in forward(self, x)
20 result = OrderedDict()
21 x = features["out"]
---> 22 x = self.classifier(x)
23 x = F.interpolate(x, size=input_shape, mode='bilinear', align_corners=False)
24 result["out"] = x
~/anaconda2/envs/pytorch_py36/lib/python3.6/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
491 result = self._slow_forward(*input, **kwargs)
492 else:
--> 493 result = self.forward(*input, **kwargs)
494 for hook in self._forward_hooks.values():
495 hook_result = hook(self, input, result)
~/anaconda2/envs/pytorch_py36/lib/python3.6/site-packages/torch/nn/modules/container.py in forward(self, input)
90 def forward(self, input):
91 for module in self._modules.values():
---> 92 input = module(input)
93 return input
94
~/anaconda2/envs/pytorch_py36/lib/python3.6/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
491 result = self._slow_forward(*input, **kwargs)
492 else:
--> 493 result = self.forward(*input, **kwargs)
494 for hook in self._forward_hooks.values():
495 hook_result = hook(self, input, result)
~/anaconda2/envs/pytorch_py36/lib/python3.6/site-packages/torchvision/models/segmentation/deeplabv3.py in forward(self, x)
89 res = []
90 for conv in self.convs:
---> 91 res.append(conv(x))
92 res = torch.cat(res, dim=1)
93 return self.project(res)
~/anaconda2/envs/pytorch_py36/lib/python3.6/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
491 result = self._slow_forward(*input, **kwargs)
492 else:
--> 493 result = self.forward(*input, **kwargs)
494 for hook in self._forward_hooks.values():
495 hook_result = hook(self, input, result)
~/anaconda2/envs/pytorch_py36/lib/python3.6/site-packages/torchvision/models/segmentation/deeplabv3.py in forward(self, x)
59 size = x.shape[-2:]
60 x = super(ASPPPooling, self).forward(x)
---> 61 return F.interpolate(x, size=size, mode='bilinear', align_corners=False)
62
63
~/anaconda2/envs/pytorch_py36/lib/python3.6/site-packages/torch2trt-0.0.0-py3.6.egg/torch2trt/torch2trt.py in wrapper(*args, **kwargs)
95
96 #print('%s : %s' % (method.__qualname__, converter.__name__))
---> 97 converter(ctx)
98
99 # convert to None so conversion will fail for unsupported layers
~/anaconda2/envs/pytorch_py36/lib/python3.6/site-packages/torch2trt-0.0.0-py3.6.egg/torch2trt/converters/interpolate/interpolate.py in convert_interpolate(ctx)
32 size = list(output.shape[2:])
33
---> 34 plugin = get_interpolate_plugin(size=size, mode=mode, align_corners=align_corners)
35
36 layer = ctx.network.add_plugin_v2([input._trt], plugin)
~/anaconda2/envs/pytorch_py36/lib/python3.6/site-packages/torch2trt-0.0.0-py3.6.egg/torch2trt/converters/interpolate/interpolate.py in get_interpolate_plugin(size, mode, align_corners)
9 PLUGIN_NAME = 'interpolate'
10 registry = trt.get_plugin_registry()
---> 11 creator = [c for c in registry.plugin_creator_list if c.name == PLUGIN_NAME and c.plugin_namespace == 'torch2trt'][0]
12 message = interpolate_Message(size=size, mode=mode, align_corners=align_corners)
13 return creator.deserialize_plugin(PLUGIN_NAME, message.SerializeToString())
IndexError: list index out of range
The easiest way to reproduce this error is
How to solve it? Thank you very much!
@jaybdub FYI, this issue may be the same as #20
I forget to put my self-compiled protobuf before system's default protobuf library, which cause the error. Solved. Closed.
protobuf I met the same problem. Which protobuf do you use?
protobuf I met the same problem. Which protobuf do you use?
I compiled the latest version on github 3.9.0
@jaybdub FYI, this issue may be the same as #20
I sucessfully install interpolate plugin and import torch2trt with no bug, but when I use "python3 -m torch2trt.test --name=interpolate", "IndexError: list index out of range" error occured. Do you sovle it ?Change the protobuf version?
My protobuf version is 3.0.0 torch version 1.1.0 torchvision version 0.3 @sonack
@jaybdub FYI, this issue may be the same as #20
I sucessfully install interpolate plugin and import torch2trt with no bug, but when I use "python3 -m torch2trt.test --name=interpolate", "IndexError: list index out of range" error occured. Do you sovle it ?Change the protobuf version?
My protobuf version is 3.0.0 torch version 1.1.0 torchvision version 0.3 @sonack
My protobuf version is 3.9.0, one thing to note is to add -L${YOUR_PROTOBUF_LIB_PATH}
to rule link of build.py
, below is my build.py
and setup.py
, FYI.
build.py
import imp
import subprocess
import os
from string import Template
PLUGINS = [
'interpolate',
]
BASE_FOLDER = 'torch2trt/converters'
NINJA_TEMPLATE = Template((
"rule link\n"
" command = g++ -shared -o $$out $$in -L{{YOUR PROTOBUF INSTALL DIR}}/lib -L$torch_dir/lib -L$cuda_dir/lib64 -L$trt_lib_dir -lc10 -lc10_cuda -ltorch -lcudart -lcaffe2 -lcaffe2_gpu -lprotobuf -lprotobuf-lite -pthread -lpthread -lnvinfer -Xlinker --unresolved-symbols=ignore-in-shared-libs\n"
"rule protoc\n"
" command = protoc $$in --cpp_out=. --python_out=.\n"
"rule cxx\n"
" command = g++ -c -fPIC $$in -I$cuda_dir/include -I$torch_dir/include -I$torch_dir/include/torch/csrc/api/include -I. -std=c++11 -I$trt_inc_dir\n"
))
PLUGIN_TEMPLATE = Template((
"build $plugin_dir/$plugin.pb.h $plugin_dir/$plugin.pb.cc $plugin_dir/${plugin}_pb2.py: protoc $plugin_dir/$plugin.proto\n"
"build $plugin.pb.o: cxx $plugin_dir/$plugin.pb.cc\n"
"build $plugin.o: cxx $plugin_dir/$plugin.cpp\n"
))
def build(cuda_dir="/usr/local/cuda",
torch_dir=imp.find_module('torch')[1],
trt_inc_dir="/usr/include/aarch64-linux-gnu",
trt_lib_dir="/usr/lib/aarch64-linux-gnu"):
global PLUGINS, BASE_FOLDER, NINJA_TEMPLATE, PLUGIN_TEMPLATE
NINJA_STR = NINJA_TEMPLATE.substitute({
'torch_dir': torch_dir,
'cuda_dir': cuda_dir,
'trt_inc_dir': trt_inc_dir,
'trt_lib_dir': trt_lib_dir,
})
plugin_o_files = []
for plugin in PLUGINS:
NINJA_STR += \
PLUGIN_TEMPLATE.substitute({
'plugin': plugin,
'plugin_dir': os.path.join(BASE_FOLDER, plugin),
})
plugin_o_files += [plugin + '.pb.o', plugin + '.o']
NINJA_STR += Template((
"build torch2trt/libtorch2trt.so: link $o_files\n"
)).substitute({'o_files': ' '.join(plugin_o_files)})
with open('build.ninja', 'w') as f:
f.write(NINJA_STR)
subprocess.call(['ninja'])
if __name__ == '__main__':
build()
Note to replace {{}} with your own path!!!!
setup.py
import os
import glob
import shutil
from setuptools import setup, find_packages
from setuptools.command.install import install
from setuptools.command.develop import develop
from distutils.cmd import Command
from build import build
package_data = {}
plugins_user_options = [
('plugins', None, 'Build plugins'),
('cuda-dir=', None, 'Location of CUDA (if not default location)'),
('torch-dir=', None, 'Location of PyTorch (if not default location)'),
('trt-inc-dir=', None, 'Location of TensorRT include files (if not default location)'),
('trt-lib-dir=', None, 'Location of TensorRT libraries (if not default location)'),
]
def initialize_plugins_options(cmd_obj):
cmd_obj.plugins = False
cmd_obj.cuda_dir = None
cmd_obj.torch_dir = None
cmd_obj.trt_inc_dir = None
cmd_obj.trt_lib_dir = None
def run_plugins_compilation(cmd_obj):
if cmd_obj.plugins:
build_args = {}
if cmd_obj.cuda_dir:
build_args['cuda_dir'] = cmd_obj.cuda_dir
if cmd_obj.torch_dir:
build_args['torch_dir'] = cmd_obj.torch_dir
if cmd_obj.trt_inc_dir:
build_args['trt_inc_dir'] = cmd_obj.trt_inc_dir
if cmd_obj.trt_lib_dir:
build_args['trt_lib_dir'] = cmd_obj.trt_lib_dir
print('Building in plugin support')
build(**build_args)
package_data['torch2trt'] = ['libtorch2trt.so']
class DevelopCommand(develop):
description = "Builds the package and symlinks it into the PYTHONPATH"
user_options = develop.user_options + plugins_user_options
def initialize_options(self):
develop.initialize_options(self)
initialize_plugins_options(self)
def finalize_options(self):
develop.finalize_options(self)
def run(self):
run_plugins_compilation(self)
develop.run(self)
class InstallCommand(install):
description = "Builds the package"
user_options = install.user_options + plugins_user_options
def initialize_options(self):
install.initialize_options(self)
initialize_plugins_options(self)
def finalize_options(self):
install.finalize_options(self)
def run(self):
run_plugins_compilation(self)
install.run(self)
class CleanCommand(Command):
"""Custom clean command to tidy up the project root."""
PY_CLEAN_FILES = ['./build', './dist', './__pycache__', './*.pyc', './*.tgz', './*.egg-info']
description = "Command to tidy up the project root"
user_options = []
def initialize_options(self):
pass
def finalize_options(self):
pass
def run(self):
root_dir = os.path.dirname(os.path.realpath(__file__))
for path_spec in self.PY_CLEAN_FILES:
# Make paths absolute and relative to this path
abs_paths = glob.glob(os.path.normpath(os.path.join(root_dir, path_spec)))
for path in [str(p) for p in abs_paths]:
if not path.startswith(root_dir):
# Die if path in CLEAN_FILES is absolute + outside this directory
raise ValueError("%s is not a path inside %s" % (path, root_dir))
print('Removing %s' % os.path.relpath(path))
shutil.rmtree(path)
cmd_list = {
"Removing generated protobuf cc files": "find . -name '*.pb.cc' -print0 | xargs -0 rm -f;",
"Removing generated protobuf h files": "find . -name '*.pb.h' -print0 | xargs -0 rm -f;",
"Removing generated protobuf py files": "find . -name '*_pb2.py' -print0 | xargs -0 rm -f;",
"Removing generated ninja files": "find . -name '*.ninja*' -print0 | xargs -0 rm -f;",
"Removing generated o files": "find . -name '*.o' -print0 | xargs -0 rm -f;",
"Removing generated so files": "find . -name '*.so' -print0 | xargs -0 rm -f;",
}
for cmd, script in cmd_list.items():
print("{}".format(cmd))
os.system(script)
setup(
name='torch2trt',
version='0.0.0',
description='An easy to use PyTorch to TensorRT converter',
cmdclass={
'install': InstallCommand,
'clean': CleanCommand,
'develop': DevelopCommand,
},
packages=find_packages(),
package_data=package_data
)
#!/usr/bin/env bash
python setup.py install --plugins --cuda-dir=/usr/local/cuda-10.0 --trt-inc-dir=/data00/xxx/TensorRT/1404/TensorRT-5.1.5.0/include --trt-lib-dir=/data00/xxx/TensorRT/1404/TensorRT-5.1.5.0/lib
replace with your own path.
I have installed this repo with plugins, but when I issue![image](https://user-images.githubusercontent.com/12935189/61279780-59f5a480-a7e9-11e9-9244-d6363b4fe9de.png)
torch2trt.CONVERTERS.keys()
:there is no interpolate layer, and when I convert a model with interpolate, it complains 'AttributeError: 'Tensor' object has no attribute '_trt'' before 'interpolate' operation, so what's wrong is here?