VisualizeResults.py fails to run.

[travelbureau]

Jetson TX2, Jetpack 3.2, PyTorch 0.3.0, Python 2 (had to add from builtins import super to Model.py not sure if this is kosher)

Input image is 512x256, all default args.

RuntimeError: cuda runtime error (7): too many resources requested for launch at /home/nvidia/pytorch/torch/lib/THCUNN/im2col.h:60

I think the issue maybe that PyTorch was built without CUDNN support. I'm rebuilding and will report back.

[sacmehta]

We tested our code with Python 3 on TX2 and it works fine without any changes. I would suggest to use Python3 instead of Python 2. This may help to avoid adding built in import statements.

[travelbureau]

Update: CUDNN support fixed the problem. I can confirm that the code works correctly with ESPNet and Python 2, if you add the line from builtins import super to Model.py.

For anyone struggling with CUDNN issues on build see this gist: https://github.com/andrewadare/jetson-tx2-pytorch. Specifically, if and only if you installed CUDNN 7.0 from the .deb files in TensorRT3.0 rather than via Jetpack.

export CUDNN_LIB_DIR=/usr/lib/aarch64-linux-gnu
export CUDNN_INCLUDE_DIR=/usr/include

To test functionality try:

import torch
torch.backends.cudnn.is_acceptable(torch.cuda.FloatTensor(1))

Note: However, ESPNet-c produces another error: RuntimeError: cuda runtime error (7) : too many resources requested for launch at /home/nvidia/pytorch/torch/lib/THCUNN/generic/SpatialUpSamplingBilinear.cu:63

Can you confirm which version of PyTorch, CUDA, and CUDNN you are using (Update: CUDA 8.0 according to paper)... This is happening on a low res image and I find it hard to believe that Python 2 vs. 3 has anything to do with it.

@sacmehta thank you for your quick reply and for releasing your code.

[travelbureau]

In fact, in the paper I notice that there doesn't seem to be a bar for ESPNet-c in Fig 11. Does ESPNet-c actually run on the TX2? Perhaps I'm misunderstanding, is ESPNet-c just a variant that led to the final ESPNet, which is the preferred architecture regardless of GPU? Apologies for not reading the paper more closely yet, I'm just bench-marking options of a robotics application. Note that I am using CUDA 9.0 and CUDNN 7.0 whereas your paper utilizes CUDA 8.0. Respectfully, I don't think this issue should be closed quite yet.

[sacmehta]

To upsample the feature maps from ESPNet-C by a factor of 8, bilinear upsampling demands huge computation resources and that is why it is failing on TX2 as it is limited by amount of resources.

I would like to note that Bilinear upsampling is not a part of ESPNet-C/ESPNet structure. It was just used to satisfy the constraints of the Cityscapes server, which expects the segmentation masks to be of the same dimensions as input image. You can comment the bilinear up-sampling part and it should work fine on TX2.

[travelbureau]

@sacmehta Thank you!