search

CVCUDA / CV-CUDA

CV-CUDA™ is an open-source, GPU accelerated library for cloud-scale image processing and computer vision.
https://cvcuda.github.io
Other
2.37k stars 217 forks source link

[BUG] Failed to use cvcuda with torch dataloader #13

Closed scse-l closed 9 months ago

scse-l commented 1 year ago

Describe the bug When I try to use cvcuda with torch dataloader, I get an exception:

TypeError: cannot pickle 'nvcv.Tensor' object

Is there any solution to use cvcuda with torch dataloader ? Or will it be compatible with torch dataloader in future?

scse-l commented 1 year ago

I also tried to use torch.Tensor returned by torch.dataloader to generate cvcuda.Tensor. It raises exception: RuntimeError: NVCV_ERROR_INVALID_ARGUMENT: Pitch of dimension 3 must be == 4 (packed), but it is 1048576. You can use following code to reproduce:

import cvcuda
import torch
from torchvision.datasets import FakeData
import torch.utils.data
from torchvision import transforms

fd = FakeData(1000, (3, 512, 512), 10, transforms.ToTensor())
dl = torch.utils.data.DataLoader(fd)
idx, (imgs, labels) = next(enumerate(dl))
imgs = imgs.cuda()
imgs = imgs.permute([0, 2, 3, 1])
cvcuda.as_tensor(imgs, "NHWC")
milesp-nvidia commented 1 year ago

Hi @scse-l, if you can run the print_env.sh script and post the results here.

scse-l commented 1 year ago

Hi @scse-l, if you can run the print_env.sh script and post the results here.

Sure.

Click here to see environment details

     **git***
     commit 0ecbaa978ec5cbd9513aa581d19b02389a75d175 (HEAD -> release_v0.2.x, tag: v0.2.0-alpha, origin/release_v0.2.x, origin/main, origin/HEAD)
     Author: Miles Price 
     Date:   Wed Dec 21 16:30:28 2022 -0800

     feat: initializing GitHub repo with tested CV-CUDA operators.
     **git submodules***
     -5ab508a01f9eb089207ee87fd547d290da39d015 3rdparty/googletest
     -0694ec6a15863bff2e0ea5efe07c78de39b9a33c 3rdparty/pybind11

     ***OS Information***
     DISTRIB_ID=Ubuntu
     DISTRIB_RELEASE=20.04
     DISTRIB_CODENAME=focal
     DISTRIB_DESCRIPTION="Ubuntu 20.04.5 LTS"
     NAME="Ubuntu"
     VERSION="20.04.5 LTS (Focal Fossa)"
     ID=ubuntu
     ID_LIKE=debian
     PRETTY_NAME="Ubuntu 20.04.5 LTS"
     VERSION_ID="20.04"
     HOME_URL="https://www.ubuntu.com/"
     SUPPORT_URL="https://help.ubuntu.com/"
     BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
     PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
     VERSION_CODENAME=focal
     UBUNTU_CODENAME=focal
     Linux myhost 5.15.0-56-generic #62~20.04.1-Ubuntu SMP Tue Nov 22 21:24:20 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

     ***GPU Information***
     Thu Jan  5 15:39:24 2023
     +-----------------------------------------------------------------------------+
     | NVIDIA-SMI 525.60.13    Driver Version: 525.60.13    CUDA Version: 12.0     |
     |-------------------------------+----------------------+----------------------+
     | GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
     | Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
     |                               |                      |               MIG M. |
     |===============================+======================+======================|
     |   0  NVIDIA GeForce ...  On   | 00000000:01:00.0 Off |                  N/A |
     |  0%   42C    P8    17W / 350W |      1MiB / 24576MiB |      0%      Default |
     |                               |                      |                  N/A |
     +-------------------------------+----------------------+----------------------+

     +-----------------------------------------------------------------------------+
     | Processes:                                                                  |
     |  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
     |        ID   ID                                                   Usage      |
     |=============================================================================|
     |  No running processes found                                                 |
     +-----------------------------------------------------------------------------+

     ***CPU***
     架构：                           x86_64
     CPU 运行模式：                   32-bit, 64-bit
     字节序：                         Little Endian
     Address sizes:                   46 bits physical, 48 bits virtual
     CPU:                             24
     在线 CPU 列表：                  0-23
     每个核的线程数：                 1
     每个座的核数：                   16
     座：                             1
     NUMA 节点：                      1
     厂商 ID：                        GenuineIntel
     CPU 系列：                       6
     型号：                           151
     型号名称：                       12th Gen Intel(R) Core(TM) i9-12900K
     步进：                           2
     CPU MHz：                        3200.000
     CPU 最大 MHz：                   5200.0000
     CPU 最小 MHz：                   800.0000
     BogoMIPS：                       6374.40
     虚拟化：                         VT-x
     L1d 缓存：                       384 KiB
     L1i 缓存：                       256 KiB
     L2 缓存：                        10 MiB
     NUMA 节点0 CPU：                 0-23
     Vulnerability Itlb multihit:     Not affected
     Vulnerability L1tf:              Not affected
     Vulnerability Mds:               Not affected
     Vulnerability Meltdown:          Not affected
     Vulnerability Mmio stale data:   Not affected
     Vulnerability Retbleed:          Not affected
     Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp
     Vulnerability Spectre v1:        Mitigation; usercopy/swapgs barriers and __user pointer sanitization
     Vulnerability Spectre v2:        Mitigation; Enhanced IBRS, IBPB conditional, RSB filling, PBRSB-eIBRS SW sequence
     Vulnerability Srbds:             Not affected
     Vulnerability Tsx async abort:   Not affected
     标记：                           fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l2 invpcid_single cdp_l2 ssbd ibrs ibpb stibp ibrs_enhanced tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid rdt_a rdseed adx smap clflushopt clwb intel_pt sha_ni xsaveopt xsavec xgetbv1 xsaves split_lock_detect avx_vnni dtherm ida arat pln pts hwp hwp_notify hwp_act_window hwp_epp hwp_pkg_req umip pku ospke waitpkg gfni vaes vpclmulqdq tme rdpid movdiri movdir64b fsrm md_clear serialize pconfig arch_lbr flush_l1d arch_capabilities

     ***CMake***

     ***g++***
     /usr/bin/g++
     g++ (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0
     Copyright (C) 2019 Free Software Foundation, Inc.
     This is free software; see the source for copying conditions.  There is NO
     warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

     ***nvcc***

     ***Python***
     /usr/bin/python
     Python 3.8.10

     ***Environment Variables***
     PATH                            : /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin
     LD_LIBRARY_PATH                 :
     NUMBAPRO_NVVM                   :
     NUMBAPRO_LIBDEVICE              :
     CONDA_PREFIX                    :
     PYTHON_PATH                     :

     conda not found
     ***pip packages***
     /usr/bin/pip
     Package                  Version
     ------------------------ --------------------
     apturl                   0.5.2
     bcrypt                   3.1.7
     blinker                  1.4
     Brlapi                   0.7.0
     certifi                  2019.11.28
     chardet                  3.0.4
     Click                    7.0
     colorama                 0.4.3
     command-not-found        0.3
     cryptography             2.8
     cupshelpers              1.0
     dbus-python              1.2.16
     defer                    1.0.6
     dill                     0.3.6
     distro                   1.4.0
     distro-info              0.23ubuntu1
     duplicity                0.8.12.0
     entrypoints              0.3
     fasteners                0.14.1
     future                   0.18.2
     httplib2                 0.14.0
     idna                     2.8
     keyring                  18.0.1
     language-selector        0.1
     launchpadlib             1.10.13
     lazr.restfulclient       0.14.2
     lazr.uri                 1.0.3
     lockfile                 0.12.2
     louis                    3.12.0
     macaroonbakery           1.3.1
     Mako                     1.1.0
     MarkupSafe               1.1.0
     monotonic                1.5
     netifaces                0.10.4
     numpy                    1.24.0
     nvcv-python              0.2.0a0
     nvidia-cublas-cu11       11.10.3.66
     nvidia-cuda-nvrtc-cu11   11.7.99
     nvidia-cuda-runtime-cu11 11.7.99
     nvidia-cudnn-cu11        8.5.0.96
     oauthlib                 3.1.0
     olefile                  0.46
     opencv-python            4.7.0.68
     paramiko                 2.6.0
     pexpect                  4.6.0
     Pillow                   7.0.0
     pip                      20.0.2
     protobuf                 3.6.1
     pycairo                  1.16.2
     pycups                   1.9.73
     PyGObject                3.36.0
     PyJWT                    1.7.1
     pymacaroons              0.13.0
     PyNaCl                   1.3.0
     pyRFC3339                1.1
     python-apt               2.0.0+ubuntu0.20.4.8
     python-dateutil          2.7.3
     python-debian            0.1.36ubuntu1
     PyTurboJPEG              1.7.0
     pytz                     2019.3
     pyxdg                    0.26
     PyYAML                   5.3.1
     reportlab                3.5.34
     requests                 2.22.0
     requests-unixsocket      0.2.0
     screen-resolution-extra  0.0.0
     SecretStorage            2.3.1
     setuptools               45.2.0
     simplejson               3.16.0
     six                      1.14.0
     ssh-import-id            5.10
     systemd-python           234
     torch                    1.13.1
     torchnvjpeg              0.1.0
     torchvision              0.14.1
     tqdm                     4.64.1
     typing-extensions        4.4.0
     ubuntu-advantage-tools   27.12
     ubuntu-drivers-common    0.0.0
     ufw                      0.36
     unattended-upgrades      0.1
     urllib3                  1.25.8
     usb-creator              0.3.7
     wadllib                  1.3.3
     wheel                    0.34.2
     xkit                     0.0.0
complexfilter commented 1 year ago

I have seen similar error. Any idea to address this?

tp-nan commented 10 months ago

Only contiguous torch.Tensor is supported at this moment:

cvcuda.as_tensor(imgs.contiguous(), "NHWC")

bhaefnerNV commented 9 months ago

@scse-l, thank you for your interest in CV-CUDA.

Indeed for constructing a CVCUDA tensor object, we expect the strides to be constant for each dimension. As pointed out by @tp-nan in his answer, this can be achieved via having a contiguous in memory tensor.

We'll close this issue for now, feel free to reopen if this did not resolve your problem.