Closed LeftAttention closed 2 years ago
It seems the loss not changing either. So losses should all be zero except for flow_reg
and flow_cor
. Did you install the cuda function from GFLA correctly? Could you try to run it on a single GPU to see if it works?
This is my system specs
NVIDIA-SMI 470.57.02 Driver Version: 470.57.02 CUDA Version: 11.4
And I have four Tesla K80
GPUs. Because of that I changed Global-Flow-Local-Attention/model/networks/block_extractor/setup.py
, Global-Flow-Local-Attention/model/networks/local_attn_reshape/setup.py
and Global-Flow-Local-Attention/model/networks/resample2d_package/setup.py
.
I modified the nvcc_args
to the following.
nvcc_args = [
'-gencode', 'arch=compute_86,code=sm_86'
#'-gencode', 'arch=compute_50,code=sm_50',
#'-gencode', 'arch=compute_52,code=sm_52',
# '-gencode', 'arch=compute_60,code=sm_60',
# '-gencode', 'arch=compute_61,code=sm_61',
# '-gencode', 'arch=compute_70,code=sm_70',
# '-gencode', 'arch=compute_70,code=compute_70'
]
Please guide me how to setup this in k80 GPU and cuda 11.2. Thanks in advance.
I followed this for the environment setup. And these are my installation logs of Global-Flow-Local-Attention custom layers.
running clean
'build/lib.linux-x86_64-3.6' does not exist -- can't clean it
'build/bdist.linux-x86_64' does not exist -- can't clean it
'build/scripts-3.6' does not exist -- can't clean it
running install
running bdist_egg
running egg_info
creating block_extractor_cuda.egg-info
writing block_extractor_cuda.egg-info/PKG-INFO
writing dependency_links to block_extractor_cuda.egg-info/dependency_links.txt
writing top-level names to block_extractor_cuda.egg-info/top_level.txt
writing manifest file 'block_extractor_cuda.egg-info/SOURCES.txt'
reading manifest file 'block_extractor_cuda.egg-info/SOURCES.txt'
writing manifest file 'block_extractor_cuda.egg-info/SOURCES.txt'
installing library code to build/bdist.linux-x86_64/egg
running install_lib
running build_ext
building 'block_extractor_cuda' extension
creating build
creating build/temp.linux-x86_64-3.6
gcc -pthread -B /home/bigthinx/anaconda3/envs/gfla/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/home/bigthinx/anaconda3/envs/gfla/lib/python3.6/site-packages/torch/lib/include -I/home/bigthinx/anaconda3/envs/gfla/lib/python3.6/site-packages/torch/lib/include/torch/csrc/api/include -I/home/bigthinx/anaconda3/envs/gfla/lib/python3.6/site-packages/torch/lib/include/TH -I/home/bigthinx/anaconda3/envs/gfla/lib/python3.6/site-packages/torch/lib/include/THC -I/usr/local/cuda/include -I/home/bigthinx/anaconda3/envs/gfla/include/python3.6m -c block_extractor_cuda.cc -o build/temp.linux-x86_64-3.6/block_extractor_cuda.o -std=c++11 -DTORCH_API_INCLUDE_EXTENSION_H -DTORCH_EXTENSION_NAME=block_extractor_cuda -D_GLIBCXX_USE_CXX11_ABI=0
cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
In file included from block_extractor_cuda.cc:2:0:
/home/bigthinx/anaconda3/envs/gfla/lib/python3.6/site-packages/torch/lib/include/torch/csrc/api/include/torch/torch.h:7:2: warning: #warning "Including torch/torch.h for C++ extensions is deprecated. Please include torch/extension.h" [-Wcpp]
#warning \
^~~~~~~
/usr/local/cuda/bin/nvcc -I/home/bigthinx/anaconda3/envs/gfla/lib/python3.6/site-packages/torch/lib/include -I/home/bigthinx/anaconda3/envs/gfla/lib/python3.6/site-packages/torch/lib/include/torch/csrc/api/include -I/home/bigthinx/anaconda3/envs/gfla/lib/python3.6/site-packages/torch/lib/include/TH -I/home/bigthinx/anaconda3/envs/gfla/lib/python3.6/site-packages/torch/lib/include/THC -I/usr/local/cuda/include -I/home/bigthinx/anaconda3/envs/gfla/include/python3.6m -c block_extractor_kernel.cu -o build/temp.linux-x86_64-3.6/block_extractor_kernel.o -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --compiler-options '-fPIC' -gencode arch=compute_86,code=sm_86 -DTORCH_API_INCLUDE_EXTENSION_H -DTORCH_EXTENSION_NAME=block_extractor_cuda -D_GLIBCXX_USE_CXX11_ABI=0 -std=c++11
creating build/lib.linux-x86_64-3.6
g++ -pthread -shared -B /home/bigthinx/anaconda3/envs/gfla/compiler_compat -L/home/bigthinx/anaconda3/envs/gfla/lib -Wl,-rpath=/home/bigthinx/anaconda3/envs/gfla/lib -Wl,--no-as-needed -Wl,--sysroot=/ build/temp.linux-x86_64-3.6/block_extractor_cuda.o build/temp.linux-x86_64-3.6/block_extractor_kernel.o -L/usr/local/cuda/lib64 -lcudart -o build/lib.linux-x86_64-3.6/block_extractor_cuda.cpython-36m-x86_64-linux-gnu.so
creating build/bdist.linux-x86_64
creating build/bdist.linux-x86_64/egg
copying build/lib.linux-x86_64-3.6/block_extractor_cuda.cpython-36m-x86_64-linux-gnu.so -> build/bdist.linux-x86_64/egg
creating stub loader for block_extractor_cuda.cpython-36m-x86_64-linux-gnu.so
byte-compiling build/bdist.linux-x86_64/egg/block_extractor_cuda.py to block_extractor_cuda.cpython-36.pyc
creating build/bdist.linux-x86_64/egg/EGG-INFO
copying block_extractor_cuda.egg-info/PKG-INFO -> build/bdist.linux-x86_64/egg/EGG-INFO
copying block_extractor_cuda.egg-info/SOURCES.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying block_extractor_cuda.egg-info/dependency_links.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying block_extractor_cuda.egg-info/top_level.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
writing build/bdist.linux-x86_64/egg/EGG-INFO/native_libs.txt
zip_safe flag not set; analyzing archive contents...
__pycache__.block_extractor_cuda.cpython-36: module references __file__
creating dist
creating 'dist/block_extractor_cuda-0.0.0-py3.6-linux-x86_64.egg' and adding 'build/bdist.linux-x86_64/egg' to it
removing 'build/bdist.linux-x86_64/egg' (and everything under it)
Processing block_extractor_cuda-0.0.0-py3.6-linux-x86_64.egg
removing '/home/bigthinx/.local/lib/python3.6/site-packages/block_extractor_cuda-0.0.0-py3.6-linux-x86_64.egg' (and everything under it)
creating /home/bigthinx/.local/lib/python3.6/site-packages/block_extractor_cuda-0.0.0-py3.6-linux-x86_64.egg
Extracting block_extractor_cuda-0.0.0-py3.6-linux-x86_64.egg to /home/bigthinx/.local/lib/python3.6/site-packages
block-extractor-cuda 0.0.0 is already the active version in easy-install.pth
Installed /home/bigthinx/.local/lib/python3.6/site-packages/block_extractor_cuda-0.0.0-py3.6-linux-x86_64.egg
Processing dependencies for block-extractor-cuda==0.0.0
Finished processing dependencies for block-extractor-cuda==0.0.0
running clean
'build/lib.linux-x86_64-3.6' does not exist -- can't clean it
'build/bdist.linux-x86_64' does not exist -- can't clean it
'build/scripts-3.6' does not exist -- can't clean it
running install
running bdist_egg
running egg_info
creating local_attn_reshape_cuda.egg-info
writing local_attn_reshape_cuda.egg-info/PKG-INFO
writing dependency_links to local_attn_reshape_cuda.egg-info/dependency_links.txt
writing top-level names to local_attn_reshape_cuda.egg-info/top_level.txt
writing manifest file 'local_attn_reshape_cuda.egg-info/SOURCES.txt'
reading manifest file 'local_attn_reshape_cuda.egg-info/SOURCES.txt'
writing manifest file 'local_attn_reshape_cuda.egg-info/SOURCES.txt'
installing library code to build/bdist.linux-x86_64/egg
running install_lib
running build_ext
building 'local_attn_reshape_cuda' extension
creating build
creating build/temp.linux-x86_64-3.6
gcc -pthread -B /home/bigthinx/anaconda3/envs/gfla/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/home/bigthinx/anaconda3/envs/gfla/lib/python3.6/site-packages/torch/lib/include -I/home/bigthinx/anaconda3/envs/gfla/lib/python3.6/site-packages/torch/lib/include/torch/csrc/api/include -I/home/bigthinx/anaconda3/envs/gfla/lib/python3.6/site-packages/torch/lib/include/TH -I/home/bigthinx/anaconda3/envs/gfla/lib/python3.6/site-packages/torch/lib/include/THC -I/usr/local/cuda/include -I/home/bigthinx/anaconda3/envs/gfla/include/python3.6m -c local_attn_reshape_cuda.cc -o build/temp.linux-x86_64-3.6/local_attn_reshape_cuda.o -std=c++11 -DTORCH_API_INCLUDE_EXTENSION_H -DTORCH_EXTENSION_NAME=local_attn_reshape_cuda -D_GLIBCXX_USE_CXX11_ABI=0
cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
In file included from local_attn_reshape_cuda.cc:2:0:
/home/bigthinx/anaconda3/envs/gfla/lib/python3.6/site-packages/torch/lib/include/torch/csrc/api/include/torch/torch.h:7:2: warning: #warning "Including torch/torch.h for C++ extensions is deprecated. Please include torch/extension.h" [-Wcpp]
#warning \
^~~~~~~
/usr/local/cuda/bin/nvcc -I/home/bigthinx/anaconda3/envs/gfla/lib/python3.6/site-packages/torch/lib/include -I/home/bigthinx/anaconda3/envs/gfla/lib/python3.6/site-packages/torch/lib/include/torch/csrc/api/include -I/home/bigthinx/anaconda3/envs/gfla/lib/python3.6/site-packages/torch/lib/include/TH -I/home/bigthinx/anaconda3/envs/gfla/lib/python3.6/site-packages/torch/lib/include/THC -I/usr/local/cuda/include -I/home/bigthinx/anaconda3/envs/gfla/include/python3.6m -c local_attn_reshape_kernel.cu -o build/temp.linux-x86_64-3.6/local_attn_reshape_kernel.o -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --compiler-options '-fPIC' -gencode arch=compute_86,code=sm_86 -DTORCH_API_INCLUDE_EXTENSION_H -DTORCH_EXTENSION_NAME=local_attn_reshape_cuda -D_GLIBCXX_USE_CXX11_ABI=0 -std=c++11
creating build/lib.linux-x86_64-3.6
g++ -pthread -shared -B /home/bigthinx/anaconda3/envs/gfla/compiler_compat -L/home/bigthinx/anaconda3/envs/gfla/lib -Wl,-rpath=/home/bigthinx/anaconda3/envs/gfla/lib -Wl,--no-as-needed -Wl,--sysroot=/ build/temp.linux-x86_64-3.6/local_attn_reshape_cuda.o build/temp.linux-x86_64-3.6/local_attn_reshape_kernel.o -L/usr/local/cuda/lib64 -lcudart -o build/lib.linux-x86_64-3.6/local_attn_reshape_cuda.cpython-36m-x86_64-linux-gnu.so
creating build/bdist.linux-x86_64
creating build/bdist.linux-x86_64/egg
copying build/lib.linux-x86_64-3.6/local_attn_reshape_cuda.cpython-36m-x86_64-linux-gnu.so -> build/bdist.linux-x86_64/egg
creating stub loader for local_attn_reshape_cuda.cpython-36m-x86_64-linux-gnu.so
byte-compiling build/bdist.linux-x86_64/egg/local_attn_reshape_cuda.py to local_attn_reshape_cuda.cpython-36.pyc
creating build/bdist.linux-x86_64/egg/EGG-INFO
copying local_attn_reshape_cuda.egg-info/PKG-INFO -> build/bdist.linux-x86_64/egg/EGG-INFO
copying local_attn_reshape_cuda.egg-info/SOURCES.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying local_attn_reshape_cuda.egg-info/dependency_links.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying local_attn_reshape_cuda.egg-info/top_level.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
writing build/bdist.linux-x86_64/egg/EGG-INFO/native_libs.txt
zip_safe flag not set; analyzing archive contents...
__pycache__.local_attn_reshape_cuda.cpython-36: module references __file__
creating dist
creating 'dist/local_attn_reshape_cuda-0.0.0-py3.6-linux-x86_64.egg' and adding 'build/bdist.linux-x86_64/egg' to it
removing 'build/bdist.linux-x86_64/egg' (and everything under it)
Processing local_attn_reshape_cuda-0.0.0-py3.6-linux-x86_64.egg
removing '/home/bigthinx/.local/lib/python3.6/site-packages/local_attn_reshape_cuda-0.0.0-py3.6-linux-x86_64.egg' (and everything under it)
creating /home/bigthinx/.local/lib/python3.6/site-packages/local_attn_reshape_cuda-0.0.0-py3.6-linux-x86_64.egg
Extracting local_attn_reshape_cuda-0.0.0-py3.6-linux-x86_64.egg to /home/bigthinx/.local/lib/python3.6/site-packages
local-attn-reshape-cuda 0.0.0 is already the active version in easy-install.pth
Installed /home/bigthinx/.local/lib/python3.6/site-packages/local_attn_reshape_cuda-0.0.0-py3.6-linux-x86_64.egg
Processing dependencies for local-attn-reshape-cuda==0.0.0
Finished processing dependencies for local-attn-reshape-cuda==0.0.0
running clean
'build/lib.linux-x86_64-3.6' does not exist -- can't clean it
'build/bdist.linux-x86_64' does not exist -- can't clean it
'build/scripts-3.6' does not exist -- can't clean it
running install
running bdist_egg
running egg_info
creating resample2d_cuda.egg-info
writing resample2d_cuda.egg-info/PKG-INFO
writing dependency_links to resample2d_cuda.egg-info/dependency_links.txt
writing top-level names to resample2d_cuda.egg-info/top_level.txt
writing manifest file 'resample2d_cuda.egg-info/SOURCES.txt'
reading manifest file 'resample2d_cuda.egg-info/SOURCES.txt'
writing manifest file 'resample2d_cuda.egg-info/SOURCES.txt'
installing library code to build/bdist.linux-x86_64/egg
running install_lib
running build_ext
building 'resample2d_cuda' extension
creating build
creating build/temp.linux-x86_64-3.6
gcc -pthread -B /home/bigthinx/anaconda3/envs/gfla/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/home/bigthinx/anaconda3/envs/gfla/lib/python3.6/site-packages/torch/lib/include -I/home/bigthinx/anaconda3/envs/gfla/lib/python3.6/site-packages/torch/lib/include/torch/csrc/api/include -I/home/bigthinx/anaconda3/envs/gfla/lib/python3.6/site-packages/torch/lib/include/TH -I/home/bigthinx/anaconda3/envs/gfla/lib/python3.6/site-packages/torch/lib/include/THC -I/usr/local/cuda/include -I/home/bigthinx/anaconda3/envs/gfla/include/python3.6m -c resample2d_cuda.cc -o build/temp.linux-x86_64-3.6/resample2d_cuda.o -std=c++11 -DTORCH_API_INCLUDE_EXTENSION_H -DTORCH_EXTENSION_NAME=resample2d_cuda -D_GLIBCXX_USE_CXX11_ABI=0
cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
In file included from resample2d_cuda.cc:2:0:
/home/bigthinx/anaconda3/envs/gfla/lib/python3.6/site-packages/torch/lib/include/torch/csrc/api/include/torch/torch.h:7:2: warning: #warning "Including torch/torch.h for C++ extensions is deprecated. Please include torch/extension.h" [-Wcpp]
#warning \
^~~~~~~
/usr/local/cuda/bin/nvcc -I/home/bigthinx/anaconda3/envs/gfla/lib/python3.6/site-packages/torch/lib/include -I/home/bigthinx/anaconda3/envs/gfla/lib/python3.6/site-packages/torch/lib/include/torch/csrc/api/include -I/home/bigthinx/anaconda3/envs/gfla/lib/python3.6/site-packages/torch/lib/include/TH -I/home/bigthinx/anaconda3/envs/gfla/lib/python3.6/site-packages/torch/lib/include/THC -I/usr/local/cuda/include -I/home/bigthinx/anaconda3/envs/gfla/include/python3.6m -c resample2d_kernel.cu -o build/temp.linux-x86_64-3.6/resample2d_kernel.o -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --compiler-options '-fPIC' -gencode arch=compute_86,code=sm_86 -DTORCH_API_INCLUDE_EXTENSION_H -DTORCH_EXTENSION_NAME=resample2d_cuda -D_GLIBCXX_USE_CXX11_ABI=0 -std=c++11
creating build/lib.linux-x86_64-3.6
g++ -pthread -shared -B /home/bigthinx/anaconda3/envs/gfla/compiler_compat -L/home/bigthinx/anaconda3/envs/gfla/lib -Wl,-rpath=/home/bigthinx/anaconda3/envs/gfla/lib -Wl,--no-as-needed -Wl,--sysroot=/ build/temp.linux-x86_64-3.6/resample2d_cuda.o build/temp.linux-x86_64-3.6/resample2d_kernel.o -L/usr/local/cuda/lib64 -lcudart -o build/lib.linux-x86_64-3.6/resample2d_cuda.cpython-36m-x86_64-linux-gnu.so
creating build/bdist.linux-x86_64
creating build/bdist.linux-x86_64/egg
copying build/lib.linux-x86_64-3.6/resample2d_cuda.cpython-36m-x86_64-linux-gnu.so -> build/bdist.linux-x86_64/egg
creating stub loader for resample2d_cuda.cpython-36m-x86_64-linux-gnu.so
byte-compiling build/bdist.linux-x86_64/egg/resample2d_cuda.py to resample2d_cuda.cpython-36.pyc
creating build/bdist.linux-x86_64/egg/EGG-INFO
copying resample2d_cuda.egg-info/PKG-INFO -> build/bdist.linux-x86_64/egg/EGG-INFO
copying resample2d_cuda.egg-info/SOURCES.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying resample2d_cuda.egg-info/dependency_links.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying resample2d_cuda.egg-info/top_level.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
writing build/bdist.linux-x86_64/egg/EGG-INFO/native_libs.txt
zip_safe flag not set; analyzing archive contents...
__pycache__.resample2d_cuda.cpython-36: module references __file__
creating dist
creating 'dist/resample2d_cuda-0.0.0-py3.6-linux-x86_64.egg' and adding 'build/bdist.linux-x86_64/egg' to it
removing 'build/bdist.linux-x86_64/egg' (and everything under it)
Processing resample2d_cuda-0.0.0-py3.6-linux-x86_64.egg
removing '/home/bigthinx/.local/lib/python3.6/site-packages/resample2d_cuda-0.0.0-py3.6-linux-x86_64.egg' (and everything under it)
creating /home/bigthinx/.local/lib/python3.6/site-packages/resample2d_cuda-0.0.0-py3.6-linux-x86_64.egg
Extracting resample2d_cuda-0.0.0-py3.6-linux-x86_64.egg to /home/bigthinx/.local/lib/python3.6/site-packages
resample2d-cuda 0.0.0 is already the active version in easy-install.pth
Installed /home/bigthinx/.local/lib/python3.6/site-packages/resample2d_cuda-0.0.0-py3.6-linux-x86_64.egg
Processing dependencies for resample2d-cuda==0.0.0
Finished processing dependencies for resample2d-cuda==0.0.0
I think if there is no device-side bug triggered, then GFLA function is probably install correctly, although I didn't try this on K-80 on my own. Can you try it on a single GPU with smaller batch size to see the behavior for sanity check?
I tried with that, but same issue. If you don't mind, would you please share the flownet trained weight?
There are five different weights here. I am confused which weight should I use. Thanks in advance.
Please use GFLA's weights of Pose-Guided Person Image Generation (the weights trained on fashion dataset.) Then extract the flownet weights from the latest_net_G.pth
(the weight of the full GFLA model) by something like
import torch
weights = torch.load("latest_net_G.pth")
flownet_weights = {w.replace("flow_net.",""): weights[w] for w in weights if w.startswith("flow_net")}
torch.save(flownet_weights, "flownet.pth")
Note GFLA is trained on 256x256 images.
-- marked as solved as no more updates. Will turn it active if there is any follow-up.
During the flow net warm up. I am getting the following logs. I am not understanding why all the values are zero.
These are my training parameters.