AnderBiguri
commented
12 months ago HI, @JiahaoPlus, Can you clarify what you mean by "stuck"? How do you measure that? What is your GPU? Can you reproduce it using TIGRE code and data (headphantom)?
Open JiahaoPlus opened 12 months ago
AnderBiguri
commented
12 months ago HI, @JiahaoPlus, Can you clarify what you mean by "stuck"? How do you measure that? What is your GPU? Can you reproduce it using TIGRE code and data (headphantom)?
caiyuanhao1998
commented
12 months ago Hi, I also met this problem. The stuck is like this:
caiyuanhao1998
commented
12 months ago I use RTX 8000 and use the eval.py to test.
JiahaoPlus
commented
12 months ago Hi @AnderBiguri, by "stuck" I mean the algorithm keeps running in Line 222 and there is no response. I found it out by setting checkpoints and running line by line. I use A5000 GPU. Will try TIGRE code and data (headphantom).
JiahaoPlus
commented
12 months ago TIGRE demo TIGRE/Python/demos/d07_Algorithms02.py with head phantom data works well, but here the CT size is small (64x64x64).
AnderBiguri
commented
12 months ago Hi @JiahaoPlus , you can change the head phantom to any size when you call it, can you try to make it the same size as your data?
I am just trying to figure out what is going on, as I have reconstructed images >2000^3, and while that call, Ax, takes long at big images, it shouldn't be more than few seconds at your image size
JiahaoPlus
commented
12 months ago Hi @AnderBiguri, thanks for your reply! I changed the head phantom size to 512x512x512, and it worked well.
However, my data cannot work. I guess the problem comes from the "geometry" parameters. If I multiply the "geo.dVoxel" by 1000 (or 100, or 10) in my data, it will not get stuck, but the reconstructed image is wrong (because of the wrong geo.dVoxel).
Hereby I attach the minimal code to reproduce the problem (you can download the data here and change the path):
import tigre
import numpy as np
import tigre.algorithms as algs
import pickle
path = '/path/to/abdomen_50.pickle'
with open(path, "rb") as handle:
data = pickle.load(handle)
#%% Geometry (unit: mm)
geo = tigre.geometry_default(high_resolution=False)
geo.DSD = data["DSD"] # Distance Source Detector
geo.DSO = data["DSO"] # Distance Source Origin
# Detector parameters
geo.nDetector = np.array(data["nDetector"]) # number of pixels
geo.dDetector = np.array(data["dDetector"]) # size of each pixel
geo.sDetector = geo.nDetector * geo.dDetector # total size of the detector
# Image parameters
geo.nVoxel = np.array(data["nVoxel"]) # number of voxels
geo.dVoxel = np.array(data["dVoxel"]) # size of each voxel
geo.sVoxel = geo.nVoxel * geo.dVoxel # total size of the image
geo.nVoxel = np.flip(geo.nVoxel)
geo.dVoxel = np.flip(geo.dVoxel)
geo.sVoxel = np.flip(geo.sVoxel)
#%% Load image and projections
image = data["image"]
train_angles = data["train"]["angles"]
train_projs = data["train"]["projections"]
#%% Algorithms
imgSART = algs.sart(train_projs, geo, train_angles, niter=4)
print(imgSART.shape)When I print "geo", I got:
ipdb> print(geo)
TIGRE parameters
-----
Geometry parameters
Distance from source to detector (DSD) = 1500.0 mm
Distance from source to origin (DSO)= 1000.0 mm
-----
Detector parameters
Number of pixels (nDetector) = [1024 1024]
Size of each pixel (dDetector) = [1. 1.] mm
Total size of the detector (sDetector) = [1024. 1024.] mm
-----
Image parameters
Number of voxels (nVoxel) = [463 512 512]
Total size of the image (sVoxel) = [463. 320. 320.] mm
Size of each voxel (dVoxel) = [1. 0.625 0.625] mm
-----
Offset correction parameters
Offset of image from origin (offOrigin) = [0 0 0] mm
Offset of detector (offDetector) = [0 0] mm
-----
Auxillary parameters
Samples per pixel of forward projection (accuracy) = 0.5
-----
Rotation of the Detector (rotDetector) = [0 0 0] radThank you very much for looking into it!
JiahaoPlus
commented
12 months ago Hi @AnderBiguri, if I change the tigre version to 2.3, everything works well! The same code could not work in the current version (2.5).
AnderBiguri
commented
12 months ago Hum, interesting @JiahaoPlus.
Are you using a conda enviroment? Are you able to provide the versions of the libraries you are using with e.g. conda list ?
I ask because TIGRE 2.5 changed some things to account for updates that break compatibility by numpy and cython. It may be possible that your code is hanging in a weird way if you don't have the newest versions of these libraries. But if that is the case, the smart thing for TIGRE to do would be to catch it at e.g. install time and warn you about incompatibilities, instead of silently hanging sometimes.
It may be related to #492 , perhaps.
I ping @tsadakane just because they had a look at this in the past and may know whats going on, but no expectations of replying or trying to fix it, tsadakane!
JiahaoPlus
commented
12 months ago Hi @AnderBiguri, yes I am using conda.
Here is the conda list with the newest TIGRE (2.5) which could not run well on my data (some packages are not relevant to TIGRE, but just in case, I put all of them here):
# Name Version Build Channel
_libgcc_mutex 0.1 main
_openmp_mutex 5.1 1_gnu
absl-py 2.0.0 pypi_0 pypi
argparse 1.4.0 pypi_0 pypi
asttokens 2.4.0 pypi_0 pypi
backcall 0.2.0 pypi_0 pypi
blessed 1.20.0 pypi_0 pypi
ca-certificates 2023.08.22 h06a4308_0
cachetools 5.3.1 pypi_0 pypi
certifi 2023.7.22 pypi_0 pypi
charset-normalizer 3.2.0 pypi_0 pypi
contourpy 1.1.1 pypi_0 pypi
cycler 0.11.0 pypi_0 pypi
cython 3.0.2 pypi_0 pypi
decorator 5.1.1 pypi_0 pypi
exceptiongroup 1.1.3 pypi_0 pypi
executing 1.2.0 pypi_0 pypi
fonttools 4.42.1 pypi_0 pypi
google-auth 2.23.0 pypi_0 pypi
google-auth-oauthlib 0.4.6 pypi_0 pypi
gpustat 1.1.1 pypi_0 pypi
grpcio 1.58.0 pypi_0 pypi
idna 3.4 pypi_0 pypi
imageio 2.25.1 pypi_0 pypi
importlib-metadata 6.8.0 pypi_0 pypi
importlib-resources 6.0.1 pypi_0 pypi
ipdb 0.13.13 pypi_0 pypi
ipython 8.15.0 pypi_0 pypi
jedi 0.19.0 pypi_0 pypi
kiwisolver 1.4.5 pypi_0 pypi
ld_impl_linux-64 2.38 h1181459_1
libffi 3.4.4 h6a678d5_0
libgcc-ng 11.2.0 h1234567_1
libgomp 11.2.0 h1234567_1
libstdcxx-ng 11.2.0 h1234567_1
markdown 3.4.4 pypi_0 pypi
markupsafe 2.1.3 pypi_0 pypi
matplotlib 3.7.0 pypi_0 pypi
matplotlib-inline 0.1.6 pypi_0 pypi
ncurses 6.4 h6a678d5_0
networkx 3.1 pypi_0 pypi
ninja 1.11.1 pypi_0 pypi
numpy 1.26.0 pypi_0 pypi
nvidia-ml-py 12.535.108 pypi_0 pypi
nvitop 1.3.0 pypi_0 pypi
oauthlib 3.2.2 pypi_0 pypi
opencv-python 4.7.0.68 pypi_0 pypi
openssl 3.0.10 h7f8727e_2
packaging 23.1 pypi_0 pypi
parso 0.8.3 pypi_0 pypi
pexpect 4.8.0 pypi_0 pypi
pickleshare 0.7.5 pypi_0 pypi
pillow 9.5.0 pypi_0 pypi
pip 23.2.1 py39h06a4308_0
prompt-toolkit 3.0.39 pypi_0 pypi
protobuf 4.24.3 pypi_0 pypi
psutil 5.9.5 pypi_0 pypi
ptyprocess 0.7.0 pypi_0 pypi
pure-eval 0.2.2 pypi_0 pypi
pyasn1 0.5.0 pypi_0 pypi
pyasn1-modules 0.3.0 pypi_0 pypi
pygments 2.16.1 pypi_0 pypi
pyparsing 3.1.1 pypi_0 pypi
python 3.9.18 h955ad1f_0
python-dateutil 2.8.2 pypi_0 pypi
pytigre 2.4.0 pypi_0 pypi
pywavelets 1.4.1 pypi_0 pypi
pyyaml 6.0 pypi_0 pypi
readline 8.2 h5eee18b_0
requests 2.31.0 pypi_0 pypi
requests-oauthlib 1.3.1 pypi_0 pypi
rsa 4.9 pypi_0 pypi
scikit-image 0.19.3 pypi_0 pypi
scipy 1.10.0 pypi_0 pypi
setuptools 68.0.0 py39h06a4308_0
six 1.16.0 pypi_0 pypi
sqlite 3.41.2 h5eee18b_0
stack-data 0.6.2 pypi_0 pypi
tensorboard 2.12.0 pypi_0 pypi
tensorboard-data-server 0.7.1 pypi_0 pypi
tensorboard-plugin-wit 1.8.1 pypi_0 pypi
termcolor 2.3.0 pypi_0 pypi
tifffile 2023.9.18 pypi_0 pypi
tk 8.6.12 h1ccaba5_0
tomli 2.0.1 pypi_0 pypi
torch 1.11.0+cu113 pypi_0 pypi
torchaudio 0.11.0+cu113 pypi_0 pypi
torchvision 0.12.0+cu113 pypi_0 pypi
tqdm 4.64.1 pypi_0 pypi
traitlets 5.10.0 pypi_0 pypi
typing-extensions 4.8.0 pypi_0 pypi
tzdata 2023c h04d1e81_0
urllib3 1.26.16 pypi_0 pypi
wcwidth 0.2.6 pypi_0 pypi
werkzeug 2.3.7 pypi_0 pypi
wheel 0.38.4 py39h06a4308_0
xz 5.4.2 h5eee18b_0
zipp 3.17.0 pypi_0 pypi
zlib 1.2.13 h5eee18b_0 Here is the TIGRE 2.3 related conda list that can work well:
# Name Version Build Channel
_libgcc_mutex 0.1 main
_openmp_mutex 5.1 1_gnu
absl-py 2.0.0 pypi_0 pypi
argparse 1.4.0 pypi_0 pypi
asttokens 2.4.0 pypi_0 pypi
backcall 0.2.0 pypi_0 pypi
blessed 1.20.0 pypi_0 pypi
ca-certificates 2023.08.22 h06a4308_0
cachetools 5.3.1 pypi_0 pypi
certifi 2023.7.22 pypi_0 pypi
charset-normalizer 3.3.0 pypi_0 pypi
contourpy 1.1.1 pypi_0 pypi
cycler 0.12.1 pypi_0 pypi
cython 0.29.25 pypi_0 pypi
decorator 5.1.1 pypi_0 pypi
exceptiongroup 1.1.3 pypi_0 pypi
executing 2.0.0 pypi_0 pypi
fonttools 4.43.1 pypi_0 pypi
google-auth 2.23.2 pypi_0 pypi
google-auth-oauthlib 0.4.6 pypi_0 pypi
gpustat 1.1.1 pypi_0 pypi
grpcio 1.59.0 pypi_0 pypi
idna 3.4 pypi_0 pypi
imageio 2.25.1 pypi_0 pypi
importlib-metadata 6.8.0 pypi_0 pypi
importlib-resources 6.1.0 pypi_0 pypi
ipdb 0.13.13 pypi_0 pypi
ipython 8.16.1 pypi_0 pypi
jedi 0.19.1 pypi_0 pypi
kiwisolver 1.4.5 pypi_0 pypi
ld_impl_linux-64 2.38 h1181459_1
libffi 3.4.4 h6a678d5_0
libgcc-ng 11.2.0 h1234567_1
libgomp 11.2.0 h1234567_1
libstdcxx-ng 11.2.0 h1234567_1
markdown 3.5 pypi_0 pypi
markupsafe 2.1.3 pypi_0 pypi
matplotlib 3.7.0 pypi_0 pypi
matplotlib-inline 0.1.6 pypi_0 pypi
ncurses 6.4 h6a678d5_0
networkx 3.1 pypi_0 pypi
ninja 1.11.1 pypi_0 pypi
numpy 1.21.6 pypi_0 pypi
nvidia-ml-py 12.535.108 pypi_0 pypi
oauthlib 3.2.2 pypi_0 pypi
opencv-python 4.7.0.68 pypi_0 pypi
openssl 3.0.11 h7f8727e_2
packaging 23.2 pypi_0 pypi
parso 0.8.3 pypi_0 pypi
pexpect 4.8.0 pypi_0 pypi
pickleshare 0.7.5 pypi_0 pypi
pillow 10.0.1 pypi_0 pypi
pip 23.2.1 py39h06a4308_0
prompt-toolkit 3.0.39 pypi_0 pypi
protobuf 4.24.4 pypi_0 pypi
psutil 5.9.5 pypi_0 pypi
ptyprocess 0.7.0 pypi_0 pypi
pure-eval 0.2.2 pypi_0 pypi
pyasn1 0.5.0 pypi_0 pypi
pyasn1-modules 0.3.0 pypi_0 pypi
pygments 2.16.1 pypi_0 pypi
pyparsing 3.1.1 pypi_0 pypi
python 3.9.18 h955ad1f_0
python-dateutil 2.8.2 pypi_0 pypi
pytigre 2.2.0 dev_0 <develop>
pywavelets 1.4.1 pypi_0 pypi
pyyaml 6.0 pypi_0 pypi
readline 8.2 h5eee18b_0
requests 2.31.0 pypi_0 pypi
requests-oauthlib 1.3.1 pypi_0 pypi
rsa 4.9 pypi_0 pypi
scikit-image 0.19.3 pypi_0 pypi
scipy 1.10.0 pypi_0 pypi
setuptools 68.0.0 py39h06a4308_0
six 1.16.0 pypi_0 pypi
sqlite 3.41.2 h5eee18b_0
stack-data 0.6.3 pypi_0 pypi
tensorboard 2.12.0 pypi_0 pypi
tensorboard-data-server 0.7.1 pypi_0 pypi
tensorboard-plugin-wit 1.8.1 pypi_0 pypi
tifffile 2023.9.26 pypi_0 pypi
tk 8.6.12 h1ccaba5_0
tomli 2.0.1 pypi_0 pypi
torch 1.11.0+cu113 pypi_0 pypi
torchaudio 0.11.0+cu113 pypi_0 pypi
torchvision 0.12.0+cu113 pypi_0 pypi
tqdm 4.64.1 pypi_0 pypi
traitlets 5.11.2 pypi_0 pypi
typing-extensions 4.8.0 pypi_0 pypi
tzdata 2023c h04d1e81_0
urllib3 2.0.6 pypi_0 pypi
wcwidth 0.2.8 pypi_0 pypi
werkzeug 3.0.0 pypi_0 pypi
wheel 0.41.2 py39h06a4308_0
xz 5.4.2 h5eee18b_0
zipp 3.17.0 pypi_0 pypi
zlib 1.2.13 h5eee18b_0 A brief summary, in my 2 conda environments:
With TIGRE 2.5, pytigre==2.4.0, cython==3.0.2, numpy==1.26.0
With TIGRE 2.3, pytigre==2.2.0, cython==0.29.25, numpy==1.21.6
tsadakane
commented
12 months ago @AnderBiguri , @JiahaoPlus
I made two fresh virtual environments of conda
test496_39
and built tigre (4598aaa32a9aa3ae31aef45f5fe2f5daa01854ea) by
pip install .In both environments, the program given at https://github.com/CERN/TIGRE/issues/496#issuecomment-1751887699 worked with the given data at the fist comment.
My PC's OS is Windows 10 + CUDA 11.8 and its GPUs are
0: NVIDIA GeForce RTX 2080 Ti
1: NVIDIA GeForce RTX 2080 Ti
2: NVIDIA GeForce GTX 1070so I used 0 and 1 by modifying the program given at https://github.com/CERN/TIGRE/issues/496#issuecomment-1751887699 by adding the following lines
listGpuNames = gpu.getGpuNames()
if len(listGpuNames) == 0:
print("Error: No gpu found")
else:
for id in range(len(listGpuNames)):
print("{}: {}".format(id, listGpuNames[id]))
gpuids = gpu.getGpuIds(listGpuNames[0])
print(gpuids)
....
imgSART = algs.sart(train_projs, geo, train_angles, niter=4, gpuids=gpuids) # add gpuids optionI also added lines for plots so that I can check the reconstructed volume.
The result of
conda list > conda_list.txtin the virtual env. test496_39 after building tigre is as follows:
# packages in environment at C:\opt\CondaEnvs\test496_39:
#
# Name Version Build Channel
bzip2 1.0.8 h8ffe710_4 conda-forge
ca-certificates 2023.7.22 h56e8100_0 conda-forge
colorama 0.4.6 pypi_0 pypi
contourpy 1.1.1 pypi_0 pypi
cycler 0.12.1 pypi_0 pypi
cython 3.0.3 pypi_0 pypi
fonttools 4.43.1 pypi_0 pypi
importlib-resources 6.1.0 pypi_0 pypi
kiwisolver 1.4.5 pypi_0 pypi
libffi 3.4.2 h8ffe710_5 conda-forge
libsqlite 3.43.0 hcfcfb64_0 conda-forge
libzlib 1.2.13 hcfcfb64_5 conda-forge
matplotlib 3.8.0 pypi_0 pypi
numpy 1.26.0 pypi_0 pypi
openssl 3.1.3 hcfcfb64_0 conda-forge
packaging 23.2 pypi_0 pypi
pillow 10.0.1 pypi_0 pypi
pip 23.2.1 pyhd8ed1ab_0 conda-forge
pyparsing 3.1.1 pypi_0 pypi
python 3.9.18 h4de0772_0_cpython conda-forge
python-dateutil 2.8.2 pypi_0 pypi
pytigre 2.4.0 pypi_0 pypi
scipy 1.11.3 pypi_0 pypi
setuptools 68.2.2 pyhd8ed1ab_0 conda-forge
six 1.16.0 pypi_0 pypi
tk 8.6.13 hcfcfb64_0 conda-forge
tzdata 2023c h71feb2d_0 conda-forge
ucrt 10.0.22621.0 h57928b3_0 conda-forge
vc 14.3 h64f974e_17 conda-forge
vc14_runtime 14.36.32532 hdcecf7f_17 conda-forge
vs2015_runtime 14.36.32532 h05e6639_17 conda-forge
wheel 0.41.2 pyhd8ed1ab_0 conda-forge
xz 5.2.6 h8d14728_0 conda-forge
zipp 3.17.0 pypi_0 pypi
Thanks a lot @tsadakane. There is certainly somewhere something off and I really can't pinpoint it.
Behavior
When I ran iterative methods like SART on a CT scan (512×512×463, can be downloaded here), it was stuck in "set_w" (w=Ax(...)) when initializing IterativeReconAlg class (link to the code). Do you know the possible reason and how to fix it?
Smaller images (like 256×256×256) work well. Running FDK on the 512×512×463 CT also works well.
Code to reproduce the problem (If applicable)
where "projections, geo, angles" can be obtained from the CT pickle file I attached above.
Specifications
(Data reference: Zha et. al, NAF)
Thanks!