search

ENHANCE-PET / FALCON

FALCON is a Python-based software application designed to facilitate PET motion correction, both for head and total-body scans. Our program is built around the fast 'greedy' registration toolkit, which serves as the registration engine. With FALCON, users can enjoy a streamlined experience for implementing motion correction.
GNU General Public License v3.0
36 stars 6 forks source link

Installation without SUDO? #43

Open rullator opened 1 year ago

rullator commented 1 year ago

Hi,

any chance to get FALCON running without root access? Installation via a local user anaconda environment? Or alternatively: do you expect FALCON to run in a virtual machine as well?

Best

LalithShiyam commented 1 year ago

Hi there, as mentioned in the readme section, if you have read/write access ofcourse you don't need to use sudo. I haven't run it in a virtual machine, but I don't see an issue. I am working to convert falcon as a pip package, so it should work with anaconda environment as well.

Please let me know if the virtual machine option works for you. The pip release will take some time and can't cater to that need right away.

LalithShiyam commented 1 year ago

Hi @rullator, did it work with virtual machines?

LalithShiyam commented 1 year ago

Hi @christanmod, many thanks for reporting the issue! May I know the image format that you are working on?

rullator commented 1 year ago

Hi @rullator, did it work with virtual machines?

I only had an old Ubuntu VM (16.04) available, which does not natively support Python 3.8. So I could not get FALCON to run. Installing a later version is possible in principle, but mainly a matter of resources (time). I have another unix server available, where I do not have root access to. So FALCON does not run there, either.

I just wanted to try the rigid motion correction for brain data at the moment. I assume FALCON would be working just fine as other programs, which I currently use (FSL, SPM).

I would come back to FALCON, when I have dynamic whole body PET data, which needs to get motion corrected. Maybe in that time, FALCON is also available as a pip package.

LalithShiyam commented 1 year ago

Hi @rullator sorry about the hassle. Sure, I have started working on the pip package. The testing part is the one that takes time. I will keep you posted.

LalithShiyam commented 1 year ago

Hi @christanmod: phew that was a lot 😄! Let me try to answer things one by one.

Do let me know regarding pointer 2 because that's the most important point. If nothing works, I am happy to have a look at your data or help you put further by having a short call.

Cheers!

LalithShiyam commented 1 year ago

Excellent. Now I understand.

This should work. If not let me know again :) and definitely let me know if it works ;)!

LalithShiyam commented 1 year ago

Strange. Where did you find the resampled image? Did you check out 4d-moco.nii.gz? If it’s not working, send me the nifti, and I can have a look - if it’s ok for you.

The reason is I don’t know what the data is and how it looks like. Is it PT, same tracer, different tracer. Really hard to tell. Would make my life easier to help you if I can see the nifti files.

LalithShiyam commented 1 year ago

@christanmod: Please let me know if the problem persists. I am closing it for now.

LalithShiyam commented 1 year ago

Hi there @christanmod, Great, now I understand. Now it's clear why we get subpar results. FALCON doesn't work when you try to register 2 different tracers. Unfortunately, it was not built for that. FALCON is a motion correction tool and it does exactly that. Aligning two different tracers is a complex issue altogether and not trivial to solve. We are currently working on that in Vienna. Hit me up if you want to collaborate in a multi center study: Lalith.shiyamsundar@meduniwien.ac.at 😄

LalithShiyam commented 9 months ago

@rullator : falconv2 is out, I think it might cater to your needs!

rullator commented 8 months ago

Yes, thanks! Installation works fine without root privileges :)

LalithShiyam commented 8 months ago

Great @rullator, let me know what you think about falconv2. Cheers!

rullator commented 8 months ago

It seems like a nice and really useful tool! Some things I did observed so far.

  1. When giving NIfTI files as input (data type 4 "int16"), the output is given as data type 16 "float32", which doubles the data size. Usually that might not be a big issue, but working with dynamic data from a LFOV camera it can become quite challenging ;)

  2. I tried to run it on a HPC cluster, but multi-threading seems to be a problem with FALCON. The tool seems to finish before it is really done with the motion correction resulting in only a single (sometimes two) moco_* files (out of 38) and no error in the log files. So far, I only observed the problem, when Number of motion correction done in parallel greater 1. Locally, with Number of motion correction done in parallel: 1 it works, but is, of course, quite time-consuming.

LalithShiyam commented 8 months ago

I am interested in the second issue @rullator, the jobs are decided based on the available hardware. Not sure why on a hpc cluster why there is just 1 job. Can you elaborate how you make it work on a cluster?

rullator commented 8 months ago

Jobs at our cluster are managed by SLURM. So, I create a job file which roughly looks like this:

`#!/bin/bash

SBATCH --job-name=FALCON_TEST

SBATCH --output=falcon-test-job-out.%J

SBATCH --time=8:00:00

SBATCH --mem=512G

SBATCH --ntasks=1

SBATCH --partition=COMPUTERNAME

SBATCH --gres=gpu:a30:4

SBATCH --mail-type=END

cd /[...]/FALCON/ source falconz_env/bin/activate falconz -d ./Data/dynPET/ -r deformable -rf 5 -sf 0`

That job file is handed over to SLURM (calling sbatch) which manages all computing jobs. When finished I got three log files, one contains the bash output und two created by FALCON.

Bash output log file: `

_ __/ | / ___/ __ | / / | \ \ /_ /| | / / / / / |/ / __/ / / / / / | /_/ / / // / /| / // // / // // |/_/__/ __/ // |/ /__/(_)/ A part of the ENHANCE community. Join us at https://enhance.pet to build the future of PET imaging together.

Shiyam Sundar, L. K., Lassen, M. L., Muzik, O. (2023, June 8). Fully automated, fast motion correction of dynamic whole-body and total-body PET/CT Imaging Studies. Journal of Nuclear Medicine. https://jnm.snmjournals.org/content/early/2023/06/08/jnumed.122.265362 Copyright 2022, Quantitative Imaging and Medical Physics Team, Medical University of Vienna

📝 NOTE:

Warning: Only 4D images will be considered in the analysis. Registration type: deformable | Shrink level: [2, 4, 8] | Multi-resolution scheme: 100x25x10 | Reference frame: 28 Available memory: 996 GB | Available threads: 128 | Number of motion correction done in parallel: 16

🌐 BINARIES DOWNLOAD:

Detected system: linux | Detected architecture: x86_64 A local instance of falcon-linux-x86_64 binaries has been detected.

🔍 STANDARDIZING INPUT DATA TO NIFTI:

Standardization complete • Elapsed time: 0:00:21

🦅 PERFORMING MOTION CORRECTION:

Number of files to motion correct: 28 | Reference file: s0004546898-0007-02001-012213_01.nii | Start frame: 0 Aligned moving imag… 10… [28/… • Time elapse… • CPU Load: 0… • Memory Load: 1.… Motion correction complete: Results in Data/FALCONZ-V02-2023-10-19-14-53-18/Motion-corrected-images | 4D MoCo file: moco_4D.nii.gz`

1st log file by FALCON: 2023-10-19 14:53:25,572 INFO [input_validation.py:43] Input validation successful. 2023-10-19 14:53:25,575 INFO [falconz.py:120] ---------------------------------------------------------------------------------------------------- 2023-10-19 14:53:25,575 INFO [falconz.py:121] STARTING FALCON-Z V.2.0.0 2023-10-19 14:53:25,575 INFO [falconz.py:122] ---------------------------------------------------------------------------------------------------- 2023-10-19 14:53:25,577 WARNING [display.py:70] Only 4D images will be considered in the analysis. 2023-10-19 14:53:25,578 INFO [display.py:119] Available memory: 996 GB | Available threads: 128 | Number of motion correction done in parallel: 16 2023-10-19 14:53:25,578 INFO [download.py:113] A local instance of falcon-linux-x86_64 has been detected. 2023-10-19 14:53:25,578 INFO [falconz.py:162] 2023-10-19 14:53:25,578 INFO [falconz.py:163] STANDARDIZING INPUT DATA TO NIFTI: 2023-10-19 14:53:25,578 INFO [falconz.py:164] 2023-10-19 14:53:47,237 INFO [falconz.py:189] Standardization complete. 2023-10-19 14:53:47,239 INFO [falconz.py:225] 2023-10-19 14:53:47,239 INFO [falconz.py:226] PERFORMING MOTION CORRECTION: 2023-10-19 14:53:47,239 INFO [falconz.py:227] 2023-10-19 14:54:07,797 INFO [image_conversion.py:237] Merging 3D nifti files in Data/FALCONZ-V02-2023-10-19-14-53-18/Motion-corrected-images with wildcard *moco_* 2023-10-19 14:54:10,514 INFO [image_conversion.py:242] Done

2nd logfile by FALCON: 2023-10-19 14:54:03,607 INFO [image_processing.py:127] Affine alignment: s0004546898-0007-02001-006372_12.nii -> s0004546898-0007-02001-012213_01.nii | Aligned image: moco-s0004546898-0007-02001-006372_12.nii | Transform file: s0004546898-0007-02001-006372_12.nii_affine.mat 2023-10-19 14:54:03,802 INFO [image_processing.py:147] Deformable alignment: s0004546898-0007-02001-006372_12.nii -> s0004546898-0007-02001-012213_01.nii | Aligned image: moco-s0004546898-0007-02001-006372_12.nii | Initial alignment: s0004546898-0007-02001-006372_12.nii_affine.mat | warp file: s0004546898-0007-02001-006372_12.nii_warp.nii.gz 2023-10-19 14:54:04,112 INFO [image_processing.py:127] Affine alignment: s0004546898-0007-02001-002124_20.nii -> s0004546898-0007-02001-012213_01.nii | Aligned image: moco-s0004546898-0007-02001-002124_20.nii | Transform file: s0004546898-0007-02001-002124_20.nii_affine.mat 2023-10-19 14:54:04,292 INFO [image_processing.py:147] Deformable alignment: s0004546898-0007-02001-009558_06.nii -> s0004546898-0007-02001-012213_01.nii | Aligned image: moco-s0004546898-0007-02001-009558_06.nii | Initial alignment: s0004546898-0007-02001-009558_06.nii_affine.mat | warp file: s0004546898-0007-02001-009558_06.nii_warp.nii.gz

LalithShiyam commented 8 months ago

Phew. I have never run it in a slurm based hpc. From the logs, It has 16 jobs/moco in parallel. So that's good. I think it has troubles in running/executing the binaries, if the motion correction is super quick. Then it is quite possible that the binaries are not being executed properly... might be that binary execution is not the same on a hpc. I am thinking on how to reproduce this here. So that I can suggest a fix. Sorry for being useless. Let me think more about this.

rullator commented 8 months ago

Thanks!

PS: I now tried it using DICOM files (Siemens Quadra, 23 frames, each with 513 slices) as inputs and ran into the following error: 2023-10-20 09:16:52,871 ERROR [falconz.py:194] Standardization failed. 2023-10-20 09:16:52,871 ERROR [falconz.py:195] Error during DICOM to NIFTI conversion using dcm2niix.

Do you have a debug mode offering more log information?

LalithShiyam commented 8 months ago

Where did this error show up? Is it in the HPC? Or regular workstation?

rullator commented 8 months ago

Sorry for being unprecise: Cluster

Keyn34 commented 8 months ago

Hi @rullator, how is the execution of binaries carried out via slurm? I guess the permission situation is very different on a regular User/PC than on an HPC cluster with scheduled tasks with slurm (if that makes sense). Did you notice issues like this before? It really looks like that the binaries are simply not executed.

LalithShiyam commented 8 months ago

@rullator I think it's good to add to see if the binaries are found or executable ready (due to permissions), I will write a patch to check this and also write an elaborate log and keep you posted. Would you be kind enough to test it once the patch is done? I can push the errors to the logs, if the exec is not working well.

rullator commented 8 months ago

Hi @rullator, how is the execution of binaries carried out via slurm? I guess the permission situation is very different on a regular User/PC than on an HPC cluster with scheduled tasks with slurm (if that makes sense). Did you notice issues like this before? It really looks like that the binaries are simply not executed.

Sorry, I am not really sure, how the binaries are really carried out. It is actually the first time I had tried FALCON. So the problem is not "new" for me, since I had not used FALCON V1.

rullator commented 8 months ago

@rullator I think it's good to add to see if the binaries are found or executable ready (due to permissions), I will write a patch to check this and also write an elaborate log and keep you posted. Would you be kind enough to test it once the patch is done? I can push the errors to the logs, if the exec is not working well.

Yes, I'm happy to test and help!

LalithShiyam commented 8 months ago

Hi @rullator, I am truly sorry about the delay. I just came back to my desk this week after a long absence and I had to plough through quite a bit of backlogs. I have added some logging for the set_permissions function in file_utilities.py, to catch if there are permission errors in your slurm cluster. Hopefully this can help us to identify the real issue. Would you be kind enough to try out falcon on a new dataset or the previous dataset cleaned?

Please update falconz to the latest version using pip install --upgrade falconz and lets see what shows up in the log files.

rullator commented 8 months ago

No worries about the delay!!

Here is the output:

_ __/ | / ___/ __ | / / | \ \ /_ /| | / / / / / |/ / __/ / / / / / | /_/ / / // / /| / // // / // // |/_/__/ __/ // |/ /__/(_)/ A part of the ENHANCE community. Join us at https://enhance.pet to build the future of PET imaging together.

Shiyam Sundar, L. K., Lassen, M. L., Muzik, O. (2023, June 8). Fully automated, fast motion correction of dynamic whole-body and total-body PET/CT Imaging Studies. Journal of Nuclear Medicine. https://jnm.snmjournals.org/content/early/2023/06/08/jnumed.122.265362 Copyright 2022, Quantitative Imaging and Medical Physics Team, Medical University of Vienna

📝 NOTE:

Warning: Only 4D images will be considered in the analysis. Registration type: deformable | Shrink level: [2, 4, 8] | Multi-resolution scheme: 100x25x10 | Reference frame: 8 Available memory: 985 GB | Available threads: 128 | Number of motion correction done in parallel: 16

🌐 BINARIES DOWNLOAD:

Detected system: linux | Detected architecture: x86_64 A local instance of falcon-linux-x86_64 binaries has been detected.

🔍 STANDARDIZING INPUT DATA TO NIFTI:

Standardization complete • Elapsed time: 0:02:39

🦅 PERFORMING MOTION CORRECTION:

Number of files to motion correct: 30 | Reference file: s0006296986-0007-02001-004779.nii | Start frame: 0 Aligned moving imag… 10… [30/… • Time elapse… • CPU Load: 0… • Memory Load: 2.… 2023-10-27 15:56:49,119 - distributed.worker.state_machine - WARNING - Async instruction for <Task cancelled name='execute(align_single_image-424276a5-cfce-4a90-9d5c-4ae2921f4e8e)' coro=<Worker.execute() done, defined at /home/[...]/FALCON/falconz_env/lib/python3.9/site-packages/distributed/worker_state_machine.py:3609>> ended with CancelledError Motion correction complete: Results in Data/FALCONZ-V02-2023-10-27-15-53-45/Motion-corrected-images | 4D MoCo file: moco_4D.nii.gz

Two log files were generated:

2023-10-27 15:53:51,441 INFO [input_validation.py:43] Input validation successful. 2023-10-27 15:53:51,445 INFO [falconz.py:120] ---------------------------------------------------------------------------------------------------- 2023-10-27 15:53:51,445 INFO [falconz.py:121] STARTING FALCON-Z V.2.0.0
2023-10-27 15:53:51,445 INFO [falconz.py:122] ---------------------------------------------------------------------------------------------------- 2023-10-27 15:53:51,447 WARNING [display.py:70] Only 4D images will be considered in the analysis. 2023-10-27 15:53:51,447 INFO [display.py:119] Available memory: 985 GB | Available threads: 128 | Number of motion correction done in parallel: 16 2023-10-27 15:53:51,447 INFO [download.py:113] A local instance of falcon-linux-x86_64 has been detected. 2023-10-27 15:53:51,448 INFO [falconz.py:162]
2023-10-27 15:53:51,448 INFO [falconz.py:163] STANDARDIZING INPUT DATA TO NIFTI: 2023-10-27 15:53:51,448 INFO [falconz.py:164]
2023-10-27 15:56:30,824 INFO [falconz.py:189] Standardization complete. 2023-10-27 15:56:30,829 INFO [falconz.py:225]
2023-10-27 15:56:30,829 INFO [falconz.py:226] PERFORMING MOTION CORRECTION: 2023-10-27 15:56:30,829 INFO [falconz.py:227]
2023-10-27 15:56:52,735 INFO [imageconversion.py:237] Merging 3D nifti files in Data/FALCONZ-V02-2023-10-27-15-53-45/Motion-corrected-images with wildcard *moco* 2023-10-27 15:56:56,392 INFO [image_conversion.py:242] Done

2023-10-27 15:56:48,520 INFO [image_processing.py:127] Affine alignment: s0006296986-0007-02001-008496.nii -> s0006296986-0007-02001-004779.nii | Aligned image: moco-s0006296986-0007-02001-008496.nii | Transform file: s0006296986-0007-02001-008496.nii_affine.mat 2023-10-27 15:56:48,599 INFO [image_processing.py:147] Deformable alignment: s0006296986-0007-02001-008496.nii -> s0006296986-0007-02001-004779.nii | Aligned image: moco-s0006296986-0007-02001-008496.nii | Initial alignment: s0006296986-0007-02001-008496.nii_affine.mat | warp file: s0006296986-0007-02001-008496.nii_warp.nii.gz 2023-10-27 15:56:48,733 INFO [image_processing.py:127] Affine alignment: s0006296986-0007-02001-011682.nii -> s0006296986-0007-02001-004779.nii | Aligned image: moco-s0006296986-0007-02001-011682.nii | Transform file: s0006296986-0007-02001-011682.nii_affine.mat 2023-10-27 15:56:48,777 INFO [image_processing.py:147] Deformable alignment: s0006296986-0007-02001-011682.nii -> s0006296986-0007-02001-004779.nii | Aligned image: moco-s0006296986-0007-02001-011682.nii | Initial alignment: s0006296986-0007-02001-011682.nii_affine.mat | warp file: s0006296986-0007-02001-011682.nii_warp.nii.gz 2023-10-27 15:56:49,186 INFO [image_processing.py:127] Affine alignment: s0006296986-0007-02001-001062.nii -> s0006296986-0007-02001-004779.nii | Aligned image: moco-s0006296986-0007-02001-001062.nii | Transform file: s0006296986-0007-02001-001062.nii_affine.mat 2023-10-27 15:56:49,246 INFO [image_processing.py:147] Deformable alignment: s0006296986-0007-02001-001062.nii -> s0006296986-0007-02001-004779.nii | Aligned image: moco-s0006296986-0007-02001-001062.nii | Initial alignment: s0006296986-0007-02001-001062.nii_affine.mat | warp file: s0006296986-0007-02001-001062.nii_warp.nii.gz

PS: I used the "quote" markup, since the "code" markup messes up the output..

LalithShiyam commented 8 months ago

Hi @rullator, based on the log files, dask struggles in executing the function.

2023-10-27 15:56:49,119 - distributed.worker.state_machine - WARNING - Async instruction for <Task cancelled name='execute(align_single_image-424276a5-cfce-4a90-9d5c-4ae2921f4e8e)' coro=<Worker.execute() done, defined at /home/[...]/FALCON/falconz_env/lib/python3.9/site-packages/distributed/worker_state_machine.py:3609>> ended with CancelledError

Let me think on how to reproduce this error...

andersenthomas commented 3 months ago

How did you manage to install without sudo? I've installed with pip but running falconz tries to write to /usr/bin/falcon-[os].zip which is not possible.

LalithShiyam commented 3 months ago

Hi @andersenthomas, Many thanks for reaching out. Would you be kind enough to tell me how your current set up is? Is it a cluster? Is it a workstation? Is it inside a new python environment? OS? The answers would help me assist you better.

Cheers, Lalith

andersenthomas commented 3 months ago

Hi,

Thanks for the fast response. I'm using a Linux-based server without virtual environment but instead user controlled. I've managed to get it to run using a virtual environement but it does not seem to work directly from the users home.

LalithShiyam commented 3 months ago

This is expected as it involves permission issues. I would strongly recommend virtual environment. Is there any reason why you don't prefer virtual environment?