search

SBC-Utrecht / PyTomPrivate

Development repository
GNU General Public License v2.0
2 stars 2 forks source link

[Draft] fixes e2e tests #123

Closed sroet closed 1 year ago

sroet commented 1 year ago

This PR will fix all the e2e test on a machine without a gpu

McHaillet commented 1 year ago

Hi! This is the full output from running python -m unittest discover from the tests/E2ETests on my laptop with GPU. Many errors sadly.

I made it foldable because its long, also the formatting is weird due to # in the output.

Click me (pytom_env) [private]:~/coding/pytom/tests/E2ETests$ python -m unittest discover EEEdirectory ./testparticles existed already - using this one mpirun -np 2 /home/marten/coding/pytom/pytom/bin/pytom /home/marten/coding/pytom/pytom/classification/auto_focus_classify.py -p pl.xml -k 2 -f 10 -b 4 -i 2 PyTom 1.0 This license affects the software package PyTom and all the herein distributed source / data files. Authors: Marten Chaillet Gijs van der Schot Ilja Gubins Mihajlo Vanevic Thomas Hrabe Yuxiang Chen Friedrich Foerster Copyright (c) 2021 Utrecht University http://www.pytom.org This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation version 2 of the License. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. The complete license can be obtained from http://www.gnu.org/licenses/gpl-2.0.html. PyTom 1.0 This license affects the software package PyTom and all the herein distributed source / data files. Authors: Marten Chaillet Gijs van der Schot Ilja Gubins Mihajlo Vanevic Thomas Hrabe Yuxiang Chen Friedrich Foerster Copyright (c) 2021 Utrecht University http://www.pytom.org This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation version 2 of the License. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. The complete license can be obtained from http://www.gnu.org/licenses/gpl-2.0.html. ---------------------------------------------------- -- Input parameters -------------------------------- ---------------------------------------------------- Ncluster = 2 frequency = 10 offset = None binning = 4 mask = None fmask = None niteration = 2 dispersion = None threshold = 0.4 noise = None noalign = None outputdir = ./ noise = None ---------------------------------------------------- ---------------------------------------------------- -- Input parameters -------------------------------- ---------------------------------------------------- Ncluster = 2 frequency = 10 offset = None binning = 4 mask = None fmask = None niteration = 2 dispersion = None threshold = 0.4 noise = None noalign = None outputdir = ./ noise = None ---------------------------------------------------- Initializing the class centroids ... Initializing the class centroids ... cannot split odd / even. Likely you used only one processor - use: mpirun -np 2 (or higher!)?! cannot split odd / even. Likely you used only one processor - use: mpirun -np 2 (or higher!)?! Traceback (most recent call last): File "/home/marten/coding/pytom/pytom/classification/auto_focus_classify.py", line 1069, in classify(pl, settings) File "/home/marten/coding/pytom/pytom/classification/auto_focus_classify.py", line 858, in classify references, frequencies = initialize(pl, settings) File "/home/marten/coding/pytom/pytom/classification/auto_focus_classify.py", line 778, in initialize res, tmp, tmp2 = calculate_averages(pp, settings["binning"], None, outdir=settings["output_directory"]) File "/home/marten/coding/pytom/pytom/classification/auto_focus_classify.py", line 413, in calculate_averages raise ParameterError('cannot split odd / even. Likely you used only one processor - use: mpirun -np 2 (or higher!)?!') pytom.basic.exceptions.ParameterError: During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/marten/coding/pytom/pytom/classification/auto_focus_classify.py", line 1071, in print(e) TypeError: __str__ returned non-string (type NoneType) cannot split odd / even. Likely you used only one processor - use: mpirun -np 2 (or higher!)?! cannot split odd / even. Likely you used only one processor - use: mpirun -np 2 (or higher!)?! Traceback (most recent call last): File "/home/marten/coding/pytom/pytom/classification/auto_focus_classify.py", line 1069, in classify(pl, settings) File "/home/marten/coding/pytom/pytom/classification/auto_focus_classify.py", line 858, in classify references, frequencies = initialize(pl, settings) File "/home/marten/coding/pytom/pytom/classification/auto_focus_classify.py", line 778, in initialize res, tmp, tmp2 = calculate_averages(pp, settings["binning"], None, outdir=settings["output_directory"]) File "/home/marten/coding/pytom/pytom/classification/auto_focus_classify.py", line 413, in calculate_averages raise ParameterError('cannot split odd / even. Likely you used only one processor - use: mpirun -np 2 (or higher!)?!') pytom.basic.exceptions.ParameterError: During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/marten/coding/pytom/pytom/classification/auto_focus_classify.py", line 1071, in print(e) TypeError: __str__ returned non-string (type NoneType) -------------------------------------------------------------------------- Primary job terminated normally, but 1 process returned a non-zero exit code. Per user-direction, the job has been aborted. -------------------------------------------------------------------------- -------------------------------------------------------------------------- mpirun detected that one or more processes exited with non-zero status, thus causing the job to be terminated. The first process to do so was: Process name: [[35147,1],0] Exit code: 1 -------------------------------------------------------------------------- Fdirectory ./testparticles existed already - using this one testing masks mpirun -np 2 /home/marten/coding/pytom/pytom/bin/pytom /home/marten/coding/pytom/pytom/classification/auto_focus_classify.py -p pl.xml -k 2 -f 10 -b 4 -i 2 -m ../testData/ribo_mask.em -c ../testData/focussed_classification_mask_ribo.mrc PyTom 1.0 This license affects the software package PyTom and all the herein distributed source / data files. Authors: Marten Chaillet Gijs van der Schot Ilja Gubins Mihajlo Vanevic Thomas Hrabe Yuxiang Chen Friedrich Foerster Copyright (c) 2021 Utrecht University http://www.pytom.org This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation version 2 of the License. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. The complete license can be obtained from http://www.gnu.org/licenses/gpl-2.0.html. PyTom 1.0 This license affects the software package PyTom and all the herein distributed source / data files. Authors: Marten Chaillet Gijs van der Schot Ilja Gubins Mihajlo Vanevic Thomas Hrabe Yuxiang Chen Friedrich Foerster Copyright (c) 2021 Utrecht University http://www.pytom.org This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation version 2 of the License. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. The complete license can be obtained from http://www.gnu.org/licenses/gpl-2.0.html. ---------------------------------------------------- -- Input parameters -------------------------------- ---------------------------------------------------- Ncluster = 2 frequency = 10 offset = None binning = 4 mask = ../testData/ribo_mask.em fmask = ../testData/focussed_classification_mask_ribo.mrc niteration = 2 dispersion = None threshold = 0.4 noise = None noalign = None outputdir = ./ noise = None ---------------------------------------------------- -- Input parameters -------------------------------- ---------------------------------------------------- Ncluster = 2 frequency = 10 offset = None ---------------------------------------------------- binning = 4 mask = ../testData/ribo_mask.em fmask = ../testData/focussed_classification_mask_ribo.mrc niteration = 2 dispersion = None threshold = 0.4 noise = None noalign = None outputdir = ./ noise = None ---------------------------------------------------- Initializing the class centroids ... Initializing the class centroids ... cannot split odd / even. Likely you used only one processor - use: mpirun -np 2 (or higher!)?! cannot split odd / even. Likely you used only one processor - use: mpirun -np 2 (or higher!)?! Traceback (most recent call last): File "/home/marten/coding/pytom/pytom/classification/auto_focus_classify.py", line 1069, in classify(pl, settings) File "/home/marten/coding/pytom/pytom/classification/auto_focus_classify.py", line 858, in classify references, frequencies = initialize(pl, settings) File "/home/marten/coding/pytom/pytom/classification/auto_focus_classify.py", line 778, in initialize res, tmp, tmp2 = calculate_averages(pp, settings["binning"], None, outdir=settings["output_directory"]) File "/home/marten/coding/pytom/pytom/classification/auto_focus_classify.py", line 413, in calculate_averages raise ParameterError('cannot split odd / even. Likely you used only one processor - use: mpirun -np 2 (or higher!)?!') pytom.basic.exceptions.ParameterError: During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/marten/coding/pytom/pytom/classification/auto_focus_classify.py", line 1071, in print(e) TypeError: __str__ returned non-string (type NoneType) cannot split odd / even. Likely you used only one processor - use: mpirun -np 2 (or higher!)?! cannot split odd / even. Likely you used only one processor - use: mpirun -np 2 (or higher!)?! Traceback (most recent call last): File "/home/marten/coding/pytom/pytom/classification/auto_focus_classify.py", line 1069, in classify(pl, settings) File "/home/marten/coding/pytom/pytom/classification/auto_focus_classify.py", line 858, in classify references, frequencies = initialize(pl, settings) File "/home/marten/coding/pytom/pytom/classification/auto_focus_classify.py", line 778, in initialize res, tmp, tmp2 = calculate_averages(pp, settings["binning"], None, outdir=settings["output_directory"]) File "/home/marten/coding/pytom/pytom/classification/auto_focus_classify.py", line 413, in calculate_averages raise ParameterError('cannot split odd / even. Likely you used only one processor - use: mpirun -np 2 (or higher!)?!') pytom.basic.exceptions.ParameterError: During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/marten/coding/pytom/pytom/classification/auto_focus_classify.py", line 1071, in print(e) TypeError: __str__ returned non-string (type NoneType) -------------------------------------------------------------------------- Primary job terminated normally, but 1 process returned a non-zero exit code. Per user-direction, the job has been aborted. -------------------------------------------------------------------------- -------------------------------------------------------------------------- mpirun detected that one or more processes exited with non-zero status, thus causing the job to be terminated. The first process to do so was: Process name: [[35120,1],1] Exit code: 1 -------------------------------------------------------------------------- Fdirectory ./testparticles existed already - using this one .directory ./testparticles existed already - using this one PyTom 1.0 This license affects the software package PyTom and all the herein distributed source / data files. Authors: Marten Chaillet Gijs van der Schot Ilja Gubins Mihajlo Vanevic Thomas Hrabe Yuxiang Chen Friedrich Foerster Copyright (c) 2021 Utrecht University http://www.pytom.org This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation version 2 of the License. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. The complete license can be obtained from http://www.gnu.org/licenses/gpl-2.0.html. Fdirectory ./testparticles existed already - using this one mpirun -np 2 pytom /home/marten/coding/pytom/pytom/classification/calculate_correlation_matrix.py -p outputCCC/pl.xml -f 40 -m ./testData/mask_45.em -o outputCCC/ PyTom 1.0 This license affects the software package PyTom and all the herein distributed source / data files. Authors: Marten Chaillet Gijs van der Schot Ilja Gubins Mihajlo Vanevic Thomas Hrabe Yuxiang Chen Friedrich Foerster Copyright (c) 2021 Utrecht University http://www.pytom.org This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation version 2 of the License. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. The complete license can be obtained from http://www.gnu.org/licenses/gpl-2.0.html. PyTom 1.0 This license affects the software package PyTom and all the herein distributed source / data files. Authors: Marten Chaillet Gijs van der Schot Ilja Gubins Mihajlo Vanevic Thomas Hrabe Yuxiang Chen Friedrich Foerster Copyright (c) 2021 Utrecht University http://www.pytom.org This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation version 2 of the License. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. The complete license can be obtained from http://www.gnu.org/licenses/gpl-2.0.html. Traceback (most recent call last): File "/home/marten/coding/pytom/pytom/classification/calculate_correlation_matrix.py", line 497, in worker.start(job, verbose) File "/home/marten/coding/pytom/pytom/classification/calculate_correlation_matrix.py", line 288, in start self.run(verbose) File "/home/marten/coding/pytom/pytom/classification/calculate_correlation_matrix.py", line 340, in run mask = read(job["Mask"], 0, 0, 0, 0, 0, 0, 0, 0, 0, binning, binning, binning) File "/home/marten/miniconda3/envs/pytom_env/lib/python3.8/site-packages/pytom/lib/pytom_volume.py", line 458, in read return _pytom_volume.read(*args) RuntimeError: ./testData/mask_45.em: Wrong file format or file doesn't exist! -------------------------------------------------------------------------- Primary job terminated normally, but 1 process returned a non-zero exit code. Per user-direction, the job has been aborted. -------------------------------------------------------------------------- -------------------------------------------------------------------------- mpirun detected that one or more processes exited with non-zero status, thus causing the job to be terminated. The first process to do so was: Process name: [[35020,1],1] Exit code: 1 -------------------------------------------------------------------------- F/home/marten/miniconda3/envs/pytom_env/lib/python3.8/subprocess.py:946: ResourceWarning: subprocess 5925 is still running _warn("subprocess %s is still running" % self.pid, ResourceWarning: Enable tracemalloc to get the object allocation traceback /home/marten/coding/pytom/tests/E2ETests/test_ConversionIMOD2PyTom.py:47: ResourceWarning: unclosed file <_io.TextIOWrapper name=10 encoding='UTF-8'> if len(os.popen('which submfg').read()) < 5: ResourceWarning: Enable tracemalloc to get the object allocation traceback Edirectory ./testparticles existed already - using this one mpirun -n 1 GLocalJob.py -p pl.xml -m ../testData/ribo_mask.em --SphericalMask -b 1 -n 1 -d ./ -s flcf --pixelSize 2.0 --particleDiameter 250.0 -j ./myGLocal.xml PyTom 1.0 This license affects the software package PyTom and all the herein distributed source / data files. Authors: Marten Chaillet Gijs van der Schot Ilja Gubins Mihajlo Vanevic Thomas Hrabe Yuxiang Chen Friedrich Foerster Copyright (c) 2021 Utrecht University http://www.pytom.org This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation version 2 of the License. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. The complete license can be obtained from http://www.gnu.org/licenses/gpl-2.0.html. particleList = pl.xml reference = using average from particleList mask = ../testData/ribo_mask.em rotations = scoring function= FLCFScore symmetries = None destination = ./ numberIterations= 1 binning = 1 sampleInfo = weighting = False compound Wedge = False Calculate on GPU= [] StartPrior: running iteration 0/1 >>>>>>>>> MPI rank: 0, Iteration: 0 Particles averaged [ ############### Particles averaged [ ############################## Particles averaged [ ############################################# Particles averaged [ ######################################################### Particles averaged [ ######################################################### ################# ] 100% Particles averaged [ ############### Particles averaged [ ############################## Particles averaged [ ############################################# Particles averaged [ ######################################################### Particles averaged [ ######################################################### ################# ] 100% >>>>>>>>> averaging done ... took 5.79 seconds Optimization terminated successfully. Current function value: -0.421819 Iterations: 3 Function evaluations: 134 Alignment densities: Rotations: 15.930, 7.977, -0.620; Translations: -0.460, -0.410, -1.195 Orientation difference: 23.914 deg >>>>>>>>> even and odd averages aligned ... took 15.59 seconds >>>>>>>>> Iteration 0: Resolution = 26.87 A; angularIncrement= 0.60 deg. >>>>>>>>> Aligning Even .... >>>>>>>>> Aligning Odd .... >>>>>>>>> Average scores: even 0.834; odd 0.275 Particles averaged [ ############### Particles averaged [ ############################## Particles averaged [ ############################################# Particles averaged [ ######################################################### Particles averaged [ ######################################################### ################# ] 100% Particles averaged [ ############### Particles averaged [ ############################## Particles averaged [ ############################################# Particles averaged [ ######################################################### Particles averaged [ ######################################################### ################# ] 100% Optimization terminated successfully. Current function value: -0.390329 Iterations: 3 Function evaluations: 147 rotation between averages > increment .... applying rotation and shift to odd particles Particles averaged [ ############### Particles averaged [ ############################## Particles averaged [ ############################################# Particles averaged [ ######################################################### Particles averaged [ ######################################################### ################# ] 100% Particles averaged [ ######## Particles averaged [ ############### Particles averaged [ ###################### Particles averaged [ ############################## Particles averaged [ ###################################### Particles averaged [ ############################################# Particles averaged [ #################################################### Particles averaged [ ######################################################### Particles averaged [ ######################################################### Particles averaged [ ######################################################### ################# ] 100% >>>>>>>>>> Final Resolution = 26.55 A. 191.78092527389526 finished .directory ./testparticles existed already - using this one mpirun -n 1 GLocalJob.py -g 0 -p pl.xml -m ../testData/ribo_mask.em --SphericalMask -b 1 -n 1 -d ./ -s flcf --pixelSize 2.0 --particleDiameter 250.0 -j ./myGLocal.xml PyTom 1.0 This license affects the software package PyTom and all the herein distributed source / data files. Authors: Marten Chaillet Gijs van der Schot Ilja Gubins Mihajlo Vanevic Thomas Hrabe Yuxiang Chen Friedrich Foerster Copyright (c) 2021 Utrecht University http://www.pytom.org This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation version 2 of the License. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. The complete license can be obtained from http://www.gnu.org/licenses/gpl-2.0.html. particleList = pl.xml reference = using average from particleList mask = ../testData/ribo_mask.em rotations = scoring function= FLCFScore symmetries = None destination = ./ numberIterations= 1 binning = 1 sampleInfo = weighting = False compound Wedge = False Calculate on GPU= [0] StartPrior: running iteration 0/1 >>>>>>>>> MPI rank: 0, Iteration: 0 Averaging particles on gpu:0 for .//0-EvenFiltered.em. Averaging particles on gpu:0 for .//0-OddFiltered.em. >>>>>>>>> averaging done ... took 1.94 seconds Optimization terminated successfully. Current function value: -0.463025 Iterations: 3 Function evaluations: 124 Alignment densities: Rotations: 30.329, 1.764, -18.883; Translations: -0.506, 0.334, -1.290 Orientation difference: 37.110 deg >>>>>>>>> even and odd averages aligned ... took 1.02 seconds >>>>>>>>> Iteration 0: Resolution = 26.81 A; angularIncrement= 0.60 deg. >>>>>>>>> Aligning Even .... init completed on: gpu:0 >>>>>>>>> Aligning Odd .... init completed on: gpu:0 >>>>>>>>> Average scores: even 0.766; odd 0.204 Averaging particles on gpu:0 for .//average-Final-Even.em. Averaging particles on gpu:0 for .//average-Final-Odd.em. Optimization terminated successfully. Current function value: -0.463964 Iterations: 3 Function evaluations: 137 rotation between averages > increment .... applying rotation and shift to odd particles Averaging particles on gpu:0 for .//average-Final-Odd.em. Averaging particles on gpu:0 for .//average-Final.em. >>>>>>>>>> Final Resolution = 26.11 A. 11.496421337127686 finished .directory ./testparticles existed already - using this one mpirun -n 1 GLocalJob.py -p pl.xml -m ../testData/ribo_mask.em --SphericalMask -b 1 -n 1 -d ./ -s nxcf --pixelSize 2.0 --particleDiameter 250.0 -j ./myGLocal.xml PyTom 1.0 This license affects the software package PyTom and all the herein distributed source / data files. Authors: Marten Chaillet Gijs van der Schot Ilja Gubins Mihajlo Vanevic Thomas Hrabe Yuxiang Chen Friedrich Foerster Copyright (c) 2021 Utrecht University http://www.pytom.org This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation version 2 of the License. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. The complete license can be obtained from http://www.gnu.org/licenses/gpl-2.0.html. particleList = pl.xml reference = using average from particleList mask = ../testData/ribo_mask.em rotations = scoring function= nxcfScore symmetries = None destination = ./ numberIterations= 1 binning = 1 sampleInfo = weighting = False compound Wedge = False Calculate on GPU= [] StartPrior: running iteration 0/1 >>>>>>>>> MPI rank: 0, Iteration: 0 Particles averaged [ ############### Particles averaged [ ############################## Particles averaged [ ############################################# Particles averaged [ ######################################################### Particles averaged [ ######################################################### ################# ] 100% Particles averaged [ ############### Particles averaged [ ############################## Particles averaged [ ############################################# Particles averaged [ ######################################################### Particles averaged [ ######################################################### ################# ] 100% >>>>>>>>> averaging done ... took 5.95 seconds Optimization terminated successfully. Current function value: -0.449972 Iterations: 2 Function evaluations: 104 Alignment densities: Rotations: 35.477, -10.090, 0.003; Translations: -0.475, -0.236, -1.278 Orientation difference: 25.388 deg >>>>>>>>> even and odd averages aligned ... took 15.29 seconds >>>>>>>>> Iteration 0: Resolution = 26.85 A; angularIncrement= 0.60 deg. >>>>>>>>> Aligning Even .... >>>>>>>>> Aligning Odd .... >>>>>>>>> Average scores: even 0.810; odd 0.190 Particles averaged [ ############### Particles averaged [ ############################## Particles averaged [ ############################################# Particles averaged [ ######################################################### Particles averaged [ ######################################################### ################# ] 100% Particles averaged [ ############### Particles averaged [ ############################## Particles averaged [ ############################################# Particles averaged [ ######################################################### Particles averaged [ ######################################################### ################# ] 100% Optimization terminated successfully. Current function value: -0.400823 Iterations: 3 Function evaluations: 141 rotation between averages > increment .... applying rotation and shift to odd particles Particles averaged [ ############### Particles averaged [ ############################## Particles averaged [ ############################################# Particles averaged [ ######################################################### Particles averaged [ ######################################################### ################# ] 100% Particles averaged [ ######## Particles averaged [ ############### Particles averaged [ ###################### Particles averaged [ ############################## Particles averaged [ ###################################### Particles averaged [ ############################################# Particles averaged [ #################################################### Particles averaged [ ######################################################### Particles averaged [ ######################################################### Particles averaged [ ######################################################### ################# ] 100% >>>>>>>>>> Final Resolution = 26.55 A. 218.82038164138794 finished .directory ./testparticles existed already - using this one mpirun -n 1 GLocalJob.py -g 0 -p pl.xml -m ../testData/ribo_mask.em --SphericalMask -b 1 -n 1 -d ./ -s nxcf --pixelSize 2.0 --particleDiameter 250.0 -j ./myGLocal.xml PyTom 1.0 This license affects the software package PyTom and all the herein distributed source / data files. Authors: Marten Chaillet Gijs van der Schot Ilja Gubins Mihajlo Vanevic Thomas Hrabe Yuxiang Chen Friedrich Foerster Copyright (c) 2021 Utrecht University http://www.pytom.org This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation version 2 of the License. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. The complete license can be obtained from http://www.gnu.org/licenses/gpl-2.0.html. particleList = pl.xml reference = using average from particleList mask = ../testData/ribo_mask.em rotations = scoring function= nxcfScore symmetries = None destination = ./ numberIterations= 1 binning = 1 sampleInfo = weighting = False compound Wedge = False Calculate on GPU= [0] StartPrior: running iteration 0/1 >>>>>>>>> MPI rank: 0, Iteration: 0 Averaging particles on gpu:0 for .//0-EvenFiltered.em. Averaging particles on gpu:0 for .//0-OddFiltered.em. >>>>>>>>> averaging done ... took 1.11 seconds Optimization terminated successfully. Current function value: -0.459620 Iterations: 4 Function evaluations: 177 Alignment densities: Rotations: 29.152, 2.460, -20.368; Translations: -0.501, 0.374, -1.224 Orientation difference: 37.464 deg >>>>>>>>> even and odd averages aligned ... took 1.16 seconds >>>>>>>>> Iteration 0: Resolution = 26.84 A; angularIncrement= 0.60 deg. >>>>>>>>> Aligning Even .... init completed on: gpu:0 >>>>>>>>> Aligning Odd .... init completed on: gpu:0 >>>>>>>>> Average scores: even 0.780; odd 0.217 Averaging particles on gpu:0 for .//average-Final-Even.em. Averaging particles on gpu:0 for .//average-Final-Odd.em. Optimization terminated successfully. Current function value: -0.459468 Iterations: 2 Function evaluations: 91 rotation between averages > increment .... applying rotation and shift to odd particles Averaging particles on gpu:0 for .//average-Final-Odd.em. Averaging particles on gpu:0 for .//average-Final.em. >>>>>>>>>> Final Resolution = 26.26 A. 10.750269412994385 finished .E - Calculating IASA potential from ../testData/3j9m.cif Number of atoms to go over is 158384 Calculating atom 0. Calculating atom 5000. Calculating atom 10000. Calculating atom 15000. Calculating atom 20000. Calculating atom 25000. Calculating atom 30000. Calculating atom 35000. Calculating atom 40000. Calculating atom 45000. Calculating atom 50000. Calculating atom 55000. Calculating atom 60000. Calculating atom 65000. Calculating atom 70000. Calculating atom 75000. Calculating atom 80000. Calculating atom 85000. Calculating atom 90000. Calculating atom 95000. Calculating atom 100000. Calculating atom 105000. Calculating atom 110000. Calculating atom 115000. Calculating atom 120000. Calculating atom 125000. Calculating atom 130000. Calculating atom 135000. Calculating atom 140000. Calculating atom 145000. Calculating atom 150000. Calculating atom 155000. molecule absorption = 0.389 solvent absorption = 0.204 E ====================================================================== ERROR: test_alignment (test_AlignmentTest.pytom_AlignmentTest) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/marten/coding/pytom/tests/E2ETests/test_AlignmentTest.py", line 30, in setUp os.chdir('/data2/mchaillet') FileNotFoundError: [Errno 2] No such file or directory: '/data2/mchaillet' ====================================================================== ERROR: test_exact_filter (test_AlignmentTest.pytom_AlignmentTest) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/marten/coding/pytom/tests/E2ETests/test_AlignmentTest.py", line 30, in setUp os.chdir('/data2/mchaillet') FileNotFoundError: [Errno 2] No such file or directory: '/data2/mchaillet' ====================================================================== ERROR: test_reconstruction (test_AlignmentTest.pytom_AlignmentTest) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/marten/coding/pytom/tests/E2ETests/test_AlignmentTest.py", line 30, in setUp os.chdir('/data2/mchaillet') FileNotFoundError: [Errno 2] No such file or directory: '/data2/mchaillet' ====================================================================== ERROR: runTest (test_ConversionIMOD2PyTom.pytom_MyFunctionTest) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/marten/coding/pytom/tests/E2ETests/test_ConversionIMOD2PyTom.py", line 48, in setUp raise Exception('Cannot run unittest. IMOD not installed.') Exception: Cannot run unittest. IMOD not installed. ====================================================================== ERROR: test_GUIWidgets (unittest.loader._FailedTest) ---------------------------------------------------------------------- ImportError: Failed to import test module: test_GUIWidgets Traceback (most recent call last): File "/home/marten/miniconda3/envs/pytom_env/lib/python3.8/unittest/loader.py", line 436, in _find_test_path module = self._get_module_from_name(name) File "/home/marten/miniconda3/envs/pytom_env/lib/python3.8/unittest/loader.py", line 377, in _get_module_from_name __import__(name) File "/home/marten/coding/pytom/tests/E2ETests/test_GUIWidgets.py", line 18, in create_project_filestructure(fname) File "/home/marten/miniconda3/envs/pytom_env/lib/python3.8/site-packages/pytom/gui/guiFunctions.py", line 715, in create_project_filestructure os.mkdir(projectdir) FileNotFoundError: [Errno 2] No such file or directory: 'E2ETests/TestGUI' ====================================================================== ERROR: test_Simulation (test_MicrographModellerTest.MicrographModellerTest) Run two simulations and test their correlation. Both will have a different realization of noise and will ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/marten/coding/pytom/tests/E2ETests/test_MicrographModellerTest.py", line 25, in setUp self.potential = iasa_integration(self.param_pot['pdb'], File "/home/marten/miniconda3/envs/pytom_env/lib/python3.8/site-packages/pytom/simulation/potential.py", line 1358, in iasa_integration real = reduce_resolution_fourier(real, voxel_size, voxel_size*2*oversampling) File "/home/marten/miniconda3/envs/pytom_env/lib/python3.8/site-packages/pytom/simulation/support.py", line 70, in reduce_resolution_fourier return applyFourierFilterFull(input, xp.fft.ifftshift(gaussian_filter)) File "/home/marten/miniconda3/envs/pytom_env/lib/python3.8/site-packages/pytom/agnostic/filter.py", line 1025, in applyFourierFilterFull return xp.fft.ifftn(xp.fft.fftn(particle) * filter).real.astype(xp.float32) File "/home/marten/miniconda3/envs/pytom_env/lib/python3.8/site-packages/cupy/fft/_fft.py", line 775, in fftn return func(a, s, axes, norm, cufft.CUFFT_FORWARD) File "/home/marten/miniconda3/envs/pytom_env/lib/python3.8/site-packages/cupy/fft/_fft.py", line 575, in _fftn raise TypeError('The input array a must be a cupy.ndarray') TypeError: The input array a must be a cupy.ndarray ====================================================================== FAIL: test_Binning (test_AutofocusClassTest.pytom_MyFunctionTest) test implementation of binning functionality ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/marten/coding/pytom/tests/E2ETests/test_AutofocusClassTest.py", line 117, in test_Binning self.cleanUp() File "/home/marten/coding/pytom/tests/E2ETests/test_AutofocusClassTest.py", line 57, in cleanUp self.remove_file( filename=tline) File "/home/marten/coding/pytom/tests/E2ETests/test_AutofocusClassTest.py", line 81, in remove_file self.assertTrue( filecheck, msg="file "+filename+" does not exist") AssertionError: False is not true : file initial_0.em does not exist ====================================================================== FAIL: test_Masks (test_AutofocusClassTest.pytom_MyFunctionTest) test implementation of binning functionality ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/marten/coding/pytom/tests/E2ETests/test_AutofocusClassTest.py", line 143, in test_Masks self.cleanUp() File "/home/marten/coding/pytom/tests/E2ETests/test_AutofocusClassTest.py", line 57, in cleanUp self.remove_file( filename=tline) File "/home/marten/coding/pytom/tests/E2ETests/test_AutofocusClassTest.py", line 81, in remove_file self.assertTrue( filecheck, msg="file "+filename+" does not exist") AssertionError: False is not true : file initial_0.em does not exist ====================================================================== FAIL: runTest (test_BinScripts.pytom_MyFunctionTest) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/marten/coding/pytom/tests/E2ETests/test_BinScripts.py", line 415, in runTest self.addProjectDirToParticleList() File "/home/marten/coding/pytom/tests/E2ETests/test_BinScripts.py", line 56, in addProjectDirToParticleList assert pl2[0].getFilename() == os.path.join(folder, part) AssertionError ====================================================================== FAIL: test_CCC_CPU (test_CCCTest.pytom_MyFunctionTest) test implementation of binning functionality ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/marten/coding/pytom/tests/E2ETests/test_CCCTest.py", line 107, in test_CCC_CPU self.cleanUp() File "/home/marten/coding/pytom/tests/E2ETests/test_CCCTest.py", line 74, in cleanUp self.remove_file( filename=f'{self.settings["outputDirectory"]}/correlation_matrix.csv') File "/home/marten/coding/pytom/tests/E2ETests/test_CCCTest.py", line 87, in remove_file self.assertTrue( filecheck, msg="file "+filename+" does not exist") AssertionError: False is not true : file outputCCC//correlation_matrix.csv does not exist ---------------------------------------------------------------------- Ran 15 tests in 524.859s FAILED (failures=4, errors=6)
McHaillet commented 1 year ago

I tried running autofocus class test after the mpi4py update, and noticed two things:

I can also make a new pull request for fixing the autofocus test?

McHaillet commented 1 year ago

I will make another pull request here to incorporate the data from the unittest data repo (zipped). And also fix the alignment test using it

McHaillet commented 1 year ago

I will make another pull request here to incorporate the data from the unittest data repo (zipped). And also fix the alignment test using it

Turns out it doesnt compress well. When we downloaded the repo zipped, it didnt incorporate the git-lfs files correctly (see: https://github.com/git-lfs/git-lfs/issues/903). So we were just compressing pointers to the actual data (haha).

If I compress the stack of tilt-series data now, it goes from 1.9 Gb to 1.7 Gb. That seems (1) too large to ship with the main repository and (2) exceeds github filesize limit of 100 Mb (without git-lfs). I could circumvent the file size limit by splitting the tilt-series in individual files of ~55 Mb, that can be gzipped to 50 Mb. But that does not change the added data size of the whole repo.

sroet commented 1 year ago

@McHaillet do you know what the intended behavior is of test_Binscripts.pypytom_MyFunctionTest.addProjectDirToParticleList? Should the test be updated to the current behavior, or should the current test pass (and this the behavior updated)?

McHaillet commented 1 year ago

@McHaillet do you know what the intended behavior is of test_Binscripts.pypytom_MyFunctionTest.addProjectDirToParticleList? Should the test be updated to the current behavior, or should the current test pass (and this the behavior updated)?

What is the error?

I would assume it passes as it should just append a path to a filename.

sroet commented 1 year ago

After an offline discussion with @McHaillet we decided to merge this partial fix and will continue in another PR