search

anmartinezs / pyseg_system

De novo analysis for cryo-electron tomography
Other
20 stars 4 forks source link

A test error #23

Open Liuyixiao-1 opened 2 years ago

Liuyixiao-1 commented 2 years ago

I followed the manual and installed pyseg on ubuntu 18.04. The version of python is 3.9 When I execute ./tracing_grid.sh, an error occurred pyseg.pexceptions.pexceptions.PySegInputError 2022-09-28 23:26:48.427 ( 7.075s) [ AC0B6740]vtkXMLUnstructuredDataR:776 ERR| vtkXMLUnstructuredGridReader (0x55b0155f1790): Cannot read cell data from Cells. Offsets and connectivity arrays must be the same type. How can I fix it?Hoping for your reply!

Liuyixiao-1 commented 2 years ago

my vtk version is 9.0.3

Liuyixiao-1 commented 2 years ago

I noticed that in the generated file "grid_noisestdit_g0.5.fits_s0.05_manifolds_J0a.NDnet.vtu" offsets and connectivity have different types, Int64 and Int32 respectively. What caused the difference in data types? `

`

anmartinezs commented 2 years ago

Hi, Could you tell me exactly the code line in https://github.com/anmartinezs/pyseg_system/blob/master/code/pyseg/scripts/tests/mcf_synthetic_test.py script raised the error.

Liuyixiao-1 commented 2 years ago

It was line 114 and error messages are as follows.

Process Process-1: Traceback (most recent call last): File "/home/fingalyu/anaconda3/envs/test/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap self.run() File "/home/fingalyu/anaconda3/envs/test/lib/python3.9/multiprocessing/process.py", line 108, in run self._target(*self._args, **self._kwargs) **File "/home/fingalyu/Softwares/pyseg_system-master/code/tests/../pyseg/scripts/tests/mcf_synthetic_test.py", line 114, in pr_mcf_graph mcf_graph.main(main_args)** File "/home/fingalyu/Softwares/pyseg_system-master/code/pyseg/scripts/mcf_graph.py", line 247, in main do_mcf_graph(input_file, output_dir, mask_file, res, tilt_rot, tilt_ang, cut_t, rob, File "/home/fingalyu/Softwares/pyseg_system-master/code/pyseg/scripts/mcf_graph.py", line 106, in do_mcf_graph manifolds = disperse.get_manifolds(no_cut=False, inv=False) File "/home/fingalyu/Softwares/pyseg_system-master/code/pyseg/disperse_io/handler.py", line 235, in get_manifolds disperse_io.manifold3d_from_vtu_to_img(filename=input_file_net, outputdir=self.__work_dir, File "/home/fingalyu/Softwares/pyseg_system-master/code/pyseg/disperse_io/disperse_io.py", line 138, in manifold3d_from_vtu_to_img raise pyseg.pexceptions.PySegInputError(expr='manifold3d_from_vtu_to_img', msg=error_msg) pyseg.pexceptions.pexceptions.PySegInputError

anmartinezs commented 2 years ago

It seems that DisPerSe is failing, could you check the DisPerSe log file? it should be placed a folder pyseg_system/data/synthetic_grid/disperse/disperse.log

Do you have in your system (or have permission to write) the pyseg_system/data/synthetic_grid folder? Have you succesfully installed DisPerSe? and Have you set properly the environment variables to have access the its executable binaries?

Liuyixiao-1 commented 2 years ago

Thanks for your reply. I can find log file in pyseg_system/data/synthetic_grid/graph_0_it0/disperse/disperse.log. Here is one of the log file. Is there anything wrong? Several folders and files(graphit , grid_it.pkl, and grid_maskstdit_mask.fits) generated in synthetic_grid folder and the command "mse" woked correctly. `** MSE v0.9.24 (64 bits) ** Will read field from file '/home/fingalyu/Softwares/pyseg_system-master/code/pyseg/scripts/tests/../../../tests//../../data/synthetic_grid/graph_0_it_0/grid_noise_std_0_it_0_g0.5.fits'. Loading [37,37,37] array from file /home/fingalyu/Softwares/pyseg_system-master/code/pyseg/scripts/tests/../../../tests//../../data/synthetic_grid/graph_0_it_0/grid_noise_std_0_it_0_g0.5.fits ... done. Generating implicit simplicial complex ... done.

Will read field from file '/home/fingalyu/Softwares/pyseg_system-master/code/pyseg/scripts/tests/../../../tests//../../data/synthetic_grid/grid_mask_std_0_it_0_mask.fits'. Loading [37,37,37] array from file /home/fingalyu/Softwares/pyseg_system-master/code/pyseg/scripts/tests/../../../tests//../../data/synthetic_grid/grid_mask_std_0_it_0_mask.fits ... done. Building mask ... done. (10800 2-faces on boundary) INFORMATION: enforcing manifolds computation. Starting Morse-Smale complex computation. Computing discrete gradient for 1316978 cells: Identifying (2,3)-cell pairs: Sorting 607836 2-cells (asc.) ... done. Computing MST ... (components) done.(1731/303918 comp.) Computing discrete Gradient (1731 comp.) ... done. Identifying (1,0)-cell pairs: Sorting 354571 1-cells (desc.) ... done. Computing MST ... (components) done.(3506/50653 comp.) Computing discrete Gradient (3506 comp.) ... done. Identifying (1,2)-cell pairs: Computing saddle-saddle DG ... done. Critical cells : 3506(+0) 0-cells, 12213(+12213) 1-cells, 10439(+10439) 2-cells, 1731(+0) 3-cells. Computing discrete Morse-Smale complex: 27889 critical points: 1731 min, 10439 s1, 12213 s2, 3506 max.

[2022年09月29日 星期四 15时03分02秒]RUNNING COMMAND:-> mse /home/fingalyu/Softwares/pyseg_system-master/code/pyseg/scripts/tests/../../../tests//../../data/synthetic_grid/graph_0_it_0/grid_noise_std_0_it_0_g0.5.fits -mask /home/fingalyu/Softwares/pyseg_system-master/code/pyseg/scripts/tests/../../../tests//../../data/synthetic_grid/grid_mask_std_0_it_0_mask.fits -dumpManifolds J0a -nsig 0.05 -downSkl -outDir /home/fingalyu/Softwares/pyseg_system-master/code/pyseg/scripts/tests/../../../tests//../../data/synthetic_grid/graph_0_it_0/disperse Will read skeleton from file '/home/fingalyu/Softwares/pyseg_system-master/code/pyseg/scripts/tests/../../../tests//../../data/synthetic_grid/graph_0_it_0/disperse/grid_noise_std_0_it_0_g0.5.fits_s0.05.down.NDskl'. Loading 3D skeleton from file /home/fingalyu/Softwares/pyseg_system-master/code/pyseg/scripts/tests/../../../tests//../../data/synthetic_grid/graph_0_it_0/disperse/grid_noise_std_0_it_0_g0.5.fits_s0.05.down.NDskl ... done. Smoothing skeleton 3 times ... done. Will write skeleton to file '/home/fingalyu/Softwares/pyseg_system-master/code/pyseg/scripts/tests/../../../tests//../../data/synthetic_grid/graph_0_it_0/disperse/grid_noise_std_0_it_0_g0.5.fits_s0.05.down.NDskl.S003.vtp'.

[2022年09月29日 星期四 15时03分05秒]RUNNING COMMAND:-> skelconv /home/fingalyu/Softwares/pyseg_system-master/code/pyseg/scripts/tests/../../../tests//../../data/synthetic_grid/graph_0_it_0/disperse/grid_noise_std_0_it_0_g0.5.fits_s0.05.down.NDskl -outDir /home/fingalyu/Softwares/pyseg_system-master/code/pyseg/scripts/tests/../../../tests//../../data/synthetic_grid/graph_0_it_0/disperse -smooth 3 -to vtp Will read network from file '/home/fingalyu/Softwares/pyseg_system-master/code/pyseg/scripts/tests/../../../tests//../../data/synthetic_grid/graph_0_it_0/disperse/grid_noise_std_0_it_0_g0.5.fits_s0.05_manifolds_J0a.NDnet'. Loading 3D network from file "/home/fingalyu/Softwares/pyseg_system-master/code/pyseg/scripts/tests/../../../tests//../../data/synthetic_grid/graph_0_it_0/disperse/grid_noise_std_0_it_0_g0.5.fits_s0.05_manifolds_J0a.NDnet" ... done. Will write network to file '/home/fingalynetconv_cmdu/Softwares/pyseg_system-master/code/pyseg/scripts/tests/../../../tests//../../data/synthetic_grid/graph_0_it_0/disperse/grid_noise_std_0_it_0_g0.5.fits_s0.05_manifolds_J0a.NDnet.vtu'.

[2022年09月29日 星期四 15时03分05秒]RUNNING COMMAND:-> netconv /home/fingalyu/Softwares/pyseg_system-master/code/pyseg/scripts/tests/../../../tests//../../data/synthetic_grid/graph_0_it_0/disperse/grid_noise_std_0_it_0_g0.5.fits_s0.05_manifolds_J0a.NDnet -outDir /home/fingalyu/Softwares/pyseg_system-master/code/pyseg/scripts/tests/../../../tests//../../data/synthetic_grid/graph_0_it_0/disperse -to vtu `

anmartinezs commented 2 years ago

It seems you have succefully intall DisPerSe third-party dependencies, but for some reason netconv DisPerSe binary fails or at least it does not create the output manifold properly. You should run in the reminal the next command in order to see if it raises an error:
netconv /home/fingalyu/Softwares/pyseg_system-master/code/pyseg/scripts/tests/../../../tests//../../data/synthetic_grid/graph_0_it_0/disperse/grid_noise_std_0_it_0_g0.5.fits_s0.05_manifolds_J0a.NDnet -outDir /home/fingalyu/Softwares/pyseg_system-master/code/pyseg/scripts/tests/../../../tests//../../data/synthetic_grid/graph_0_it_0/disperse -to vtu

anmartinezs commented 2 years ago

Do you get a /home/fingalyu/Softwares/pyseg_system-master/code/pyseg/scripts/tests/../../../tests//../../data/synthetic_grid/graph_0_it_0/disperse/grid_noise_std_0_it_0_g0.5.fits_s0.05_manifolds_J0a.NDnet.vtu file, could you open in Paraview? Do you have the Cell data property 'source_index'?

Liuyixiao-1 commented 2 years ago

I don't have paraview in my system, so I open the grid_noise_std_0_it_0_g0.5.fits_s0.05_manifolds_J0a.NDnet.vtu with VScode instead. I ran the command netconv /home/fingalyu/Softwares/pyseg_system-master/code/pyseg/scripts/tests/../../../tests//../../data/synthetic_grid/graph_0_it_0/disperse/grid_noise_std_0_it_0_g0.5.fits_s0.05_manifolds_J0a.NDnet -outDir /home/fingalyu/Softwares/pyseg_system-master/code/pyseg/scripts/tests/../../../tests//../../data/synthetic_grid/graph_0_it_0/disperse -to vtu. It worked correctly Actually Cell data property 'source_index' are contained in .vtu file. It was DataArray Name="source_index" type="Float64" format="appended" offset="4407524"/>

Liuyixiao-1 commented 2 years ago

The first few lines of the file look like this `

` As I described 3 days ago， offsets and connectivity have different types, Int64 and Int32 respectively. I think this is the reason for the error message 'vtkXMLUnstructuredGridReader (0x55b0155f1790): Cannot read cell data from Cells. Offsets and connectivity arrays must be the same type.' But I don't know how to fix it.
anmartinezs commented 2 years ago

It is normal to have two 'source_index' properties, one for points and another for cells.

anmartinezs commented 2 years ago

Could you show me the lines of your code around line 138 in file /home/fingalyu/Softwares/pyseg_system-master/code/pyseg/disperse_io/disperse_io.py ? This is where the error is raised.

anmartinezs commented 2 years ago

If you open the file with Paraview, it is very easy to be install, you can ensure that manifold file is not corrupted.

anmartinezs commented 2 years ago

Find attached an exemplar of header for .vtu file which works:

`

`
Liuyixiao-1 commented 2 years ago

The code around line 138 looked like this image And I opened the grid_noise_std_0_it_0_g0.5.fits_s0.05_manifolds_J0a.NDnet.vtu in Paraview. It opened with no error message and looked like this image

anmartinezs commented 2 years ago

Do you have the latest version of PySeg? on Github the line 138 is empty, the error is actually raised in the line 137 in https://github.com/anmartinezs/pyseg_system/blob/master/code/pyseg/disperse_io/disperse_io.py: raise pyseg.pexceptions.PySegInputError(expr='manifold3d_from_vtu_to_img', msg=error_msg)

What is different in your code?

Liuyixiao-1 commented 2 years ago

I double-checked my code and found only 116 extra lines of blank lines. I've downloaded the latest PySeg code but it still doesn't work. I'm thinking if the error is caused by the VTK version. Can you please check the vtk version you are using in python 3.9 and let me know?

anmartinezs commented 2 years ago

Yes, this what I suspected from the begining but I wanted to discard any other possible source of error. Find attached my current settings:

Python 3.8.3 | packaged by conda-forge | (default, Jun 1 2020, 17:43:00) [GCC 7.5.0] on linux Type "help", "copyright", "credits" or "license" for more information. readline: /etc/inputrc: line 19: term: unknown variable name readline: /etc/inputrc: line 19: term: unknown variable name

import vtk vtk.vtkVersion().GetVTKVersion() '8.2.0'

Liuyixiao-1 commented 2 years ago

Hi,Thanks for your help, I reconfigured a new environment with python 3.8. Now I can run tracing_gird.sh properly, but another error appeared when I ran gen_microsomes.py according to the tutorial. It is ERROR: the process 0 ended unsuccessfully [-9]. I found that exitcode -9 indicates a memory error. So I checked the memory usage. I found that when the code got to line 475 the memory was quickly occupied and the process was killed which caused the error.

Liuyixiao-1 commented 2 years ago

The RAM of my PC is 48GB

anmartinezs commented 2 years ago

Yeah, this is very likely the problem. We used to work with cluster nodes with 350 Gb or more, and the script gen_microsomes.py (it generates a synthetic datasets), is memory demanding. If you want to reduce the memory demands of this script you can modify the variable to control the number of parallel process: mp_npr =1 # number of parallel processes

anmartinezs commented 2 years ago

Set mp_npr to a lower value reduces the memory loads but increases the running time.

Liuyixiao-1 commented 2 years ago

I had set mp_npr = 1, however the process was also been killed automatically.What parts should I check or fix? 2022-10-12 19-51-33屏幕截图

anmartinezs commented 2 years ago

If you problem is still the memory, you can try to reduce the output tomogram size: tm_size = (1200, 1200, 640) # pixels