gui crashed in 1st step

[cruzchue]

The manifold-gui crashed after first step ("import" tab, "view orientation distribution" button). I loaded the input files and input values (volume, aligment, etc.), but it seems to be an error related to VTK. I am attaching the logfile below.

Question: as GUI is giving problems, I was wondering if "manifold-cli" would be a cleaner alternative for now.

manifold-gui

 __  __             _  __       _     _ _____ __  __
|  \/  | __ _ _ __ (_)/ _| ___ | | __| | ____|  \/  |
| |\/| |/ _` | '_ \| | |_ / _ \| |/ _` |  _| | |\/| |
| |  | | (_| | | | | |  _| (_) | | (_| | |___| |  | |
|_|  |_|\__,_|_| |_|_|_|  \___/|_|\__,_|_____|_|  |_|

version: 0.1.dev241+g447ab76

Calculating projection direction information
Number of PDs: 99
Neighborhood epsilon: 0.43803364967714076
Number of Graph Edges: (409, 2)

Performing connected component analysis.
Number of connected components: 1
Number of Graph Edges: (409, 2)
2023-11-15 13:59:40.730 (9672.554s) [        30C54740]vtkOpenGLRenderWindow.c:517    ERR| vtkXOpenGLRenderWindow (0x5582e90427a0): Unable to find a valid OpenGL 3.2 or later implementation. Please update your video card driver to the latest version. If you are using Mesa please make sure you have version 11.2 or later and make sure your driver in Mesa supports OpenGL 3.2 such as llvmpipe or openswr. If you are on windows and using Microsoft remote desktop note that it only supports OpenGL 3.2 with nvidia quadro cards. You can use other remoting software such as nomachine to avoid this issue.
2023-11-15 13:59:40.730 (9672.554s) [        30C54740]     vtkOpenGLState.cxx:1795  WARN| Hardware does not support the number of textures defined.
2023-11-15 13:59:40.731 (9672.555s) [        30C54740]     vtkOpenGLState.cxx:1795  WARN| Hardware does not support the number of textures defined.
2023-11-15 13:59:40.734 (9672.558s) [        30C54740]     vtkOpenGLState.cxx:1795  WARN| Hardware does not support the number of textures defined.
2023-11-15 13:59:40.734 (9672.558s) [        30C54740]     vtkOpenGLState.cxx:1795  WARN| Hardware does not support the number of textures defined.
2023-11-15 13:59:40.734 (9672.558s) [        30C54740]     vtkOpenGLState.cxx:1795  WARN| Hardware does not support the number of textures defined.
2023-11-15 13:59:40.735 (9672.559s) [        30C54740]     vtkOpenGLState.cxx:1795  WARN| Hardware does not support the number of textures defined.
2023-11-15 13:59:40.735 (9672.559s) [        30C54740]     vtkOpenGLState.cxx:1795  WARN| Hardware does not support the number of textures defined.
2023-11-15 13:59:40.738 (9672.562s) [        30C54740]   vtkShaderProgram.cxx:437    ERR| vtkShaderProgram (0x5582eacf27f0): 1: #version 150
2: #ifndef GL_ES
3: #define highp
4: #define mediump
5: #define lowp
6: #endif // GL_ES
7: #define attribute in
8: #define varying out
9: 
10: 
11: /*=========================================================================
12: 
13:   Program:   Visualization Toolkit
14:   Module:    vtkPolyDataVS.glsl
15: 
16:   Copyright (c) Ken Martin, Will Schroeder, Bill Lorensen
17:   All rights reserved.
18:   See Copyright.txt or http://www.kitware.com/Copyright.htm for details.
19: 
20:      This software is distributed WITHOUT ANY WARRANTY; without even
21:      the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR
22:      PURPOSE.  See the above copyright notice for more information.
23: 
24: =========================================================================*/
25: 
26: in vec4 vertexMC;
27: 
28: 
29: 
30: // frag position in VC
31: out vec4 vertexVCVSOutput;
32: 
33: // optional normal declaration
34: //VTK::Normal::Dec
35: in vec3 normalMC;
36: uniform mat3 normalMatrix;
37: out vec3 normalVCVSOutput;
38: 
39: // extra lighting parameters
40: //VTK::Light::Dec
41: 
42: // Texture coordinates
43: //VTK::TCoord::Dec
44: 
45: // material property values
46: in vec4 scalarColor;
47: out vec4 vertexColorVSOutput;
48: 
49: // clipping plane vars
50: //VTK::Clip::Dec
51: 
52: // camera and actor matrix values
53: uniform mat4 MCDCMatrix;
54: uniform mat4 MCVCMatrix;
55: 
56: // Apple Bug
57: //VTK::PrimID::Dec
58: 
59: // Value raster
60: //VTK::ValuePass::Dec
61: 
62: // picking support
63: //VTK::Picking::Dec
64: 
65: void main()
66: {
67:   //VTK::CustomBegin::Impl
68: 
69:   vertexColorVSOutput = scalarColor;
70: 
71:   normalVCVSOutput = normalMatrix * normalMC;
72: //VTK::Normal::Impl
73: 
74:   //VTK::TCoord::Impl
75: 
76:   //VTK::Clip::Impl
77: 
78:   //VTK::PrimID::Impl
79: 
80:   vertexVCVSOutput = MCVCMatrix * vertexMC;
81:   gl_Position = MCDCMatrix * vertexMC;
82: 
83: 
84:   //VTK::ValuePass::Impl
85: 
86:   //VTK::Light::Impl
87: 
88:   //VTK::Picking::Impl
89: 
90:   //VTK::CustomEnd::Impl
91: }

2023-11-15 13:59:40.740 (9672.564s) [        30C54740]   vtkShaderProgram.cxx:438    ERR| vtkShaderProgram (0x5582eacf27f0): 0:1(10): error: GLSL 1.50 is not supported. Supported versions are: 1.10, 1.20, 1.00 ES, and 3.00 ES

Segmentation fault (core dumped)

[blackwer]

Are you trying to run this remotely? It's claiming it can't find a GL context that is new enough.

What OS are you running this on and is it local?

[cruzchue]

1.- I connect to the server through FastX (https://www.starnet.com/fastx/), which opens a desktop environment. No advertisement, but FastX is very fast and when using python/matplotlib I never had a problem.

That is, I do NOT "ssh -XY" for visualization. I am attaching a snapshot of my FastX connection with some applications open (manifold-gui, chimera, vmd, matlab) just to show my environment .

2.- The OS is Centos OS 7

[blackwer]

Unfortunately mayavi is more aggressively requiring more recent opengl specifications -- and mayavi was what I inherited from the original GUI. It's on the list of things to change the backend of. Maybe we can engineer a temporary workaround for now though. Can you give me the output from

glxinfo | grep OpenGL

[cruzchue]

thanks for looking into it. Indeed, mayavi has also been an issue for us. That rendering should be nice for CryoEM volumes, but now it is difficult to use. I will try to work on my iMac to provide an update before thanksgiving.

Below, the output of the command.

glxinfo  | grep OpenGL
OpenGL vendor string: VMware, Inc.
OpenGL renderer string: llvmpipe (LLVM 7.0, 256 bits)
OpenGL version string: 2.1 Mesa 18.3.4
OpenGL shading language version string: 1.20
OpenGL extensions:
OpenGL ES profile version string: OpenGL ES 2.0 Mesa 18.3.4
OpenGL ES profile shading language version string: OpenGL ES GLSL ES 1.0.16
OpenGL ES profile extensions:

[blackwer]

you did conda install for mayavi right? try

conda create -n myfreshenv python=3.10
conda activate myfreshenv
conda install mesalib mayavi vtk -c conda-forge
pip install "git+ssh://git@github.com/flatironinstitute/ManifoldEM"

which in theory will use a software rendering backend when it can

[cruzchue]

same error with a fresh environment in a Linux Server. Below I paste the last lines of the logfile:

84:   //VTK::ValuePass::Impl
85: 
86:   //VTK::Light::Impl
87: 
88:   //VTK::Picking::Impl
89: 
90:   //VTK::CustomEnd::Impl
91: }

2023-11-20 15:02:43.501 ( 159.984s) [         DF38740]   vtkShaderProgram.cxx:438    ERR| vtkShaderProgram (0x5557f0a03cc0): 0:1(10): error: GLSL 1.50 is not supported. Supported versions are: 1.10, 1.20, 1.00 ES, and 3.00 ES

Segmentation fault (core dumped)
(myfreshenv) [cruzchu@neumann 20_ghTest]$ 

I can run manfold-gui on an iMac, but the process gets stuck at the Spectral Analysis step. Need to check once more and then I will update

[blackwer]

same error with a fresh environment in a Linux Server. Below I paste the last lines of the logfile:

I'll put some effort into providing a matplotlib fallback mechanism for the 3d viz next week. Technically you only need the GUI for anchor/sense selection right now, but that's a very important bit. Sorry about this - remote 3D is a serious sticking point in these kinds of workflows, and we've been largely able to work around that limitation by using the workstations here.

The CLI is largely self-documenting if you want to experiment there

# global help. commands output are in the order you need to use them
manifold-cli -h
# help for "init" stage
manifold-cli init -h

manifold-cli init -v $HOME/projects/codes/ManifoldEM/J310/J310_003_volume_map.mrc -a $HOME/projects/codes/ManifoldEM/J310/from_csparc.star -i $HOME/projects/codes/ManifoldEM/J310/signal_subtracted.mrcs -s 1.22 -d 160.0 -r 3.02 -x 1 -p test
manifold-cli threshold --low 100 --high 20000
# run on 64 processes
manifold-cli -n 64 calc-distance
manifold-cli -n 64 manifold-analysis
manifold-cli -n 64 psi-analysis
manifold-cli -n 64 nlsa-movie
#here's where you would want to use the gui... anchor selection etc
manifold-cli -n 64 find-ccs
manifold-cli -n 64 energy-landscape

I still need to add some extra visualization to look at the results. So far no one has actually used the CLI so it needs some love for manipulating/viewing the data

I can run manfold-gui on an iMac, but the process gets stuck at the Spectral Analysis step. Need to check once more and then I will update

I'll take a look if you can provide any info. There are some severe issues with memory usage that are on the agenda to address.

[cruzchue]

The "Spectral Analysis" step is idle since yesterday after 2% progress (see snapshot). Below I am including the log.

manifold-gui

 __  __             _  __       _     _ _____ __  __
|  \/  | __ _ _ __ (_)/ _| ___ | | __| | ____|  \/  |
| |\/| |/ _` | '_ \| | |_ / _ \| |/ _` |  _| | |\/| |
| |  | | (_| | | | | |  _| (_) | | (_| | |___| |  | |
|_|  |_|\__,_|_| |_|_|_|  \___/|_|\__,_|_____|_|  |_|

version: 0.1.dev241+g447ab76

Calculating projection direction information
Number of PDs: 136
Neighborhood epsilon: 0.3663510010898601
Number of Graph Edges: (946, 2)

Performing connected component analysis.
Number of connected components: 1
Number of Graph Edges: (946, 2)
Computing the distances...
Computing the eigenfunctions...
Computing the NLSA snapshots...
Exception in thread Thread-4 (op):
Traceback (most recent call last):
  File "/Users/cruzchu/anaconda3/envs/flatironManifoldEM/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
    self.run()
  File "/Users/cruzchu/anaconda3/envs/flatironManifoldEM/lib/python3.10/threading.py", line 953, in run
    self._target(*self._args, **self._kwargs)
  File "/Users/cruzchu/anaconda3/envs/flatironManifoldEM/lib/python3.10/site-packages/ManifoldEM/psiAnalysis.py", line 292, in op
    local_psi_func(datai)
  File "/Users/cruzchu/anaconda3/envs/flatironManifoldEM/lib/python3.10/site-packages/ManifoldEM/psiAnalysis.py", line 211, in psi_analysis_single
    IMGT, Topo_mean, psirec, psiC1, sdiag, VX, mu, tau = _NLSA(NLSAPar, DD, pos_path, pos_psi1, imgAll, msk2, CTF,
  File "/Users/cruzchu/anaconda3/envs/flatironManifoldEM/lib/python3.10/site-packages/ManifoldEM/psiAnalysis.py", line 54, in _NLSA
    lambdaC, psiC, sigmaC, mu, logEps, logSumWij, popt, R_squared = DMembeddingII.op(
  File "/Users/cruzchu/anaconda3/envs/flatironManifoldEM/lib/python3.10/site-packages/ManifoldEM/DMembeddingII.py", line 235, in op
    popt, logSumWij, resnorm, R_squared = fergusonE(np.sqrt(yVal), logEps, a0)
  File "/Users/cruzchu/anaconda3/envs/flatironManifoldEM/lib/python3.10/site-packages/ManifoldEM/core.py", line 158, in fergusonE
    popt, pcov = curve_fit(fun, logEps, logSumWij, p0=a0)
  File "/Users/cruzchu/anaconda3/envs/flatironManifoldEM/lib/python3.10/site-packages/scipy/optimize/_minpack_py.py", line 968, in curve_fit
    raise RuntimeError("Optimal parameters not found: " + errmsg)
RuntimeError: Optimal parameters not found: Number of calls to function has reached maxfev = 1000.

[blackwer]

The "Spectral Analysis" step is idle since yesterday after 2% progress (see snapshot). Below I am including the log.

Ah it looks like an actual math failure! I've never actually encountered an issue with this, so can we try this dataset on a linux machine now? Just to rule out that part of the environment. I've fortunately already made a workaround for the GL problems.

Install the latest version of ManifoldEM from main

pip install "git+ssh://git@github.com/flatironinstitute/ManifoldEM"
# you can also just
# export MANIFOLD_DISABLE_VIZ=1
MANIFOLD_DISABLE_VIZ=1 manifold-gui

Edit: If this works and it's still failing in the spectral analysis bit, please move that to a separate issue

[blackwer]

can you provide the project .toml file for one of these runs where the run is crashing in the spectral analysis step?