visit-dav / visit

VisIt - Visualization and Data Analysis for Mesh-based Scientific Data
https://visit.llnl.gov
BSD 3-Clause "New" or "Revised" License
435 stars 114 forks source link

Visit dumps core applying operators to defined expressions #12272

Closed biagas closed 3 years ago

biagas commented 3 years ago

We're trying to combine visit-users email list with GitHub issues. When replying on visit-users, please reply only to emails with a Subject line that includes [visit-dav/live-customer-response].


I am reporting the following behavior in Visit 3.1.3. The example discussed below pertains to 3D data stored in a SILO file, drawn using the Pseudocolor plot type. My software environment is Fedora 33 with VTK 8.2.

(1) Plot operators (Clip, Slice, etc.) work as expected with Pseudocolor plots with database variables (those defined in the SILO file), (2) User defined expressions can also be plotted, but only if no operators are applied, (3) Attempting to apply an operator to a Pseudocolor plot of a user defined expression dumps core

Here is a typical log:

Process 1055579 (engine_par) of user 1000 dumped core.

Stack trace of thread 1055579:
#0  0x00007f9d7a5afbc5 raise (libc.so.6 + 0x3dbc5)
visit-dav/visit#12095  0x00007f9d7a5988a4 abort (libc.so.6 + 0x268a4)
visit-dav/visit#12094  0x00007f9d7bb9bbb9 _ZL18signalhandler_corei (libvisitcommon.so + 0x1d7bb9)
visit-dav/visit#12096  0x00007f9d7a5afc50 __restore_rt (libc.so.6 + 0x3dc50)
visit-dav/visit#12097  0x00007f9d7ad15bb7 _ZN23vtkAOSDataArrayTemplateIhE10GetPointerEx (libvtkCommonCore.so.1 + 0x280bb7)
visit-dav/visit#12098  0x00007f9d7b97f6fa _ZN13vtkDataWriter10WriteArrayEPSoiP16vtkAbstractArrayPKcxx (libvtkIOLegacy.so.1 + 0x596fa)
visit-dav/visit#12099  0x00007f9d7b981bae _ZN13vtkDataWriter14WriteFieldDataEPSoP12vtkFieldData (libvtkIOLegacy.so.1 + 0x5bbae)
visit-dav/visit#12100  0x00007f9d7b98305e _ZN13vtkDataWriter16WriteDataSetDataEPSoP10vtkDataSet (libvtkIOLegacy.so.1 + 0x5d05e)
visit-dav/visit#12101  0x00007f9d7b98fed2 _ZN17vtkPolyDataWriter9WriteDataEv (libvtkIOLegacy.so.1 + 0x69ed2)
visit-dav/visit#12102  0x00007f9d7b902d1c _ZN9vtkWriter11RequestDataEP14vtkInformationPP20vtkInformationVectorS3_ (libvtkIOCore.so.1 + 0x56d1c)
visit-dav/visit#12103 0x00007f9d7b3106fd _ZN12vtkExecutive13CallAlgorithmEP14vtkInformationiPP20vtkInformationVectorS3_ (libvtkCommonExecutionModel.so.1 + 0x5a6fd)
visit-dav/visit#12104 0x00007f9d7b30d499 _ZN23vtkDemandDrivenPipeline11ExecuteDataEP14vtkInformationPP20vtkInformationVectorS3_ (libvtkCommonExecutionModel.so.1 + 0x57499)
visit-dav/visit#12105 0x00007f9d7b30ba52 _ZN24vtkCompositeDataPipeline11ExecuteDataEP14vtkInformationPP20vtkInformationVectorS3_ (libvtkCommonExecutionModel.so.1 + 0x55a52)
visit-dav/visit#12106 0x00007f9d7b31561f _ZN23vtkDemandDrivenPipeline14ProcessRequestEP14vtkInformationPP20vtkInformationVectorS3_ (libvtkCommonExecutionModel.so.1 + 0x5f61f)
visit-dav/visit#12107 0x00007f9d7b337101 _ZN32vtkStreamingDemandDrivenPipeline14ProcessRequestEP14vtkInformationPP20vtkInformationVectorS3_ (libvtkCommonExecutionModel.so.1 + 0x81101)
visit-dav/visit#12108 0x00007f9d7b33a2a7 _ZN32vtkStreamingDemandDrivenPipeline6UpdateEiP20vtkInformationVector (libvtkCommonExecutionModel.so.1 + 0x842a7)
visit-dav/visit#12109 0x00007f9d7b902b6f _ZN9vtkWriter5WriteEv (libvtkIOCore.so.1 + 0x56b6f)
visit-dav/visit#12110 0x00007f9d7b97aba9 _ZN16vtkDataSetWriter9WriteDataEv (libvtkIOLegacy.so.1 + 0x54ba9)
visit-dav/visit#12111 0x00007f9d7b902d1c _ZN9vtkWriter11RequestDataEP14vtkInformationPP20vtkInformationVectorS3_ (libvtkIOCore.so.1 + 0x56d1c)
visit-dav/visit#12112 0x00007f9d7b3106fd _ZN12vtkExecutive13CallAlgorithmEP14vtkInformationiPP20vtkInformationVectorS3_ (libvtkCommonExecutionModel.so.1 + 0x5a6fd)
visit-dav/visit#12113 0x00007f9d7b30d499 _ZN23vtkDemandDrivenPipeline11ExecuteDataEP14vtkInformationPP20vtkInformationVectorS3_ (libvtkCommonExecutionModel.so.1 + 0x57499)
visit-dav/visit#12114 0x00007f9d7b30ba52 _ZN24vtkCompositeDataPipeline11ExecuteDataEP14vtkInformationPP20vtkInformationVectorS3_ (libvtkCommonExecutionModel.so.1 + 0x55a52)
visit-dav/visit#12115 0x00007f9d7b31561f _ZN23vtkDemandDrivenPipeline14ProcessRequestEP14vtkInformationPP20vtkInformationVectorS3_ (libvtkCommonExecutionModel.so.1 + 0x5f61f)
visit-dav/visit#12116 0x00007f9d7b337101 _ZN32vtkStreamingDemandDrivenPipeline14ProcessRequestEP14vtkInformationPP20vtkInformationVectorS3_ (libvtkCommonExecutionModel.so.1 + 0x81101)
visit-dav/visit#12117 0x00007f9d7b33a2a7 _ZN32vtkStreamingDemandDrivenPipeline6UpdateEiP20vtkInformationVector (libvtkCommonExecutionModel.so.1 + 0x842a7)
visit-dav/visit#12118 0x00007f9d7b902b6f _ZN9vtkWriter5WriteEv (libvtkIOCore.so.1 + 0x56b6f)
visit-dav/visit#12119 0x00007f9d7c9df54f _ZN21avtDataRepresentation11vtkToStringEb (libavtpipeline_par.so + 0xb754f)
visit-dav/visit#12120 0x00007f9d7c9df738 _ZN21avtDataRepresentation13GetDataStringERiR11DataSetTypeb (libavtpipeline_par.so + 0xb7738)
visit-dav/visit#12121 0x00007f9d7ca180f8 _ZN16avtDataSetWriter13WriteDataTreeE7ref_ptrI11avtDataTreeER19avtDataObjectString (libavtpipeline_par.so + 0xf00f8)
visit-dav/visit#12122 0x00007f9d7ca18372 _ZN16avtDataSetWriter15DataObjectWriteER19avtDataObjectString (libavtpipeline_par.so + 0xf0372)
visit-dav/visit#12123 0x00007f9d7ca11b39 _ZN19avtDataObjectWriter5WriteER19avtDataObjectString (libavtpipeline_par.so + 0xe9b39)
visit-dav/visit#12124 0x00007f9d7e82b124 _ZL32ParallelMergeClonedWriterOutputs7ref_ptrI13avtDataObjectEiiPFviPKcPvES4_ (libengine_par.so + 0x72124)
visit-dav/visit#12125 0x00007f9d7e82c1ca _ZN6Engine10GatherDataER7ref_ptrI19avtDataObjectWriterEbbiifPFviPKcPvES6_PFvR19avtDataObjectStringS6_ES6_RNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEPbPi (libengine_par.so + 0x731ca)
visit-dav/visit#12126 0x00007f9d7e82ce6d _ZN17EngineRPCExecutorI10ExecuteRPCE7ExecuteEPS0_ (libengine_par.so + 0x73e6d)
visit-dav/visit#12127 0x00007f9d7bce5b19 _ZN7Subject6NotifyEv (libvisitcommon.so + 0x321b19)
visit-dav/visit#12128 0x00007f9d7bbee90d _ZN16AttributeSubject6NotifyEv (libvisitcommon.so + 0x22a90d)
visit-dav/visit#12129 0x00007f9d7e85f681 _ZN7MPIXfer7ProcessEv (libengine_par.so + 0xa6681)
visit-dav/visit#12130 0x00007f9d7e82abf6 _ZN6Engine13PAR_EventLoopEv (libengine_par.so + 0x71bf6)
visit-dav/visit#12131 0x00000000004023cd _Z10EngineMainiPPc (engine_par + 0x23cd)
visit-dav/visit#12132 0x00007f9d7a59a1a2 __libc_start_main (libc.so.6 + 0x281a2)
visit-dav/visit#12133 0x00000000004021ee _start (engine_par + 0x21ee)

The error is evidently VTK related, but this is rather beyond my expertise. Would appreciate suggestions on how to fix this problem.

Thanks,

-- Vladimir Florinski

angel-devicente commented 3 years ago

Hello,

could this be related to the issue I reported some time ago? https://github.com/visit-dav/live-customer-response/issues/157

In my case VisIt would not core dump, but the plot would not be produced and would get the error "[...] could not be generated by the compute engine on host "localhost"."

Cheer, AdV

biagas commented 3 years ago

Hello Vladimir,

When you say your software environment includes VTK-8.2, does that mean you have compiled VisIt from source against VTK-8.2? Did you use the build_visit script?

VisIt currently only supports and is known to work with VTK-8.1, so using 8.2 could be the source of the problem.

Are you able to work with a VisIt release tarball on your system? If not, can you rebuild VisIt with vtk-8.1?

markcmiller86-visit commented 3 years ago

Kathleen,

I do not use build_visit, in part because it did not work for me in the past, but mainly because it adds an extra layer above cmake, which should do the job on its own as designed. Basically, I proceed like this:

... a lot of patching applied to make the code to compile ... cmake -Wno-dev \ -DVISIT_PARALLEL:BOOL=ON \ -DVISIT_MPI_COMPILER:STRING="mpicc" \ -DVISIT_PYTHON_SCRIPTING:BOOL=OFF \ -DVISIT_PYTHON_FILTERS:BOOL=OFF \ -DVISIT_VTK_VERSION:STRING="8.2.0" \ -DVISIT_VTK_DIR:PATH=%{_prefix} \ -DVISIT_VTK_SKIP_INSTALL:BOOL=ON \ -DVISIT_QT_DIR:PATH=%{_prefix} \ -DVISIT_QT_SKIP_INSTALL:BOOL=ON \ -DVISIT_QWT_DIR:PATH=%{_prefix} \ -DVISIT_ZLIB_DIR:PATH=%{_prefix} \ -DVISIT_ZLIB_SKIP_INSTALL:BOOL=ON \ -DSILO_DIR:PATH=%{_prefix} \ -DHDF5_DIR:PATH=%{_prefix} \ -DNETCDF_DIR:PATH=%{_prefix} \ -DCMAKE_INSTALL_PREFIX:PATH=/opt/Visit make

(_prefix is /usr).

I have been using Visit 3.0.0 with VTK 8.2 for about a year with no problems. It is a major headache for me to try and build with software that is not native on my system. Ideally, I would prefer to figure out what causes the problem and fix it. I understand that it would also help the developers since the project will have to transition to a newer VTK sooner or later.

Thanks,

On Wed, Nov 18, 2020 at 10:10 AM Kathleen Biagas via visit-users < visit-users@elist.ornl.gov> wrote:

Hello Vladimir,

When you say your software environment includes VTK-8.2, does that mean you have compiled VisIt from source against VTK-8.2? Did you use the build_visit script?

VisIt currently only supports and is known to work with VTK-8.1, so using 8.2 could be the source of the problem.

Are you able to work with a VisIt release tarball on your system? If not, can you rebuild VisIt with vtk-8.1?

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/visit-dav/live-customer-response/issues/179#issuecomment-729783134, or unsubscribe https://github.com/notifications/unsubscribe-auth/AOLAYZFM7ZGMT4PI7LW2NFLSQPWXJANCNFSM4T2EYCCQ .

VisIt Users Wiki: http://visitusers.org/ Frequently Asked Questions for VisIt: https://wci.llnl.gov/simulation/computer-codes/visit/FAQ.html To Unsubscribe: send a blank email to visit-users-unsubscribe@elist.ornl.gov More Options: https://elist.ornl.gov/mailman/listinfo/visit-users

-- Vladimir Florinski

biagas commented 3 years ago

Yes, VisIt itself can be built quite successfully without the build_visit script. However, some thirdparty libaries (like VTK) need to be built with our script to ensure visit-centric patches and build settings are used.

Did you use the build_visit script to build third party libraries like VTK and QT? Or are you using pre-installed versions? Which version of QT and python are you using with VisIt?

The next VTK version VisIt will utilize will be in the 9.x series, the work to port VisIt to this version is a work-in-progress.

markcmiller86 commented 3 years ago

could this be related to the issue I reported some time ago? visit-dav/visit#12250

@angel-devicente, from our prespective, the two are related because they are built by means other than build_visit. IMHO, Spack is still experimental and building your own as Vladimir is doing makes it difficult for us to reproduce the behavior. We simply do not have the resources to support every possible configuration and so have to make choices.

angel-devicente commented 3 years ago

@angel-devicente, from our prespective, the two are related because they are built by means other than build_visit. IMHO, Spack is still experimental and building your own as Vladimir is doing makes it difficult for us to reproduce the behavior. We simply do not have the resources to support every possible configuration and so have to make choices.

Of course, I understand. In my case I use Spack because the build_visit script was failing badly in my system (ArchLinux, the problems of living on the bleeding edge, I guess :-) ), but I thought maybe it was not a coincidence that in both cases Pseudocolors of regular variables worked no problem, but VisIt failed the moment we added expressions?

biagas commented 3 years ago

Vladimir,

Can you provide more information about the expressions you are creating that cause VisIt to crash?

markcmiller86 commented 3 years ago

@biagas... one thing I noticed about Vladimir's stack dump is that it appears to be happening when gathering the data on the engine to ship back to the viewer. I wonder...if the user set SR mode to Always, would they still see this error?

markcmiller86 commented 3 years ago

Vladimir...sorry to be so cryptic in my comment here. What I mean is, go to Options->Renderin... and follow these instructions to set Scalable Rendering mode to Always. See if that doesn't somehow work-around this bug.

markcmiller86-visit commented 3 years ago

Thank you for looking into my problem. Here is the extra information you requested.

First, my Visit installations do not make any changes to VTK. Some wrangling is required to get Visit to build, but the changes are miniscule and they don't affect VTK at all. For example, adding vtkRenderingGL2PSOpenGL2 library in one CMakeLists.txt or changing some member functions to use "unsigned char" arguments instead of "char". In my experience, Visit works reliably with stock VTK (currently with Fedora 33, but I have been using it from the time Fedora versions were in single digits).

Kathleen, I am not asking to support Visit on my system. It would be, of course, a waste of time for you and other developers. I am only asking for some analysis from the developers of what could be wrong and some guidance on to how to fix it. I will be looking forward to running Visit to work with stock VTK 9 once that is released for my system.

Second, any expression can cause the crash I reported. For example, I can define an expression as simply a copy of a database variable, and it would crash. I can even use built in expressions such as volume, Jacobian, etc., and it would crash also. However, everything works with database variables.

Third, enabling Scaled Rendering stopped the crashes. I hope this would provide some clue as to the nature of the problem.

Best,

On Wed, Nov 18, 2020 at 1:55 PM Mark C. Miller via visit-users < visit-users@elist.ornl.gov> wrote:

Vladimir...sorry to be so cryptic in my comment here. What I mean is, go to Options->Renderin... and follow these instructions https://visit-sphinx-github-user-manual.readthedocs.io/en/develop/gui_manual/Preferences/Rendering_Options_Window.html?highlight=scalable%20rendering#advanced-rendering-options to set Scalable Rendering mode to Always. See if that doesn't somehow work-around this bug.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/visit-dav/live-customer-response/issues/179#issuecomment-729917777, or unsubscribe https://github.com/notifications/unsubscribe-auth/AOLAYZCHLYREZKG2EMXJZATSQQREPANCNFSM4T2EYCCQ .

VisIt Users Wiki: http://visitusers.org/ Frequently Asked Questions for VisIt: https://wci.llnl.gov/simulation/computer-codes/visit/FAQ.html To Unsubscribe: send a blank email to visit-users-unsubscribe@elist.ornl.gov More Options: https://elist.ornl.gov/mailman/listinfo/visit-users

-- Vladimir Florinski

markcmiller86 commented 3 years ago

Third, enabling Scaled Rendering stopped the crashes.

Ok, good to know. Thanks. That is because it winds up taking a different logic path during that data gathering step on the engine and avoiding whatever was causing the segv altogether. The downside is that you won't have GPU-accelerated rendering for the smaller datasets that might possibly benefit from it. I am sure when we upgrade to VTK-8.2 (or 9), we'll be able to reproduce and correct the problem then.