PetscScalar conversion errors when Complex type enabled

[cticenhour]

As of 8e353b8, I am experiencing build errors in petsc_vector.h and petsc_dm_wrapper.C when --enable-complex is used as a configure flag. Just in case this is a bug (hopefully just a configuration error on my part...) I wanted to submit a bug report.

My machine configuration info is below. Also of note - I built PETSc and libMesh as MOOSE submodules using the update_and_rebuild_ MOOSE scripts with modified flags (I am exploring what it would take to properly configure PETSc and libMesh for a possible future complex-compatible version of MOOSE....the MOOSE Team had a recent user question about it and I'd like it for my own electromagnetics research using MOOSE).

OS: Mojave 10.14.3 PETSc version: 3.10.2 Compiler: clang 6.0.1

libMesh configuration flags: --with-methods=opt oprof dbg --prefix=/Users/icenct/projects/moose/scripts/../libmesh/installed --enable-silent-rules --enable-unique-id --disable-warnings --with-thread-model=openmp --disable-maintainer-mode --enable-metaphysicl-required --enable-complex

PETSc configuration flags: --download-hypre=0 --with-ssl=0 --with-debugging=no --with-pic=1 --with-shared-libraries=1 --with-cc=mpicc --with-cxx=mpicxx --with-fc=mpif90 --download-fblaslapack=1 --download-metis=1 --download-ptscotch=1 --download-parmetis=1 --download-superlu_dist=1 --download-mumps=1 --download-scalapack=1 --CC=mpicc --CXX=mpicxx --FC=mpif90 --F77=mpif77 --F90=mpif90 --CFLAGS='-fPIC -fopenmp' --CXXFLAGS='-fPIC -fopenmp' --FFLAGS='-fPIC -fopenmp' --FCFLAGS='-fPIC -fopenmp' --F90FLAGS='-fPIC -fopenmp' --F77FLAGS='-fPIC -fopenmp'

[pbauman]

It's possible I'm blind and not seeing it, but you would need to configure PETSc to build with complex numbers as well with --with-scalar-type=complex in the PETSc configure line. Otherwise it will build for reals.

[roystgnr]

Huh. I'd agree with Paul, except if you're trying to build complex libMesh against real PETSc, how did configure not scream at you before ever giving you a Makefile? We just discussed this as an aside in https://github.com/libMesh/libmesh/issues/2060#issuecomment-470197204

[cticenhour]

Sorry about the confusion! I did forget two PETSc flags (copy and pasted this from my build script earlier and missed a line). The final flags were --with-scalar-type=complex and --download-cmake

@roystgnr libMesh configure did scream at me until making these changes :-)

[jwpeterson]

I am experiencing build errors in petsc_vector.h and petsc_dm_wrapper.C

What are the errors?

[cticenhour]

They were mostly Number to Real/double conversion errors. I wasn't sure how to include all of the error messages in my original posting, but I figured it out below. I built this on my laptop at home today and reproduced the same issues (on a Mojave MacBook Pro, same configuration as the desktop). One additional error in mesh_function.C also appeared.

../../tests/mesh/mesh_function.C:123:14: error: no viable conversion from 'libMesh::Number' (aka 'complex<double>') to 'libMesh::Real' (aka 'double')
        Real mesh_function_value = projection_function(*node,
             ^                     ~~~~~~~~~~~~~~~~~~~~~~~~~~

../src/solvers/petsc_dm_wrapper.C:611:36: error: non-const lvalue reference to type 'SparseMatrix<libMesh::Number>' (aka 'SparseMatrix<complex<double> >') cannot bind to a value of unrelated type 'PetscMatrix<libMesh::Real>' (aka 'PetscMatrix<double>')
          system.projection_matrix(*_ctx_vec[i-1]->K_interp_ptr);
                                   ^~~~~~~~~~~~~~~~~~~~~~~~~~~~
./include/libmesh/system.h:1675:50: note: passing argument to parameter 'proj_mat' here
  void projection_matrix (SparseMatrix<Number> & proj_mat) const;
                                                 ^
In file included from ../src/solvers/petsc_dm_wrapper.C:31:
In file included from ./include/libmesh/petsc_dm_wrapper.h:39:
./include/libmesh/petsc_vector.h:1080:10: error: cannot convert 'const PetscScalar' (aka 'const complex<double>') to 'double' without a conversion operator
  return static_cast<T>(_read_only_values[local_index]);
         ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
./include/libmesh/petsc_vector.h:72:3: note: in instantiation of member function 'libMesh::PetscVector<double>::operator()' requested here
  PetscVector (const Parallel::Communicator & comm_in,
  ^
/opt/moose/llvm-6.0.1/include/c++/v1/memory:3078:32: note: in instantiation of member function 'libMesh::PetscVector<double>::PetscVector' requested here
    return unique_ptr<_Tp>(new _Tp(std::__1::forward<_Args>(__args)...));
                               ^
../src/solvers/petsc_dm_wrapper.C:1027:26: note: in instantiation of function template specialization 'std::__1::make_unique<libMesh::PetscVector<double>, const libMesh::Parallel::Communicator &>' requested here
      _vec_vec[i] = std::make_unique<PetscVector<Real>>(comm);
                         ^
In file included from ../src/solvers/petsc_dm_wrapper.C:31:
In file included from ./include/libmesh/petsc_dm_wrapper.h:39:
./include/libmesh/petsc_vector.h:1103:19: error: cannot convert 'const PetscScalar' (aka 'const complex<double>') to 'double' without a conversion operator
      values[i] = static_cast<T>(_read_only_values[local_index]);
                  ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
./include/libmesh/petsc_vector.h:72:3: note: in instantiation of member function 'libMesh::PetscVector<double>::get' requested here
  PetscVector (const Parallel::Communicator & comm_in,
  ^
/opt/moose/llvm-6.0.1/include/c++/v1/memory:3078:32: note: in instantiation of member function 'libMesh::PetscVector<double>::PetscVector' requested here
    return unique_ptr<_Tp>(new _Tp(std::__1::forward<_Args>(__args)...));
                               ^
../src/solvers/petsc_dm_wrapper.C:1027:26: note: in instantiation of function template specialization 'std::__1::make_unique<libMesh::PetscVector<double>, const libMesh::Parallel::Communicator &>' requested here
      _vec_vec[i] = std::make_unique<PetscVector<Real>>(comm);
                         ^

[jwpeterson]

These all look legit to me. At first I couldn't understand the mesh function unit test one, so I checked the CIVET output and for some reason cppunit isn't found/configured on those builds:

checking for cppunit-config... none
checking whether we can build a trivial CppUnit program... no

so we don't run the unit tests!

The main issue is that the CIVET complex testing is done with --disable-petsc so we didn't have to have a complex-enabled PETSc on all the build boxes. We actually use complex-PETSc enabled builds around here but typically not with metaphysicl/projection matrix code enabled so we haven't run into the compilation errors. If I get some time this weekend I'll look into fixing these... thanks for the report.

[jwpeterson]

So for the complex-enabled testing on CIVET these are the modules loaded:

Currently Loaded Modulefiles:
  1) advanced_modules   4) miniconda          7) cmake
  2) moose/.gcc-7.3.1   5) ccache             8) boost
  3) mpich-gcc          6) autotools

and there's no cppunit module. On the other recipes there's a moose/.cppunit-1.12.1_gcc-7.3.1 module loaded, so that might be a quick and easy thing to fix...

[cticenhour]

Ahh I see. I noticed that complex-enabled PETSc wasn't enabled in the CIVET builds, but I didn't notice cppunit was missing. Glad I could be of some help!