How to link a parallelization customized code written by PETSc to FreeFem++?

[zcwang6]

Dear FreeFem++ developers, Thank you for developing such an impressive program. I have been using it for about a year and I am impressed by the powerful plugins of FreeFem. I am trying to link a parallelization cpp code written by PETSc into FreeFem but I have two issues: The first issue — for sequential cpp code, the steps are

• (a) making cpp function
• (b) finding a suitable operator for this function

• (c) adding this function into FreeFem table by loading the initialize function.

  I assumed the parallel case should follow the same procedures. However, PETSc needs to call PetscInitialize(&argc, &argv, PETSC_NULL, help) function at the beginning. My understanding is that you did the initialization by creating a different init function in the PETSc_code.hpp(\plugin\mpi). However, I do not know where the pointer “pkarg” comes from.

  

The second issue —— I assumed the steps of generating the .dylib for FreeFem++ are the general way for other cpp functions. So the steps I used are :

• (a)¯mpicxx -c -g -DNDEBUG -O3 -O3 -O3 -std=c++11 -DBAMG_LONG_LONG -DNCHECKPTR -fPIC -I/usr/local/ff-petsc/real/include ‘-I/usr/local/lib/ff++/4.5/include’ ‘petsc_free_port_1.cpp’ -lpetsc
• (b)¯mpicxx -bundle -undefined dynamic_lookup -g -DNDEBUG -O3 -O3 -O3 -std=c++11 -DBAMG_LONG_LONG -DNCHECKPTR -fPIC -I/usr/local/ff-petsc/real/include ‘petsc_free_port_1.o’ -o petsc_test_port1.dylib ‘-lpetsc’.

I was wondering whether the above steps are correct. Thank you in advance for helping me with these questions.

[prj-]

I'll start with the second issue. If at the beginning of your .cpp you put the following line:

//ff-c++-LIBRARY-dep: cxx11 petsc mpi

You'll be able to simply do the following: ff-c++ -auto petsc_free_port_1.cpp to properly compile and link your plugin, cf. this example.

For the first issue, I would not bother with PetscInitialize and stuff. In your .edp, just do load "PETSc", and it will be taken care of, so you don't have to worry about this in your .cpp. You are more than welcome to contribute to PETSc-code.hpp where everything PETSc-related resides. In which case you can just make a pull request and I'll get your stuff merged into FreeFEM.

Let me know if something is not clear to you.

[prj-]

You can check the new example I've just pushed on the develop branch: function-PETSc.edp, a FreeFEM script loading a user-written plugin, and function-PETSc.cpp, a plugin with user-written PETSc code.

[zcwang6]

I'll start with the second issue. If at the beginning of your .cpp you put the following line:

//ff-c++-LIBRARY-dep: cxx11 petsc mpi

You'll be able to simply do the following: ff-c++ -auto petsc_free_port_1.cpp to properly compile and link your plugin, cf. this example.

For the first issue, I would not bother with PetscInitialize and stuff. In your .edp, just do load "PETSc", and it will be taken care of, so you don't have to worry about this in your .cpp. You are more than welcome to contribute to PETSc-code.hpp where everything PETSc-related resides. In which case you can just make a pull request and I'll get your stuff merged into FreeFEM.

Let me know if something is not clear to you.

Thanks for your quick reply, I will try your suggestions right now. Thanks

[zcwang6]

You can check the new example I've just pushed on the develop branch: function-PETSc.edp, a FreeFEM script loading a user-written plugin, and function-PETSc.cpp, a plugin with user-written PETSc code.

The issue I got from ff-c++ -auto petsc_function.cpp is like this. I have GCC 9.3.0_1 instead of 9.3.0, and lgfortran is unavailable on my computer. I usually manually go and change this command. I don't know if there any way to set it appropriately. Thank you

[prj-]

Do you compile FreeFEM yourself or use a packaged version? You should be able to fix this by finding and editing one of these files (I can't tell you where they are since it will depend on your installation, my best bet is to look where the ff-c++ file is): WHERE_LIBRARY-config or WHERE_LIBRARY-download and remove -lgfortran.

[zcwang6]

Do you compile FreeFEM yourself or use a packaged version? You should be able to fix this by finding and editing one of these files (I can't tell you where they are since it will depend on your installation, my best bet is to look where the ff-c++ file is): WHERE_LIBRARY-config or WHERE_LIBRARY-download and remove -lgfortran.

Thanks

[zcwang6]

Do you compile FreeFEM yourself or use a packaged version? You should be able to fix this by finding and editing one of these files (I can't tell you where they are since it will depend on your installation, my best bet is to look where the ff-c++ file is): WHERE_LIBRARY-config or WHERE_LIBRARY-download and remove -lgfortran.

I found it out, one more question, why in here, the compiler is cxx, not mpicxx?

[prj-]

mpicxx is nothing else than a wrapper on top of cxx, that includes the MPI headers and links the MPI libraries.

$ mpicxx -show
clang++ -Wl,-flat_namespace -Wl,-commons,use_dylibs -I/usr/local/Cellar/mpich/3.3.2/include -L/usr/local/Cellar/mpich/3.3.2/lib -lmpicxx -lmpi -lpmpi

So even when you use mpicxx, your compiler is in fact cxx, with the proper includes and libraries. That's what FreeFEM does. It always uses cxx, and if you have a mpi dependence in ff-c++-LIBRARY-dep, it will add the includes and libraries. See for example the OpenMPI FAQ for more details about wrappers, the same applies to MPICH and all other MPI implementations.

[zcwang6]

mpicxx is nothing else than a wrapper on top of cxx, that includes the MPI headers and links the MPI libraries.

$ mpicxx -show
clang++ -Wl,-flat_namespace -Wl,-commons,use_dylibs -I/usr/local/Cellar/mpich/3.3.2/include -L/usr/local/Cellar/mpich/3.3.2/lib -lmpicxx -lmpi -lpmpi

So even when you use mpicxx, your compiler is in fact cxx, with the proper includes and libraries. That's what FreeFEM does. It always uses cxx, and if you have a mpi dependence in ff-c++-LIBRARY-dep, it will add the includes and libraries. See for example the OpenMPI FAQ for more details about wrappers, the same applies to MPICH and all other MPI implementations.

Thanks for this knowledge, I am not really good at the shell script file

I went into ff-c++ and changed the FFFLIBS from gcc 9.3.0 into 9.3.0_1 and removed the -lgfortran at the end. it seems like it still gives the same error. Thanks

[prj-]

I never told you to edit this file. I mentioned WHERE_LIBRARY-config and WHERE_LIBRARY-download.

[zcwang6]

I never told you to edit this file. I mentioned WHERE_LIBRARY-config and WHERE_LIBRARY-download.

sorry for that, is that a script as well. sorry for this boring question

[prj-]

Please read again my previous answer.

[zcwang6]

Please read again my previous answer. Thank you

[zcwang6]

Please read again my previous answer.

It solved, thanks for your time, I can run your example now. I will try to read KN<> type data in my petsc cpp code and run it. I will let you know if that works. I apologized for missing the point of your former message. Also thanks for the excellent job you did for freefem. I will close this issue after its working.

[prj-]

No problem, I'm glad it's working. Feel free to ask for more help if your code doesn't run appropriately. Otherwise, if you think your code could benefit to others, feel free to create a pull request.

[zcwang6]

No problem, I'm glad it's working. Feel free to ask for more help if your code doesn't run appropriately. Otherwise, if you think your code could benefit to others, feel free to create a pull request.

sure, thanks

[frederichecht]

please install gfortran with brew ..

Le 14 mai 2020 à 17:20, zcwang6 notifications@github.com a écrit :

You can check the new example I've just pushed on the develop branch: function-PETSc.edp https://github.com/FreeFem/FreeFem-sources/commit/fe943e0700f7c78302bb9b3483762a3b4c8157e3#diff-0d6208859a921eb75994bafb8668d6ee, a FreeFEM script loading a user-written plugin, and function-PETSc.cpp https://github.com/FreeFem/FreeFem-sources/commit/fe943e0700f7c78302bb9b3483762a3b4c8157e3#diff-a6fe64986db5f073e0f46cb97d588175, a plugin with user-written PETSc code.

https://user-images.githubusercontent.com/30075449/81952096-0a803080-95d4-11ea-8d20-272c248f4666.pngThe issue I got from ff-c++ -auto petsc_function.cpp is like this. I have GCC 9.3.0_1 instead of 9.3.0, and lgfortran is unavailable on my computer. I usually manually go and change this command. I don't know if there any way to set it appropriately. Thank you — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/FreeFem/FreeFem-sources/issues/137#issuecomment-628705159, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABLPNDG5PA3PPHYEXNFGQTDRRQD3BANCNFSM4NAXTQUA.

[prj-]

No, do not install gfortran if you don't need Fortran...

[zcwang6]

No, do not install gfortran if you don't need Fortran...

Sure

[zcwang6]

No, do not install gfortran if you don't need Fortran...

Hey, could you please tell me what is "in->operator PetscScalar()" doing here. I think you are passing the address of KN<> into PETSc:: Vec. I also trying to pass an address of Aarm:: vec into PETSc:: Vec in my cpp code. (I pass the address of arma:: vec directly) But I got "candidate function not viable: no known conversion from 'arma::vec' (aka 'Col') to 'const PetscScalar ' (aka 'const double ') for 5th argument PETSC_EXTERN PetscErrorCode VecCreateMPIWithArray(MPI_Comm,PetscInt,PetscInt,PetscInt,const PetscScalar[],Vec);" I think you are doing the right thing, but I can not understand here.

[zcwang6]

No, do not install gfortran if you don't need Fortran...

Hey, could you please tell me what is "in->operator PetscScalar()" doing here. I think you are passing the address of KN<> into PETSc:: Vec. I also trying to pass an address of Aarm:: vec into PETSc:: Vec in my cpp code. (I pass the address of arma:: vec directly) But I got "candidate function not viable: no known conversion from 'arma::vec' (aka 'Col') to 'const PetscScalar ' (aka 'const double _') for 5th argument PETSC_EXTERN PetscErrorCode VecCreateMPIWithArray(MPIComm,PetscInt,PetscInt,PetscInt,const PetscScalar[],Vec);" I think you are doing the right thing, but I can not understand here.

I guess you just passed the address of the first element of KN<> into PETSc:: Vec, right? Thank you.

[prj-]

You are right. This is not a FreeFEM question, but either a PETSc or Armadillo one. Here is what I get with a quick Google search.

[zcwang6]

You are right. This is not a FreeFEM question, but either a PETSc or Armadillo one. Here is what I get with a quick Google search.

Thanks, I can transfer the data between Armadillo and PETSc now. thank you.

[zcwang6]

You are right. This is not a FreeFEM question, but either a PETSc or Armadillo one. Here is what I get with a quick Google search.

Another question here, when I use ffddm method in FreeFem++, I am wondering that how can I get the correct gradient of my variables by dx(),dy(),dz() functions, for example, calculating the stress tensor from the displacement field (u,v,w). I noticed that if I calculate it on the sub-domain directly, it yields the wrong results. So, I calculated it globally and then bring it back into the local sub-domain by Rih[mpirank] matrix, but this way lost its benefit of parallel computing. Is there any way to calculate the derivative locally in FreeFem++?

Thank you.

[prj-]

This is clearly suboptimal. You can do that using the PETSc interface.

Mat A;
createMat(Th, A, Pk)
fespace Vh(Th, Pk);
Vh yourGradient;
exchange(A, yourGradient[], scaled = true);

See this example. There is probably a similar approach with ffddm, but this has nothing to do with the issue at hand, so it would be best to close this and post another issue if you want to use ffddm instead of PETSc.

[zcwang6]

suboptimal

Thanks. I will open another issue about it.