[MPI] Using communication plan at python level.

[marcnunezc]

Hi @jcotela,

We are trying to run a simple MPI case where we compute the nodal area of a mesh at python level, to get familiar with MPI communications and utilities, but without using the Synchronize() methods. So far I had the example working by communicating every rank with each other:

(simplified to get to the point)

self.size = self.communicator.Size()
for rank in range(0,self.size):
    areas_to_send=....
    areas_to_receive  = self.communicator.SendRecvDoubles(areas_to_send, rank, rank)

Now I would like to use a correct/efficient communication plan. Is it possible to access the communication plan from the ParallelFillCommunicator or from the CommunicationColoringUtilities in python? Or is there any other function that I shall use?

Also, seems like the method NeighbourIndices() of the communicator is accessible from python, but I can't seem to read anything from it.

For instance, if I try to print it:

  File "test_nodal_area_good.py", line 110, in _SynchronizeValuesAtEveryRank
    print(self.model_part.GetCommunicator().NeighbourIndices())
TypeError: Unable to convert function return value to a Python type! The signature was
    (self: Kratos.Communicator) -> boost::numeric::ublas::vector<int, boost::numeric::ublas::unbounded_array<int, std::allocator<int> > >

Regards,

[philbucher]

I have several comments:

• I was using the GraphColoringProcess when I needed to compute the communication plan, not sure what is the difference to the other one (@RiccardoRossi ?)
• Are you sure you want to do this in python? MPI is already a huge pain in C++, having another layer inbetween will complicate things even more.
• The functions you are using (as of now) COPY the values when going from C++ to python and vise-versa
• afaik @jcotela will (or already did?) implement data exchange like you want, also for asynchronous communication => then you don't need any comm-plan, it should be also faster. I.e. you would use a "simple" fct that does all the stuff in the background (sth like this)
• If I were you I would use the provided things as much as possible, this will make your life much easier

[marcnunezc]

Hi @philbucher, sorry if was not clear from my first message!

This is essentially an exercise proposed by @RiccardoRossi to get familiar with MPI functionalities. By no means it is going to be used in an actual case or implementation.

This is the reason why we didn't want to use the Synchronize() functions (in order to do it "by hand" and implement the "background stuff" you mention) and the reason we are doing it in python.

As you say there are several coloring/scheduling processes so I was not sure what is the one meant to be used and if there's one available at python level.

[philbucher]

ah I understand

In this case I recommend to try out sth without Kratos What helped me a lot was this: https://mpitutorial.com/tutorials/

[jcotela]

Hi @marcnunezc, the way to access an existing communication pattern is as you where doing, by asking the communicator to give you the Colors, neighbor indices and so on... If it is not working from python, we'll need to take a look.

The asynchronous implementation that @philbucher was mentioning already exists in mpi_communicator and was implemented by @roigcarlo (check the functions for sending nodes/elements to a different process, I think they are all called Transfer...). @philbucher and I want to implement that in a more generic way (not just for nodes) in the mpi_data_communicator, but that part is still in my todo list.

Also, if you are looking into having your own communicators, please check #5489, the functionalities are already there but having feedback from users would be valuable.

[jcotela]

Hi @marcnunezc what is the state of this? Was #5680 enough for your needs or is there something else blocking resolution of this issue?

[marcnunezc]

Hi @jcotela,

Yes, #5680 was enough to solve this issue and we can close this in my opinion.

Thanks for your help!