search

MolSSI-MDI / MDI_Library

A library that enables code interoperability via the MolSSI Driver Interface.
BSD 3-Clause "New" or "Revised" License
31 stars 10 forks source link

Using other efficient libraries for communication #90

Closed zerothi closed 3 years ago

zerothi commented 3 years ago

This library is great. However, there are many attempts to solve the transport problem between different parts.

These may be good as the lower level communication layers:

This would leverage a huge part of the development and should be easier to maintain and extend.

taylor-a-barnes commented 3 years ago

@zerothi Thanks very much for your kind words and for these observations.

Up to this point, development on the MDI Library has heavily focused on the high level API and on functionality that falls outside the scope of a general-purpose communication library (i.e. unit conversions, plugin support, the "node system", etc.). Only approximately 11% of the core MDI Library's code base directly touches low-level, protocol-specific nuances of MPI or socket-based communication. Add in the development time associated with all the other components of the MDI Project (testing, documentation, deployment, the MDI Standard, driver implementations, engine implementations, etc.), and I doubt that more than 2-3% of development time has gone into the low-level communication layer.

That having been said, we would like to revisit the low-level communication details at some point, with the intention of improving the maintainability of the code and enhancing the scalability of the communication in an HPC environment. Existing communication libraries are appealing for this purpose, and Conduit and ADIOS2 are definitely on our radar. I wouldn't be surprised if one / both of them are eventually utilized by MDI.

zerothi commented 3 years ago

Ok, great. It seems that these libraries takes many different transport layers into consideration! :)

Good to know that you have this planned. :)