search

illinois-ceesd / mirgecom

MIRGE-Com is the workhorse simulation application for the Center for Exascale-Enabled Scramjet Design at the University of Illinois.
Other
11 stars 19 forks source link

Mismatched send receives in parallel lazy #913

Open anderson2981 opened 1 year ago

anderson2981 commented 1 year ago

It's possible to generate non-working parallel lazy code within the coupling. The issue appears to be related to not using data on one side of the coupled domain that is exchanged with the other side. For example, if I need the temperature_gradient on the fluid side, then I need to exchange temperature on the wall side. If the function doesn't actually use the wall-side temperature, then it's not available for exchange and I get mis-matched send/receives.

A function that gets around this issue can be found here:

https://github.com/illinois-ceesd/drivers_y3-prediction/blob/8e9fb061bbdb21354933df31d526a0014cc5fbb8/y3prediction/prediction.py#L2321

This function grabs the necessary data, in this case the gradients, and returns them to the caller for viz. It also uses a call to the coupled Navier-Stokes operator and does a dummy update of the conserved variables cv, wv. These are returned, but never used outside this compiled function.

inducer commented 1 year ago

That's indeed a difference between sends and receives: Receives disappear if unused, but sends do not. It would of course be best to not communicate data that isn't used, but if that's not practical, we can probably think of a robust way to address this. I've filed https://github.com/inducer/pytato/issues/441 in pytato to discuss this.

cc @majosm

anderson2981 commented 1 year ago

I'm not sure this quite addresses the current situation. From my perspective, I need the data on the fluid side and setup a data exchange (send/receive) to get it. So I need both the send and the receive. The issue appears to be that if data is not used on the side that needs to do the sending, then it doesn't exist in the DAG and can't be be sent at all.