Option to activate/deactivate Distributed Compilation.
This small PR is based on the following comment (DAPP/DaCe Mattermost channel):
I have an unexpected behaviour in DaCe distributed compilation.
Currently, if you have an MPI program, distributed compilation is the default behaviour (as seen in this file). I was expecting that after the loading of the compiled sdfg every rank would do symbol specialization.
Although, this is not the case, i.e. every rank uses the compiled sdfg from rank 0, which specializes its symbols with the values corresponding to rank 0. Therefore, the compiled sdfg loaded by all the other ranks use a wrong sdfg (symbols are not specialized with the values of the correct rank).
To validate this behaviour, I have de-activated the distributed compilation and set dace.config.Config.set("cache", value="unique"). Indeed, this approach works without any issue.
Is there a way to change this unexpected behaviour, i.e. to have by default the distributed compilation but every rank to perform symbol specialization.
To give a bit more context, I am generating an sdfg that uses closures heavily, i.e. all the gt4py fields are defined externally to the sdfg (could that be an issue)?
Option to activate/deactivate Distributed Compilation.
This small PR is based on the following comment (DAPP/DaCe Mattermost channel): I have an unexpected behaviour in DaCe distributed compilation. Currently, if you have an MPI program, distributed compilation is the default behaviour (as seen in this file). I was expecting that after the loading of the compiled sdfg every rank would do symbol specialization. Although, this is not the case, i.e. every rank uses the compiled sdfg from rank 0, which specializes its symbols with the values corresponding to rank 0. Therefore, the compiled sdfg loaded by all the other ranks use a wrong sdfg (symbols are not specialized with the values of the correct rank). To validate this behaviour, I have de-activated the distributed compilation and set
dace.config.Config.set("cache", value="unique"). Indeed, this approach works without any issue. Is there a way to change this unexpected behaviour, i.e. to have by default the distributed compilation but every rank to perform symbol specialization. To give a bit more context, I am generating an sdfg that uses closures heavily, i.e. all the gt4py fields are defined externally to the sdfg (could that be an issue)?