Closed hz-xiaxz closed 1 month ago
Hi,
Basically this should work. But what you have run into for some reason with that error is Parallel run mode. Usually that should not happen if you use MPIScheduler
without setting the ranks_per_run
option, I’ll investigate it later.
But Parallel run mode would enable you to do something similar: you can MPI-parallelize your own section of the code. That is why it asks you to implement this version of measure
and other methods, which get a communicator. Maybe an elegant way to do this in the first place is to use parallel run mode to put that whole update loop into your code rather than in the jobfile.
function Carlo.sweep!(mc::MC, ctx::MCContext, comm::MPI.Comm)
sample_gradients_in_parallel!(mc, ctx, comm)
if time_to_update_parameters()
update_parameters!(mc, comm)
end
end
The downside is that you have to write some MPI code yourself, but the result will be faster because you don’t have to write everything to disk every time.
Thanks for your prompt reply! I'll further check the Parallel run mode out.
One more question here, do you mean that I should wrap the updateConfiguration
(the old sweep!(mc, ctx)
function), the measure!
functions into the new sweep!(mc,ctx,comm)
function? And thus new sweep!(mc,ctx,comm)
will only update the parameters in a sweep
?
The new sweep function has to do everything the old one does and then some manual communication to exchange the data for the gradients before updating the parameters and syncing them across the workers. (Probably you can get away with an MPI.gather and MPI.broadcast)
Instead of using the Carlo measurements for accumulating the gradients, you would have to average them manually.
-------- Original Message -------- On 7/28/24 04:00, LeoXia wrote:
Thanks for your prompt reply! I'll further check the Parallel run mode out. One more question here, do you mean that I should wrap the updateConfiguration (the old sweep!(mc, ctx) function), the measure! functions into the new sweep!(mc,ctx,comm) function? And thus new sweep!(mc,ctx,comm) will only update the parameters in a sweep?
— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.Message ID: @.***>
Got it, thanks for you help! And I think this issue can be closed.
Hello! Thanks for your robust and nice-documented package! In Variational Monte Carlo tasks one needs to perform Monte Carlo calculation in sequence. That's because the variational parameter of the subsequent task should relate to the MC results from the previous one. To employ this, I hacked the job file as follow:
It runs all well with
SingleScheduler
, but withMPIScheduler
it throws errors likeStacktrace:running in parallel run mode but measure(::MC, ::MCContext, ::MPI.Comm) not implemented
. I wonder if MPI in this job script is simplyNot Supported
or can be done with the configuration ofMPI.comm
? I still use thempirun -n 96 julia ./job.jl
command line script to run this job.To put it more further, will it be more elegant to add this feature into the
JobTools
module? Like lettingtasks=make_tasks(tm)
have differentparallel
andsequential
modes? I'm not quite sure if it is easy to be done.