rainwoodman
commented
2 years ago What you described looked alright to me from the high level.
I am somewhat worried about the line calling readout. I recall usually we shall deal with decomposition and particle exchange. But it could be that the readout function handles this internally. Could you take a look which readout function you are calling?
Could you confirm quickly if the difference not within round off error?
On Mon, Oct 10, 2022 at 3:32 AM Andrej Obuljen @.***> wrote:
Hi,
I’m trying to make this code https://github.com/andrejobuljen/Hi-Fi_mocks run in parallel. It's built upon nbodykit and in essence does the following:
- Generates initial linear density field (d_ic) on the mesh,
- Computes displacement field on the mesh,
- Generates uniform catalog of particles and computes d_ic at each particles position,
- Shifts each particle by the displacement field at its position,
- Assigns shifted particles to the mesh by weighting each particle by d_ic to obtain the output d1 field.
Ideally I would like to obtain the same output d1 field when I run it on a single core and when I run it on multiple cores. Below is a the core function which computes d1. I then save it doing FieldMesh(d1).save(…) and I get different output fields when I run with: python code.py or with mpirun -np 4 python code.py. The output fields look similar and I'm guessing what happens is that I'm only saving the field from a single core...
def generate_d1(delta_ic, cosmo, nbar, zic, zout, plot=True, weight=True, Rsmooth=0, seed=1234, Rdelta=0, comm=None):
scale_factor = 1/(1+zout) Nmesh = delta_ic.Nmesh BoxSize = delta_ic.BoxSize[0] prefactor = cosmo.scale_independent_growth_factor(zout)/cosmo.scale_independent_growth_factor(zic) disp_f = [get_displacement_from_density_rfield(delta_ic, component=i, Psi_type='Zeldovich', smoothing={'R': Rsmooth,}) for i in range(3)] Nptcles_per_dim = int((nbar*BoxSize**3)**(1/3)) pos = UniformCatalog(nbar, BoxSize=BoxSize, seed=seed)['Position'] N = pos.shape[0] dtype = np.dtype([('Position', ('f8', 3)), ('delta_1', 'f8')]) catalog = np.empty(pos.shape[0], dtype=dtype) catalog['Position'][:] = pos[:] del pos catalog['delta_1'][:] = delta_ic.readout(catalog['Position'], resampler='cic')*prefactor catalog['delta_1'][:] -= np.mean(catalog['delta_1']) displacement = np.zeros((N, 3)) for i in range(3): displacement[:, i] = disp_f[i].readout(catalog['Position'], resampler='cic') displacement *= prefactor catalog['Position'][:] = (catalog['Position'] + displacement) % BoxSize del displacement, disp_f catalog = ArrayCatalog(catalog, BoxSize=BoxSize * np.ones(3), Nmesh=Nmesh, comm=comm) d1 = catalog.to_mesh(value='delta_1', compensated=True).to_real_field() return d1I tried going over the instructions to parallelise, but unfortunately nothing worked so far, so I was just wondering if there is a quick hack to make it work, that would be great. Please let me know if more info is needed.
Thanks and bests, Andrej
— Reply to this email directly, view it on GitHub https://github.com/bccp/nbodykit/issues/672, or unsubscribe https://github.com/notifications/unsubscribe-auth/AABBWTFBAW6773KJOICLJNLWCPWDHANCNFSM6AAAAAARBGZRWE . You are receiving this because you are subscribed to this thread.Message ID: @.***>
andrejobuljen
Hi,
I’m trying to make this code run in parallel. It's built upon nbodykit and in essence does the following:
Ideally I would like to obtain the same output d1 field when I run it on a single core and when I run it on multiple cores. Below is a the core function which computes d1. I then save it doing FieldMesh(d1).save(…) and I get different output fields when I run with: python code.py or with mpirun -np 4 python code.py. The output fields look similar and I'm guessing what happens is that I'm only saving the field from a single core...
I tried going over the instructions to parallelise, but unfortunately nothing worked so far, so I was just wondering if there is a quick hack to make it work, that would be great. Please let me know if more info is needed.
Thanks and bests, Andrej