Skewer outputs from different nodes do not correspond to particular HEALPix pixel

[andreufont]

I thought that all skewers of a particular HEALPix pixel would be contained in the same output file, written by a certain node. But it looks like different skewers from the same HEALPix pixel are stored in different files.

I can see this clearly when plotting the angular position of the quasars in two different files, running something like the code below. Note that this might be related to the fact that some quasars have Dec=NaN, and some quasars have DEC < -90.0.

import numpy as np from astropy.io import fits import matplotlib.pyplot as plt

def prune_catalog(filename): hdulist = fits.open(filename)

get quasar catalog

initial_catalog = hdulist[1].data
# some quasars have Dec=NaN, let's get rid of them for now
bad_dec = np.isnan(initial_catalog['DEC']) | (initial_catalog['DEC'] < -90.0) | (initial_catalog['DEC'] > 90.0)
good_dec = np.invert(bad_dec)
good_dec = np.where(good_dec)[0]
catalog = initial_catalog[good_dec]
print('initial size',len(initial_catalog))
print('final size',len(catalog))
return catalog

basedir='/Users/font/Projects/DESI/desi_lya_mocks/Sims/NERSC_Lya4096/output_colore/' qso0=prune_catalog(basedir+'out_srcs_s0_0.fits') qso1=prune_catalog(basedir+'out_srcs_s0_1.fits') plt.plot(qso0['RA'],qso0['DEC'],'o') plt.plot(qso1['RA'],qso1['DEC'],'o') plt.show()

[andreufont]

I attach two plots from a recent N=8192 run, showing the quasar positions from two files (node=0 and node=1). Since there were 256 nodes, I was expecting each node to print a file with 3 HEALPix pixels (nside=8 gives 768 pixels), but as you can see in the first plot each node covers a strange part of the footprint.

Moreover, if I look at the actual HEALPix pixels in the file (with nside=8) I get the following funky looking plot (note that I zoom in to RA < 100 degrees).

Clearly more than six pixels in the two nodes...

[andreufont]

I reproduce the problem when using only 2 MPI nodes. I add below the HEALPix pixels (Nside=1, 12 pixels) covered by node 0 and 1:

Clearly HEALPix pixels are split into different nodes.

[damonge]

@andreufont beautiful plots XD. I'm investigating the NaNs now. Note that there is a minimum Nside!=1, so that might explain why several Nside=1 pixels are distributed over several nodes. There's still clearly something fishy going on, so will report back ASAP

[andreufont]

@damonge - Where is this minimum of Nside !=1? In CoLoRe? What does the code do, in that case? Does it crash, or do something crazy silently? In any case, I don't think this is the case, since CoLoRe uses a value of Nside to match whatever resolution the box has, and it is much higher than Nside=1. What I meant by Nside=1 in the plot is not what CoLoRe internally does, but what I use to make the plot.

[damonge]

Basically, the base Nside is at least 2 (not 1), so even with 2 nodes, each node will contain a set of Nside=2 pixels. This means that if you look at Nside=1 pixels, they will naturally be split across nodes no matter what. Your first plot is definitely worrying though.

[damonge]

Seems fixed after fixing #25