damonge / CoLoRe

CoLoRe - Cosmological Lofty Realizations
GNU General Public License v3.0
17 stars 13 forks source link

Compute and/or write skewers for a region of sky only #29

Open andreufont opened 6 years ago

andreufont commented 6 years ago

It would be quite useful for debugging to have an option to specify a region of sky that you want to focus on.

The lazy version would be to specify a list of HEALPix pixels, or the number of HEALPix pixels to write.

In a more elaborate version you would give some sort of survey footprint and use that. But may be this belongs to a post-process code, and not really in CoLoRe, since its survey will have its own way of defining footprints, masks, and so on.

damonge commented 6 years ago

@andreufont : what would be the gain in doing this? Is it final file size? Because in terms of run time, things would probably not change much, since the full box would still need to be generated.

I'm planning on modifying the code to deal with smaller sky fractions (e.g. select a circle with a given angular radius and simulate only the box that encloses the associated lightcone). This should imply a noticeable improvement in run time, but it's a major modification that will probably take some time to happen.

andreufont commented 6 years ago

@damonge : it is just an issue of disk space. Sometimes you just want to get several thousands of skewers on a small region of the sky (few hundred square degrees or so), for instance to do small tests before accepting a PR. And in the current code, if you want to have the right density of skewers, you end up writing a single huge file with hundreds of Gbs of data, what is quite painful. But now that you have started a new branch to write a file per HEALPix pixel, this is not as much of an issue anymore. You can run your tests on one of the files, and ignore the rest.

damonge commented 6 years ago

OK, I see. I'll leave this open for the time being then, in case you need something more efficient. One possible nice thing to implement would be for the user to be able to choose the Nside of the pixels into which the catalog gets divided (right now this is decided by the code automatically in terms of the number of nodes used to run it)

andreufont commented 6 years ago

In a previous message you mentioned that the minimum value used was Nside=2, right? This would be enough, since it would mean that each file would have approximately 1/48 th of the skewers, i.e., a few Gb. But yes, it would be even better if we could specify Nside, as requested in #24, but it is not really a priority.