Open jlvdb opened 2 years ago
Given that RAIL now speaks ceci, it might be worth looking into what existing infrastructure from TXPipe could potentially be called in a RAIL stage specifically for the cross-correlation step. @joezuntz has thought about this a bunch and probably has additional ideas for efficiency.
If I remember correctly that YAW is similar to the-wizz in switching from actual correlation function calculations to KDTree matching, then we probably don't need any of the actual TreeCorr stuff. But, I think we do need to figure out how to do proper mask generation/use and generating randoms in a way that is consistent with how other DESC people are doing those operations.
Good points, Alex and Sam, I will check TXPipe and talk to Joe if necessary. I'll try to use as much existing infrastructure as possible unless that requires changing too much of the existing code base.
Implement the original code at
https://github.com/jlvdb/yet_another_wizz
to fit into RAIL and improve the performance to match the scale of the future datasets.Main challenges:
scipy.spatial.cKDTree
to handle inverse distance weights)