search

desihub / desitest

Testing coordination for DESI code
BSD 3-Clause "New" or "Revised" License
0 stars 1 forks source link

Halo -> MTL wrapper #26

Open michaelJwilson opened 5 years ago

michaelJwilson commented 5 years ago

There's a need for a simple transition from halo catalogues (at the right target density) to an MTL file format that skips the cost/expertise of running select_mock_targets and generates input for assignment. Ideally, this should also be implemented by minitest to ensure everything is kept up to date with the (MTL) format choice.

This facilitates the GQC/Sim group simply creating desi-like output for fiber assignment and as a start to the code base.

Could well be questions on how to best select targets etc, but the aim is to be simple than select_mock_targets, which people have been having trouble running successfully.

Interested parties: Mariana V Srivatsan Albert

That may have something on this already.

sbailey commented 5 years ago

I'm not sure whether we need a function that tries to meet all cases, vs. just documentation with a worked example that can be adapted to a specific analysis.

See the FiberAssign tutorial in desihub/tutorials as a starting point for how to create an MTL file from scratch with the minimal set of columns required. Basically we'd need code to take a halo catalog and add those columns.

mmarianav commented 5 years ago

Yes now we are digging on this option, creating the MTL file from scratch , I think for our purposes is enought ,the interest to run select_mock_targets is to generate again Dark Sky Mocks being completly compatible with codes (i.e mostly fiber assign). I will update on this soon.

michaelJwilson commented 5 years ago

Sounds good. Not sure what you mean by meets all cases, given that a mock will largely be ra, dec, and redshift. There seems to be a bottleneck where it takes a lot of compute time, and even John, to keep mocks up-to-date with data systems.

I do wonder about a 'pro', but lightweight, version of select_mock_targets. i.e the goal shouldn't be just to create something that looks like an MTL, but that produces realistic assignment - requiring standards, skies, etc? This also draws in expected target densities etc, which could be enforced in a slightly more developed script.

Anyway, happy with whatever works for everyone.

sbailey commented 5 years ago

+1 for a lightweight-but-pro version of select_mock_targets that is aimed at mocks through fiberassign including standards and sky, but is not aimed at simulations beyond that, e.g. not trying to generate realistic distributions of magnitudes that pass target selection, etc.

moustakas commented 5 years ago

Apologies for the radio silence. I'll take this as-assigned, but I'm preparing for some travel next week and so it may be a couple weeks before I can get something functional into the software stack. Will post additional questions here but in the meantime feel free to add feature requests.

sbailey commented 5 years ago

I suggest that QGC/sim groups should identify someone within their group to take ownership of this, and then work with experts like John, Jaime, Sarah, Ted, myself to make it happen. Over multiple conversations there appears to be a misconception (due to lack of documentation and lack of broadly spread expertise) that everything has to run through select_mock_targets and make_mtl, which is more complex than many cases need. We should train the working groups in the underlying tools so that all requests don't have to route via @moustakas .

Put another way, much of this request is to make a simplified mocks -> fiberassign that doesn't need @moustakas, so we should try to do that without needing @moustakas. We can save him for the more complex end-to-end simulations that do need consistency between target selection, magnitudes, spectra, etc.

mmarianav commented 5 years ago

@sbailey I agree that this it is probably a misconception, that everything has to run through select_mock_targets I am happy to take this on me (I am working with my student @CesarArroyo09 who is preparing a jupyter notebook with the kind of things the clustering goup needs that is mostly running fiber assignement on randoms, etc...)

sbailey commented 5 years ago

Thanks @mmarianav and @CesarArroyo09 . When you've had a chance to try https://github.com/desihub/tutorials/blob/master/FiberAssign.ipynb and try adapting the recipe for mocks, I'd be happy to schedule a zoom meeting to discuss any problems or questions. There is also the distinct possibility that we "experts" also have misconceptions about what is really needed and how "easy" or not this is. :)