Comparison of integrate_lightcurve between limbdark and exoplanet

[ianw89]

The strategy I used was to run both codes from Julia with the same configuration multiple times and take the average time (perhaps median is better?)

The orbital configuration isn't changing from run to run - what changes is the number of data points, or integrations, to perform. Period, impact parameter, planet size, and all stellar parameters are held constant for the comparison - thus the 3rd graph in the figure below.

The picture below also shows a couple sample light curves as evidence that the two codes are set up with the same scenario. I manually looked a bunch of light curves but didn't systematically review the parameter space. I could definitely do this as a follow up if desired, and generate a grid of light curve plots or something.

Please give me any feedback I should follow up on before checking in!

[ericagol]

This looks great! @dfm told me that exoplanet currently uses a constant number of sub-exposure time points for the integration, so we are currently comparing apples with oranges. He also mentioned that there is some overhead in calling exoplanet for the first time due to setting up the packages (e.g. Theano). So, we may need his help in using exoplanet for a comparison, or we may have to use a development branch of exoplanet which does use adaptive integration, and only time the inner function which calls the adaptive time steps to get rid of the overhead.

[dfm]

There's definitely going to be a lot of overhead in the way the exoplanet version is calculated here. Every time you call eval it's recompiling the graph and all the ops. Instead, you'll want to use a theano.function to pre-compile the graph. I'm not familiar with how you would do this with Julia, but in Python I would probably use theano.function directly.

All this being said, it's not clear to me what the aim of this experiment is. It's going to be very hard to do an apples-to-apples comparison and it's not clear to me that just demonstrating that Limbdark.jl is faster than exoplanet teaches us anything (the goals of the two projects are quite different, I would say!). Can you say a few more words about what y'all are trying to capture?

[ericagol]

@dfm I wanted to check that the Limbdark.jl adaptive integrator wasn't behaving too inefficiently (I don't care about the speed of the actual light curve component, just the integrator, and any associated overheads for allocating memory, etc.). So, I was hoping to make a comparison to your adaptive simpson routine, but you mentioned that it's in the development branch still.

[dfm]

Ah - I see. I definitely wouldn't use exoplanet as the baseline for this in this case. I've actually found that there are enough edge cases where the polynomial approximations to the orbit break down that I'm not sure that I'll ever release the relevant comparison for what you want. I do have a C++ implementation of all of these ideas here: https://github.com/dfm/vice It has Python bindings and there's an example notebook here: https://github.com/dfm/vice/blob/master/scaling.ipynb But, again, this is pretty much just an experiment so it's not obvious that it's a good baseline!

[ianw89]

@ericagol how do you want to proceed here? Based on this thread it sounds like getting useful comparison in this sort of manner would be quite challenging. Do you think I should instead do some perf auditing on the relevant limbdark code and see if I can find any issues, move on to something else? Or should I try using the C++ implementation dfm mentioned?

[ericagol]

I agree - the test I was hoping to carry out for efficiency of the time-integration will not be straightforward in using exoplanet.

I would be interested in a performance audit of the limbdark code. There is overhead in carrying out the adaptive simpson method, which involves recursively calling the model at refined time points until a certain tolerance is reached. This takes more time compared with simply calling the model at each time point if the time points were known in advance. All to say: I suspect this could be improved more.

However, this code is already fairly efficient, so I would also be interested if you would prefer to move onto another topic that we discussed.