pycbc_faithsim: Spurious Faithfulness Calculations?

[zachetienne]

In short: The two virtually identical waveforms compared here (red & green) outputagreement.pdf apparently yield a faithfulness of 0.197 as measured by pycbc_faithsim. Something strange is happening.

Details: I am trying to confirm that SEOBNRv3_opt and SEOBNRv3 generate results that agree to roundoff error so that code review on v3_opt can commence, unlocking v3's full potential for precessing PE. All of my group's preliminary timeseries-based tests (which span many thousands of waveforms) indicate tiny phase and amplitude disagreements throughout IMR, consistent with roundoff-error disagreement.

However, when I tried pycbc_faithsim (from pycbc 1.6.0) with 100,000 random-parameter inputs, I found that some cases generated faithfulnesses as low at 0.2 (!) I found this disturbing and confusing, so I pulled the input parameters from the very worst case. To wit, they are (if using lalsim-inspiral):

-a SEOBNRv3 [OR SEOBNRv3_opt_rk4] -f 19 -M 3.165550041198730469e+01 -m 1.114187002182006836e+00 -X -2.155276983976364136e-01 -Y -2.959555983543395996e-01 -Z 4.699443876743316650e-01 -x -1.100414991378784180e-01 -y 2.937038056552410126e-02 -z -2.348669022321701050e-01

So I plugged these input parameters into lalsim-inspiral and generated the timeseries using lalsimulation/src/lalsim-inspiral. Lo and behold, I cannot even distinguish the two timeseries! (Compare red solid and green dashed lines in attached plot: outputagreement.pdf the rest of the waveform has the same level of agreement.)

So how can we get a faithfulness of 0.197 with a pair of waveforms when, visually speaking, their timeseries are indistinguishable?

To reproduce my result, follow these instructions: 0) Log in to a machine that has pycbc 1.6.0 installed. 1) Download a fresh lalsuite 2) Go into the lalsuite directory and "git checkout 50ddfa5fce0742e50ff3b144169c1b0582b96547" 3) Apply the SEOBNRv3_opt/SEOBNRv3_opt_rk4 patch SEOBNRv3_opt-no_trailing_whitespace.patch.gz 4) Edit lalsimulation/src/LALSimIMRSpinPrecEOB.c , and replace REAL8 EPS_ABS = 1.0e-8; const REAL8 EPS_REL = 1.0e-8; with REAL8 EPS_ABS = 1.0e-10; const REAL8 EPS_REL = 1.0e-10 (This effectively reduces the error from integration to roundoff-level; enabling us to verify that v3opt and v3 agree to roundoff error.) 5) Build and install lalsuite according to instructions here: http://ligo-cbc.github.io/pycbc/latest/html/install_lalsuite.html 6) Grab & unzip my pycbc faithfulness directory, which contains input parameters for pycbc_faithsim checkone.zip 7) Copy over the necessary pycbc scripts from your 1.6.0 installation into the scripts/ subdirectory. 8) In faithsim.sh, fix the directory linking to /home/zachariah.etienne/H1L1-AVERAGE_ASD-1127271617-1027800.txt 9) Run ./faithsim.sh 10) The output will be collected in result-SEOBNRv3optrk4_VS_SEOBNRv3.dat, where the first column is faithfulness (1.968199999999999950e-01).

Please help!

[spxiwh]

lalsuite version 50ddfa5fce0742e50ff3b144169c1b0582b96547 seems unrelated to SEOBNRv3_opt. And indeed there is no waveform called SEOBNRv3_opt_rk4 or SEOBNRv3_opt on that branch. Is the lalsuite tag correct?

[zachetienne]

Whoops, skipped a step! I modified the instructions and attached the patch to add these approximants.

[spxiwh]

Why not apply the latest patch here?

https://bugs.ligo.org/redmine/issues/4849

(which is what I'm installing at the moment after getting a pointer from Andrea)

[zachetienne]

... because I just discovered that the latest patch has an error. I'm working on updating it now.

[zachetienne]

Just fixed the patch against latest master (as of 8 hours ago). Here's the location: https://bugs.ligo.org/redmine/attachments/5443/AGAINST_MASTER__SEOBNRv3_opt__Fix__SpinAlignedEOBversion-to-PrecEOBversion.patch

[spxiwh]

Hi @zachetienne. I managed to reproduce your plots using 50ddfa5fce0742e50ff3b144169c1b0582b96547 and the mentioned patch.

BUT your outputagreement.pdf is not the same waveform as the faithsim used. Specifically you ignored two intrinsic parameters: [the poorly named] coa_phase and inclination. If one sets these to the values specified by the faithsim file, and generates two waveforms directly from PyCBC I see:

https://galahad.aei.mpg.de/~spxiwh/LVC/tmp_zach/test_plot.png

Note that if setting both inclination and coa_phase to 0 the 2 waveforms agree.

Also note the following:

https://github.com/ligo-cbc/pycbc/blob/master/bin/pycbc_faithsim#L69

You probably want to also check h_cross in these calculations. The polarization column of the sim_inspiral could probably used as the appropriate angle here.

[zachetienne]

Very interesting. This must be related to a recent change in LALSimulation; when I performed the faithsim analysis comparing v3 and v3optrk4 a few months ago, the mismatches were tiny and within roundoff error.

Thanks to your explanation, I have confidence that our newest patch against master fixes this issue. In short, there is a new variable called "polariz" within lalsimulation/src/LALSimInspiral.c, which we learned a few days ago must be adjusted by +pi/2. In our patch against 50ddfa5..., we did not properly set this parameter.

After applying our most recent patch, I restarted the 100k waveform faithfulness tests, and so far the results are looking much better, with mismatches between v3opt and v3 within roundoff error again. I'll update this "issues" report once the tests are complete

[zachetienne]

You were correct; the additional parameters were at fault, and now the problem is solved. No issues with pycbc. Hooray!