Broken silures catalog with no L and no valid relaxed data

[Galadirith]

@llondon6 With the current silures catalog there are no L attributes because I believe it was built before https://github.com/llondon6/nrutils_dev/commit/cd1331897720b7e1c7e4dc7ac1465e801f65e1a2#diff-5c8bebdb8173e0ca200215a18c36663d. This is now an issue because external_sign_convention has now been forced to None in https://github.com/llondon6/nrutils_dev/commit/d9c5d153d720673a890f4df6b22e277d52dbfbb8#diff-30ce82bdc95d9bf98cc05c25af6268c5R2409

https://github.com/llondon6/nrutils_dev/blob/d9c5d153d720673a890f4df6b22e277d52dbfbb8/nrutils/core/nrsc.py#L2409

which triggers https://github.com/llondon6/nrutils_dev/commit/d9c5d153d720673a890f4df6b22e277d52dbfbb8#diff-30ce82bdc95d9bf98cc05c25af6268c5R2622

https://github.com/llondon6/nrutils_dev/blob/d9c5d153d720673a890f4df6b22e277d52dbfbb8/nrutils/core/nrsc.py#L2622-L2630

which has no path that is free of L.

On attempting to rebuild the silures catalog you get failures because of there are no afterjunk attributes in many the silures data directories (for example BAM/q8EqualSpin/q8am085v6D10_96c025 has after-junkradiation-spin1x = ie nothing) which triggers https://github.com/llondon6/nrutils_dev/commit/cd1331897720b7e1c7e4dc7ac1465e801f65e1a2#diff-5c8bebdb8173e0ca200215a18c36663dR179

https://github.com/llondon6/nrutils_dev/blob/cd1331897720b7e1c7e4dc7ac1465e801f65e1a2/nrutils/handlers/bam.py#L179-L192

which subsequently causes https://github.com/llondon6/nrutils_dev/commit/cd1331897720b7e1c7e4dc7ac1465e801f65e1a2#diff-5c8bebdb8173e0ca200215a18c36663dR205 to fail because of a call norm(None).

https://github.com/llondon6/nrutils_dev/blob/cd1331897720b7e1c7e4dc7ac1465e801f65e1a2/nrutils/handlers/bam.py#L205

I will work on a patch for this, but reporting if you get round to this before me, or actually there is something I have done wrong that you can see. Thanks :blush:

[Galadirith]

Confirmed that the silures catalog does work with ae5c1406df9841fad1157db28417e11b822409af although it is possible some of the simulations are broken in a way that subsequent commits have tried to fix (like fixing the frame convention when Jz is negative.

[llondon6]

@Galadirith Thanks for letting me know about this oversight. I've started rebuilding the silures catalog using the commands below:

from nrutils import scbuild
scbuild('silures')

I'll give you a ping and update the repo's version when it's done.

[Galadirith]

@llondon6 Awesome, thanks so much for letting me know 😄 When I tried to do that myself I got failures, which hash are you using?

[llondon6]

Hi Ed,

Sorry about the delayed response. I am using the current master. What's the error message you're getting? My connection to minion was interrupted so it will take a short while longer for me to update the catalog :-/

Lionel

On Wed, Jul 18, 2018 at 9:34 AM, Edward Fauchon-Jones < notifications@github.com> wrote:

@llondon6 https://github.com/llondon6 Awesome, thanks so much for letting me know 😄 When I tried to do that myself I got failures, which hash are you using?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/llondon6/nrutils_dev/issues/38#issuecomment-405854111, or mute the thread https://github.com/notifications/unsubscribe-auth/AHpNtTfeGvTOc1RK8c-C_Vw7onaCNJL-ks5uHvL9gaJpZM4VUB-q .

--

---

L. London

Research Associate, Gravitational Waves

School of Physics and Astronomy

Cardiff University

[Galadirith]

@llondon6 no worries :blush: I just reverted to ae5c140 to get some data out. For the error messages I get, when I try to rebuild as in https://github.com/llondon6/nrutils_dev/issues/38#issuecomment-405835209 (having added my local clone UIB data path to the silures.ini::catalog_dir variable) I get lots of

(bam.py!)>> Warning: The afterjunk spins appear to have been stored incorrectly in the bbh file. Values of NONE will be stored to the current object. 
(scentry!,nrsc.py,@282)>> The following error message will be logged: unsupported operand type(s) for -: 'NoneType' and 'NoneType'

After the build has finished, and I can't load in the data. For example

A = scsearch(keyword='q8am085v6D10_96c025', verbose=True)
y = gwylm(scentry_obj=A[0], lmax=2, verbose=True)

(scsearch)>> Found keyword (=q8am085v6D10_96c025) keyword.
(scsearch)>> Found verbose (=True) keyword.
(scsearch)>> List of keywords or string keyword found: ALL scentry objects matching will be passed. To pass ANY entries matching the keywords, input the keywords using an iterable of not of type list.
## Found 1 possibly degenerate simulations:
[0001][raid] q8am085v6D10_96c025: 1chi0.85-2chi0.85-saa-q8.00   (q8am085v6D10_96c025)

(gwylm)>> Found dt (=0.15) keyword.
(gwylm)>> Found lmax (=2) keyword.
(gwylm)>> Found scentry_obj (=<nrutils.core.nrsc.scentry instance at 0x7fb2277c85f0>) keyword.
(gwylm)>> Found verbose (=True) keyword.
(__init__)>> Using default M_RELATIVE_SIGN_CONVENTION of 1
(gwylm)>> The (extraction_parameter,level) is (2,5), which differs from the config values of (5,5). You have either manually input the non-config values, or the handler has set them by looking at the contents of the simulation directory. 
(__make_lmlist__)>> The following spherical multipoles will be loaded:[(2, 2), (2, -2), (2, -1), (2, 0), (2, 1)]
(load)>> Using the default extraction_parameter of 2
(load)>> Using the default level of 5
(load)>> Loading: psi3col.r2.l5.l2.m2.gz
(load)>> Imposing a default padding of 3 to the data.
(pad_wfarr)>> The shape was (7286, 3). Now the shape is (7289, 3).
AttributeErrorTraceback (most recent call last)
<ipython-input-19-1de5fdd6d8f1> in <module>()
      1 A = scsearch(keyword='q8am085v6D10_96c025', verbose=True)
----> 2 y = gwylm(scentry_obj=A[0], lmax=2, verbose=True)

/root/.local/lib/python2.7/site-packages/nrutils/core/nrsc.pyc in __init__(this, scentry_obj, lm, lmax, dt, load, clean, extraction_parameter, level, w22, lowpass, calcstrain, calcnews, enforce_polarization_convention, fftfactor, pad, __M_RELATIVE_SIGN_CONVENTION__, verbose)
   2209         # Load the waveform data
   2210         if load==True:
-> 2211             this.__load__(lmax=lmax,lm=lm,dt=dt,pad=pad)
   2212 
   2213         #--%%--%%--%%--%%--%%--%%--%%--%%--%%--%%--%%--%%--%%--%%--%%--%%--%%--%%--#

/root/.local/lib/python2.7/site-packages/nrutils/core/nrsc.pyc in __load__(this, lmax, lm, extraction_parameter, level, dt, pad, verbose)
   2419         this.external_sign_convention = None
   2420         for lm in this.__lmlist__:
-> 2421             this.load(lm=lm,dt=dt,extraction_parameter=extraction_parameter,level=level,pad=pad,verbose=verbose)
   2422 
   2423         # Ensuer that all modes are the same length

/root/.local/lib/python2.7/site-packages/nrutils/core/nrsc.pyc in load(this, lm, file_location, dt, extraction_parameter, level, output, pad, verbose)
   2635                 if int(scipy_version.split('.')[1])<16:
   2636                     # Account for old scipy functionality
-> 2637                     external_sign_convention = sign(this.L[-1]) * sign(m) * mode( sign( y_.dphi[msk_] ) )[0][0]
   2638                     initially_msign_matches_wsign = sign(m) == mode( sign( y_.dphi[msk_] ) )[0][0]
   2639                 else:

AttributeError: gwylm instance has no attribute 'L'

[llondon6]

Oh, I see -- that makes sense. I had thought you encountered an error in scbuild. I am now running the build process in a screen on leo. Sorry again about this :-)

On Wed, Jul 18, 2018 at 1:37 PM, Edward Fauchon-Jones < notifications@github.com> wrote:

@llondon6 https://github.com/llondon6 no worries 😊 I just reverted to ae5c140 https://github.com/llondon6/nrutils_dev/commit/ae5c1406df9841fad1157db28417e11b822409af to get some data out. For the error messags I get, when I try to rebuild as in #38 (comment) https://github.com/llondon6/nrutils_dev/issues/38#issuecomment-405835209 (having added my local clone UIB data path to the silures.ini::catalog_dir variable) I get lots of

(bam.py!)>> Warning: The afterjunk spins appear to have been stored incorrectly in the bbh file. Values of NONE will be stored to the current object. (scentry!,nrsc.py,@282)>> The following error message will be logged: unsupported operand type(s) for -: 'NoneType' and 'NoneType'

After the build has finished, and I can't load in the data. For example

A = scsearch(keyword='q8am085v6D10_96c025', verbose=True) y = gwylm(scentry_obj=A[0], lmax=2, verbose=True)

(scsearch)>> Found keyword (=q8am085v6D10_96c025) keyword. (scsearch)>> Found verbose (=True) keyword. (scsearch)>> List of keywords or string keyword found: ALL scentry objects matching will be passed. To pass ANY entries matching the keywords, input the keywords using an iterable of not of type list.

Found 1 possibly degenerate simulations:

[0001][raid] q8am085v6D10_96c025: 1chi0.85-2chi0.85-saa-q8.00 (q8am085v6D10_96c025)

(gwylm)>> Found dt (=0.15) keyword. (gwylm)>> Found lmax (=2) keyword. (gwylm)>> Found scentry_obj (=<nrutils.core.nrsc.scentry instance at 0x7fb2277c85f0>) keyword. (gwylm)>> Found verbose (=True) keyword. (init)>> Using default M_RELATIVE_SIGN_CONVENTION of 1 (gwylm)>> The (extraction_parameter,level) is (2,5), which differs from the config values of (5,5). You have either manually input the non-config values, or the handler has set them by looking at the contents of the simulation directory. (__make_lmlist__)>> The following spherical multipoles will be loaded:[(2, 2), (2, -2), (2, -1), (2, 0), (2, 1)] (load)>> Using the default extraction_parameter of 2 (load)>> Using the default level of 5 (load)>> Loading: psi3col.r2.l5.l2.m2.gz (load)>> Imposing a default padding of 3 to the data. (pad_wfarr)>> The shape was (7286, 3). Now the shape is (7289, 3). AttributeErrorTraceback (most recent call last)

in () 1 A = scsearch(keyword='q8am085v6D10_96c025', verbose=True) ----> 2 y = gwylm(scentry_obj=A[0], lmax=2, verbose=True) /root/.local/lib/python2.7/site-packages/nrutils/core/nrsc.pyc in __init__(this, scentry_obj, lm, lmax, dt, load, clean, extraction_parameter, level, w22, lowpass, calcstrain, calcnews, enforce_polarization_convention, fftfactor, pad, __M_RELATIVE_SIGN_CONVENTION__, verbose) 2209 # Load the waveform data 2210 if load==True: -> 2211 this.__load__(lmax=lmax,lm=lm,dt=dt,pad=pad) 2212 2213 #--%%--%%--%%--%%--%%--%%--%%--%%--%%--%%--%%--%%--%%--%%--%%--%%--%%--%%--# /root/.local/lib/python2.7/site-packages/nrutils/core/nrsc.pyc in __load__(this, lmax, lm, extraction_parameter, level, dt, pad, verbose) 2419 this.external_sign_convention = None 2420 for lm in this.__lmlist__: -> 2421 this.load(lm=lm,dt=dt,extraction_parameter=extraction_parameter,level=level,pad=pad,verbose=verbose) 2422 2423 # Ensuer that all modes are the same length /root/.local/lib/python2.7/site-packages/nrutils/core/nrsc.pyc in load(this, lm, file_location, dt, extraction_parameter, level, output, pad, verbose) 2635 if int(scipy_version.split('.')[1])<16: 2636 # Account for old scipy functionality -> 2637 external_sign_convention = sign(this.L[-1]) * sign(m) * mode( sign( y_.dphi[msk_] ) )[0][0] 2638 initially_msign_matches_wsign = sign(m) == mode( sign( y_.dphi[msk_] ) )[0][0] 2639 else: AttributeError: gwylm instance has no attribute 'L' — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub , or mute the thread .

--

---

L. London

Research Associate, Gravitational Waves

School of Physics and Astronomy

Cardiff University

[Galadirith]

@llondon6 Oh no no no, no apologies :blush: So I realised after commenting that actually the simulation I loaded there was not in the silures catalog, because silures.db catalog that was generated is 6B

A = scsearch(keyword='silures', verbose=True)

(scsearch)>> Found keyword (=silures) keyword.
(scsearch)>> Found verbose (=True) keyword.
(scsearch)>> List of keywords or string keyword found: ALL scentry objects matching will be passed. To pass ANY entries matching the keywords, input the keywords using an iterable of not of type list.
(scsearch!)>> !! Found 0 simulations.

This should pick up silures simulations as my silures root is /data/silures. So all the errors during the build I guess forces a bad database to be produced.

[Galadirith]

@llondon6 So I think there are genuinely cases where the afterjunk... is stored incorrectly, and this is causing problems with the BAM loader as described in https://github.com/llondon6/nrutils_dev/issues/38#issue-342196781. But also even when the data is stored correctly I think for aligned spin cases a condition is failing when it should pass

https://github.com/llondon6/nrutils_dev/blob/cd1331897720b7e1c7e4dc7ac1465e801f65e1a2/nrutils/handlers/bam.py#L130-L133

If any of the S1/2 components are zero this fails, for example a simple case

np.r_[0,0,0.5].astype(list).astype(bool).all()
# False

when I think the intention here is that such an array should yield true, but I couldn't definitely have misunderstood why you were intending here.

[Galadirith]

@llondon6 just to add this is not time critical for me. All these issues don't apply to our precessing series simulations; as far as I am aware none of these issues affect those simulations.

So unless you need the silures catalog right now I would say not to rush any fix if there's even a chance it might muck up the precessing series processing :blush: The old revisions work just fine with silures 👍

[llondon6]

@Galadirith I've added an updated silures catalog to the repo. I think you're right about the astype(bool) bug, and you've likely mentioned this to me before. Indeed, it should only matter if there's a strong need to distinguish between afterjunk and initial parameter values. Maybe astype(bool).any() would work better?

[Galadirith]

@llondon6 Awesome, thank you so much :blush: How did you get the catalog to build without making any changes? did the build just work for you? (feel free to stop humouring me at any time you definitely have more important things to be doing)

Maybe astype(bool).any() would work better?

Mmm yeh, I think it might, if one of the values is ok then it probably means they are all ok.