`pycbc_single_template --psd-output` output .hdf file error

[Fergus-Hayes]

Calling the command:

$ pycbc_single_template --segment-length 528 --segment-start-pad 144 --segment-end-pad 16 --psd-estimation median --psd-segment-length 16 --psd-segment-stride 8 --psd-inverse-length 16 --approximant "SPAtmplt:mtotal<4" "SEOBNRv4_ROM:else" --order -1 --psd-num-segments 63 --taper-data 1 --allow-zero-padding --autogating-threshold 100 --autogating-cluster 0.5 --autogating-width 0.25 --autogating-taper 0.25 --autogating-pad 16 --minimum-chisq-bins 3 --strain-high-pass 15. --pad-data 8 --template-start-frequency 27 --chisq-bins '0.72*get_freq("fSEOBNRv4Peak",params.mass1,params.mass2,params.spin1z,params.spin2z)**0.7' --sample-rate 2048 --mass1 19.19753 --mass2 1.0981159 --spin1z -0.33484802 --spin2z 0.011133107 --low-frequency-cutoff 25.8 --gps-start-time 1169299588 --gps-end-time 1169300612 --trigger-time 1169300100.8 --window 1.48876953125 --frame-type H1_HOFT_C00 --channel-name H1:GDS-CALIB_STRAIN --psd-output ./PSD_1169300100_H1.hdf --output-file ./SNRTS_1169300100_H1.hdf

brings up the error:

  return self._data.__rdiv__(other)
Traceback (most recent call last):
  File "/cvmfs/oasis.opensciencegrid.org/ligo/sw/pycbc/x86_64_deb_8/virtualenv/pycbc-v1.9.4/bin/pycbc_single_template", line 4, in <module>
    __import__('pkg_resources').run_script('PyCBC==1.9.4', 'pycbc_single_template')
  File "/cvmfs/oasis.opensciencegrid.org/ligo/sw/pycbc/x86_64_deb_8/virtualenv/pycbc-v1.9.4/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 658, in run_script
    self.require(requires)[0].run_script(script_name, ns)
  File "/cvmfs/oasis.opensciencegrid.org/ligo/sw/pycbc/x86_64_deb_8/virtualenv/pycbc-v1.9.4/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 1438, in run_script
    exec(code, namespace, namespace)
  File "/cvmfs/oasis.opensciencegrid.org/ligo/deploy/sw/pycbc/x86_64_deb_8/virtualenv/pycbc-v1.9.4/lib/python2.7/site-packages/PyCBC-1.9.4-py2.7.egg/EGG-INFO/scripts/pycbc_single_template", line 238, in <module>
    precision='single')
  File "/cvmfs/oasis.opensciencegrid.org/ligo/deploy/sw/pycbc/x86_64_deb_8/virtualenv/pycbc-v1.9.4/lib/python2.7/site-packages/PyCBC-1.9.4-py2.7.egg/pycbc/psd/__init__.py", line 464, in associate_psds_to_segments
    precision=precision)
  File "/cvmfs/oasis.opensciencegrid.org/ligo/deploy/sw/pycbc/x86_64_deb_8/virtualenv/pycbc-v1.9.4/lib/python2.7/site-packages/PyCBC-1.9.4-py2.7.egg/pycbc/psd/__init__.py", line 425, in generate_overlapping_psds
    dyn_range_factor=dyn_range_factor, precision=precision)
  File "/cvmfs/oasis.opensciencegrid.org/ligo/deploy/sw/pycbc/x86_64_deb_8/virtualenv/pycbc-v1.9.4/lib/python2.7/site-packages/PyCBC-1.9.4-py2.7.egg/pycbc/psd/__init__.py", line 113, in from_cli
    (psd.astype(float64) / (dyn_range_factor ** 2)).save(opt.psd_output)
  File "/cvmfs/oasis.opensciencegrid.org/ligo/deploy/sw/pycbc/x86_64_deb_8/virtualenv/pycbc-v1.9.4/lib/python2.7/site-packages/PyCBC-1.9.4-py2.7.egg/pycbc/types/frequencyseries.py", line 418, in save
    compression_opts=9, shuffle=True)
  File "/cvmfs/oasis.opensciencegrid.org/ligo/deploy/sw/pycbc/x86_64_deb_8/virtualenv/pycbc-v1.9.4/lib/python2.7/site-packages/h5py/_hl/group.py", line 109, in create_dataset
    self[name] = dset
  File "/cvmfs/oasis.opensciencegrid.org/ligo/deploy/sw/pycbc/x86_64_deb_8/virtualenv/pycbc-v1.9.4/lib/python2.7/site-packages/h5py/_hl/group.py", line 277, in __setitem__
    h5o.link(obj.id, self.id, name, lcpl=lcpl, lapl=self._lapl)
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "h5py/h5o.pyx", line 202, in h5py.h5o.link
RuntimeError: Unable to create link (name already exists)

while removing the --psd-output option to .hdf will produce the snr time series fine.

[ahnitz]

@Fergus-Hayes Huh... can you verify if this happens if neither of the output files actually exist, or does it just happen in an empty directory otherwise?

[Fergus-Hayes]

This will happen in an empty directory yeah

[titodalcanton]

Saving to .txt files works, by the way.

[ahnitz]

@titodalcanton @Fergus-Hayes I think I know what the issue is. Given the options, there are actually multiple PSDs being generated which cover slightly different times. The save command is getting called for each, I think. This is a bug we'll need to fix....

[ahnitz]

The reason it doesn't fail for the .txt output, is that it's just overwriting the file and you'll end up with whatever the last PSD was.

[titodalcanton]

Some proposals for a fix:

1. Save all PSDs. For HDF output this is straightforward to do with different datasets tagged with epochs. For .npy and .txt files the PSDs could use different columns. It might not be straightforward to read those FrequencySeries back with PyCBC's primitives though.
2. Save an average PSD. The user will want the ability to choose the averaging method. Something like --psd-output median psd.txt.
3. Only save one of the PSDs. The user needs to choose which one, or a time of interest. Something like --psd-output 1214672180 psd.txt.
4. Last resort: fail unless the segmentation results in a single PSD being produced.

In principle options 1-3 could all be supported by turning the existing argument into --psd-output <action> <file>, where <action> could be all, mean/median, or a GPS time.

[GarethCabournDavies]

Confirming that this error persists. I think that implementing the "all" option should be the first thing to do to prevent failure. Allowing selection of which PSD gets saved is an enhancement which can come after