Error creating corner plots

[ACCarnall]

I've modified the test_spectra example for use with my spectrophotomeric fits, my test_spectra file looks like this:

import sys
sys.path.append("../PyP-BEAGLE")
import os
import argparse
import ConfigParser
import logging
from matplotlib import rc
from astropy.io import ascii
from astropy.io import fits
from glob import glob
from beagle_spectra import Spectrum
from beagle_pdf import PDF
from beagle_utils import BeagleDirectories, get_files_list, find_file
from beagle_parsers import standard_parser
import beagle_multiprocess

from pathos.multiprocessing import ProcessingPool 

if __name__ == '__main__':
    parser = standard_parser()
    args = parser.parse_args()
    logging.basicConfig(level=args.loglevel)

    # Set directory containing BEAGLE results files
    BeagleDirectories.results_dir = args.results_dir

    # Read parameter file
    config = ConfigParser.SafeConfigParser()
    param_file = args.param_file
    BeagleDirectories.param_file = param_file 
    name = os.path.join(BeagleDirectories.results_dir, param_file) 
    config.read(name)

    # File containing list of input spectra
    inputSpectraFileName = os.path.expandvars(config.get('main', 'LIST OF SPECTRA'))
    inputSpectraFile = open(inputSpectraFileName, 'r')

    # Global font size
    font = {'size': 16}
    rc('font', **font)

    # Get list of results files and object IDs from the results directory
    file_list, IDs = get_files_list()

    params_file = os.path.join(BeagleDirectories.results_dir, "params_names.json")
    # Initialize an instance of the main "Spectrum" class
    my_spectrum = Spectrum(params_file)
    my_spectrum.observed_spectrum.configure(config=config)

    # Set parameter names and labels
    my_PDF = PDF(params_file)

    file_name = "BEAGLE_summary_catalogue.fits"
    name = os.path.join(BeagleDirectories.results_dir, args.summary_config) 
    my_spectrum.summary_catalogue.compute(file_list, name)

    my_spectrum.summary_catalogue.load()

    for ID in IDs:
        spec_file_name = glob("/Users/adam/BEAGLE-general/data/VANDELS_spectra/" + ID + "*_table.fits")[0]
        my_spectrum.observed_spectrum.load(spec_file_name)

        my_spectrum.plot_marginal(ID)
        my_PDF.plot_triangle(ID)

This runs fine on the first four objects, but when it comes to the fifth corner plot I get the following error:

~/PyP-BEAGLE/tests$ ./test_spectra.sh 
/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/scipy/interpolate/interpolate.py:416: RuntimeWarning: invalid value encountered in subtract
  slope = (y_hi - y_lo) / (x_hi - x_lo)[:, None]
WARNING:root:outlier fraction 0.138888888889 
WARNING:root:outlier fraction 0.032044198895 
WARNING:root:outlier fraction 0.121130551817 
WARNING:root:outlier fraction 0.048730964467 
WARNING:root:outlier fraction 0.065306122449 
WARNING:root:fine_bins not large enough to well sample smoothing scale - formation_redshift
Traceback (most recent call last):
  File "test_spectra.py", line 65, in <module>
    my_PDF.plot_triangle(ID)
  File "/Users/adam/PyP-BEAGLE/PyP-BEAGLE/beagle_pdf.py", line 177, in plot_triangle
    g.triangle_plot(samples, filled=True, line_args=line_args)
  File "/Library/Python/2.7/site-packages/getdist/plots.py", line 1829, in triangle_plot
    label_right=True, no_zero=True, no_ylabel=True, no_ytick=True, line_args=line_args)
  File "/Library/Python/2.7/site-packages/getdist/plots.py", line 1299, in plot_1d
    bounds = self.add_1d(root, root_param, i, normalized=normalized, **line_args[i])
  File "/Library/Python/2.7/site-packages/getdist/plots.py", line 807, in add_1d
    density = self.sampleAnalyser.get_density(root, param, likes=self.settings.plot_meanlikes)
  File "/Library/Python/2.7/site-packages/getdist/plots.py", line 528, in get_density
    density = samples.get1DDensityGridData(name, meanlikes=likes)
  File "/Library/Python/2.7/site-packages/getdist/mcsamples.py", line 1297, in get1DDensityGridData
    density1D = Density1D(fine_x, P=conv, view_ranges=[par.range_min, par.range_max])
  File "/Library/Python/2.7/site-packages/getdist/densities.py", line 134, in __init__
    self.setP(P)
  File "/Library/Python/2.7/site-packages/getdist/densities.py", line 89, in setP
    raise DensitiesError("Array size mismatch in Density arrays: P %s, axis %s" % (size, ax.size))
getdist.densities.DensitiesError: Array size mismatch in Density arrays: P 1170, axis 200

I'm not sure whether the warnings that appear earlier on might be related to the eventual problem.

[ACCarnall]

I've tried running this on a couple of the other objects and I've hit a couple of other errors as well as some objects for which the code works fine:

Traceback (most recent call last):
  File "test_spectra.py", line 67, in <module>
    my_PDF.plot_triangle(ID)
  File "/Users/adam/PyP-BEAGLE/PyP-BEAGLE/beagle_pdf.py", line 177, in plot_triangle
    g.triangle_plot(samples, filled=True, line_args=line_args)
  File "/Library/Python/2.7/site-packages/getdist/plots.py", line 1829, in triangle_plot
    label_right=True, no_zero=True, no_ylabel=True, no_ytick=True, line_args=line_args)
  File "/Library/Python/2.7/site-packages/getdist/plots.py", line 1299, in plot_1d
    bounds = self.add_1d(root, root_param, i, normalized=normalized, **line_args[i])
  File "/Library/Python/2.7/site-packages/getdist/plots.py", line 807, in add_1d
    density = self.sampleAnalyser.get_density(root, param, likes=self.settings.plot_meanlikes)
  File "/Library/Python/2.7/site-packages/getdist/plots.py", line 528, in get_density
    density = samples.get1DDensityGridData(name, meanlikes=likes)
  File "/Library/Python/2.7/site-packages/getdist/mcsamples.py", line 1265, in get1DDensityGridData
    bins = np.bincount(bin_indices, weights=self.weights, minlength=fine_bins)
ValueError: The first argument of bincount must be non-negative

Traceback (most recent call last):
  File "test_spectra.py", line 67, in <module>
    my_PDF.plot_triangle(ID)
  File "/Users/adam/PyP-BEAGLE/PyP-BEAGLE/beagle_pdf.py", line 177, in plot_triangle
    g.triangle_plot(samples, filled=True, line_args=line_args)
  File "/Library/Python/2.7/site-packages/getdist/plots.py", line 1831, in triangle_plot
    if self.settings.no_triangle_axis_labels: self._spaceTicks(ax.xaxis)
  File "/Library/Python/2.7/site-packages/getdist/plots.py", line 1726, in _spaceTicks
    lims = [min(tick[0] - gap_wanted, lims[0]), max(tick[-1] + gap_wanted, lims[1])]
IndexError: list index out of range

Do you think this could be an error with the results of BEAGLE or just with PyP-BEAGLE? The spectral plots look fine for all but a couple of objects (one is the same, but the rest are different). I can send any more information you think may be helpful.

[jacopo-chevallard]

Hi Adam, these errors seem all be related to the getdist package on which PyP-BEAGLE depends...

Can you check which version of getdist you're using? you can just run

head /Library/Python/2.7/site-packages/getdist/__init__.py

My version is 0.2.4.

Then, if you want me to check this error, you should put all the relevant files (Beagle input and output) into a Dropbox folder and link it here

[ACCarnall]

Thanks Jacopo, I originally had version 0.2.5, but I've downgraded back to 0.2.4 and I'm still receiving the same error messages.

The dropbox link is here: https://www.dropbox.com/sh/dkgyj6wa2h7zom8/AAC6CJ-Z2pvWL-QMkPXnHU45a?dl=0

Note that I've made some changes to beagle_spectra.py within PyP-BEAGLE to get it to show the correct wavelength range on the marginal plots, my edited version of the file is also included.

[jacopo-chevallard]

I can reproduce the first error, more soon...

[jacopo-chevallard]

I understood the roots of the problem, I tried to open a issue here, let's see if the author of getdist (Antony Lewis) can help us

[jacopo-chevallard]

@ACCarnall As you can see from the issue linked above, Antony quickly replied, and we are trying to fix the problem

[ACCarnall]

Thanks a lot Jacopo!

[jacopo-chevallard]

@ACCarnall I think it would greatly accelerate the debugging providing Antony with a full working example of the problem, i.e. a tar archive containing all the python code and input files necessary to reproduce the error. Can you take care of this and put the archive on the issue linked above?

[ACCarnall]

Hi Jacopo, so would a tar file of the things I put on dropbox for you last week suffice?

[jacopo-chevallard]

I'm afraid not, since this person knows nothing about Beagle. We should provide him with all the python routines and input files, and a very simple bash script which will call the python routines with the correct arguments, the material must be prepared in such a way that the only thing that Antony has to do is to launch the script and get the error.

Ideally, you would test this on a machine where PyP-BEAGLE is not present, and find which files are necessary to reproduce the error. I can help you if you get stuck somewhere.

[ACCarnall]

Hi Jacopo, I've created this example, the only dependency that I guess he might not have is astropy, but I'm not sure how to remove that as a dependency? I've changed the beagle_pdf file so it doesn't rely on any other beagle files, but haven't deleted anything else from it to make it simpler as I wasn't sure what was important and what wasn't! Running test.py produces the error.

error_example.zip

[jacopo-chevallard]

Great, well done!!

[ACCarnall]

Happy for me to post it to the other thread or would you like to make some changes?

[jacopo-chevallard]

Oups, I did already, as I was able to run it as well! You should have received a notification, as I cited you in the thread

[jacopo-chevallard]

Closing, let me know if need to re-open!