Filter file parsing problem

[jacopo-chevallard]

This issue continue from #4

This problem arised in release [0.1.3](), when @kevinhainline pointed out this error:

% python test_photometry.py -r ../../BEAGLE_workdir/results/21129_data_test_3 -p fit_21129_test_3.param -v
param_file:  fit_21129_test_3.param
INFO:root:Loading the `PhotometricFilters` file: /Users/knh/Desktop/NIRCam/BEAGLE/BEAGLE_workdir/filters/filters_UVUDF_noz.dat
Traceback (most recent call last):
  File "test_photometry.py", line 75, in <module>
    my_photometry.filters.load(filters_file)
  File "../PyP-BEAGLE/beagle_filters.py", line 111, in load
    wl[j], t_wl[j] = l.split()
ValueError: too many values to unpack

That's my fault, with Docker-BEAGLE indeed you don't need to specify system-wide environment variables, but you do need this for PyP-BEAGLE.

I added to release 0.1.4 some example scripts to set the environment variables, you can copy them to your .bashrc or .tcshrc.

About the problem with the filter, can you add before line 109 a print statement, so that we see what it tries to parse when it breaks?

for j, l in enumerate(lines[self.start_line[i]+1:self.end_line[i]]):
                wl[j], t_wl[j] = l.split()

becomes

for j, l in enumerate(lines[self.start_line[i]+1:self.end_line[i]]):
                print "line: ", l
                wl[j], t_wl[j] = l.split()

[kevinhainline]

I took a look at that, and the results were a little weird:

$ python test_photometry.py -r ../../BEAGLE-general-master/results/21129_data_test_3 -p fit_21129_test_3.param -v
param_file:  fit_21129_test_3.param
INFO:root:Loading the `PhotometricFilters` file: /Users/knh/Desktop/NIRCam/BEAGLE/BEAGLE-general-master/filters/filters_UVUDF_noz.dat
line:  12194.0   9.178e-01 

line:  12204.0   9.269e-01 

line:  12214.0   9.345e-01 

line:  12224.0   9.359e-01 

line:  12234.0   9.422e-01 

line:  12244.0   9.481e-01 

line:  12254.0   9.549e-01 

line:  12264.0   9.600e-01 

line:  12274.0   9.620e-01 

line:  12284.0   9.688e-01 

line:  12294.0   9.722e-01 

line:  12304.0   9.759e-01 

line:  12314.0   9.802e-01 

line:  12324.0   9.833e-01 

line:  12334.0   9.861e-01 

line:  12344.0   9.906e-01 

line:  12354.0   9.940e-01 

...

line:  13554.0   0.000e+00 

line:  13564.0   0.000e+00 

line:  13574.0   0.000e+00 

line:  13584.0   0.000e+00 

line:  13594.0   0.000e+00 

line:  13604.0   0.000e+00 

line:  13614.0   0.000e+00 

line:  # WIRCAM H filter

Traceback (most recent call last):
  File "test_photometry.py", line 75, in <module>
    my_photometry.filters.load(filters_file)
  File "../PyP-BEAGLE/beagle_filters.py", line 112, in load
    wl[j], t_wl[j] = l.split()
ValueError: too many values to unpack

This doesn't seem to be the right place to look for the filter files within filterfrm.res, I think.

Also, I've noticed that in the new BEAGLE-general github repository, there's no filters.log file (necessary for PyP-BEAGLE, I think), or FILTERBIN.RES file (necessary for BEAGLE).

[jacopo-chevallard]

The filters.log and FILTERBIN.RES files are produced automatically when you run build_filterbin, did you run build_filterbin yourself?

[kevinhainline]

I don't think so, no. They were included in an older version of BEAGLE that I had.

[jacopo-chevallard]

Oh, that's it !! You should follow this wiki page and build yourself the filters.log and FILTERBIN.RES starting from the ASCII filterfrm.res file. This should solve the problem!

[kevinhainline]

Ok, so I went and built my own filters.log and FILTERBIN.res file:

% gfortran --fixed-line-length-none -o add_filters trapz1.f add_filters.f

And then:

% ./build_filterbin

And then I ran BEAGLE (it took almost five hours to fit the object.)

Then I went to run PyP-BEAGLE, and I got this error:

$ python test_photometry.py -r ../../BEAGLE-general-master/results/21129_data_test_4 -p fit_21129_test_4.param -v
param_file:  fit_21129_test_4.param
INFO:root:Loading the `PhotometricFilters` file: /Users/knh/Desktop/NIRCam/BEAGLE/BEAGLE-general-master/filters/filters_UVUDF_noz.dat
1e-29
Traceback (most recent call last):
  File "test_photometry.py", line 91, in <module>
    my_photometry.plot_marginal(ID, replot=True)
  File "../PyP-BEAGLE/beagle_photometry.py", line 171, in plot_marginal
    obs_flux, obs_flux_err = self.observed_catalogue.extract_fluxes(self.filters, ID)
  File "../PyP-BEAGLE/beagle_photometry.py", line 93, in extract_fluxes
    flux[j] = row[name] * aper_corr * filters.units / Jy
ValueError: setting an array element with a sequence.

So, I did some digging, and it looks like it wasn't setting row correctly, which came down to the fact that I had to change some code in test_photometry.py:

    #for ID in rand_IDs:
    #   print ID
    #    my_photometry.plot_marginal(ID, replot=True)
    #    my_PDF.plot_triangle(ID, M_star=True, replot=True)
    #stop

    obj_ID = '21129'
    my_photometry.plot_marginal(obj_ID, replot=True)
    my_PDF.plot_triangle(obj_ID, M_star=True, replot=True)

At this point, I re-ran things:

$ python test_photometry.py -r ../../BEAGLE-general-master/results/21129_data_test_4 -p fit_21129_test_4.param -v
param_file:  fit_21129_test_4.param
INFO:root:Loading the `PhotometricFilters` file: /Users/knh/Desktop/NIRCam/BEAGLE/BEAGLE-general-master/filters/filters_UVUDF_noz.dat
Traceback (most recent call last):
  File "test_photometry.py", line 97, in <module>
    my_photometry.plot_marginal(obj_ID, replot=True)
  File "../PyP-BEAGLE/beagle_photometry.py", line 185, in plot_marginal
    hdulist = fits.open(fits_file)
  File "/Users/knh/Ureka/variants/common/lib/python2.7/site-packages/astropy/io/fits/hdu/hdulist.py", line 129, in fitsopen
    return HDUList.fromfile(name, mode, memmap, save_backup, cache, **kwargs)
  File "/Users/knh/Ureka/variants/common/lib/python2.7/site-packages/astropy/io/fits/hdu/hdulist.py", line 271, in fromfile
    save_backup=save_backup, cache=cache, **kwargs)
  File "/Users/knh/Ureka/variants/common/lib/python2.7/site-packages/astropy/io/fits/hdu/hdulist.py", line 792, in _readfrom
    ffo = _File(fileobj, mode=mode, memmap=memmap, cache=cache)
  File "/Users/knh/Ureka/variants/common/lib/python2.7/site-packages/astropy/io/fits/file.py", line 140, in __init__
    self._open_filename(fileobj, mode, clobber)
  File "/Users/knh/Ureka/variants/common/lib/python2.7/site-packages/astropy/io/fits/file.py", line 476, in _open_filename
    self._file = gzip.open(self.name, PYFITS_MODES[mode])
  File "/Users/knh/Ureka/python/lib/python2.7/gzip.py", line 34, in open
    return GzipFile(filename, mode, compresslevel)
  File "/Users/knh/Ureka/python/lib/python2.7/gzip.py", line 94, in __init__
    fileobj = self.myfileobj = __builtin__.open(filename, mode or 'rb')
IOError: [Errno 2] No such file or directory: '../../BEAGLE-general-master/results/21129_data_test_4/21129_BANGS.fits.gz'

After some more digging, I realized you're going to need to change beagle_utils.py:

class BeagleDirectories(object):

    pypbeagle_data = os.path.join("pyp-beagle", "data")
    pypbeagle_plot = os.path.join("pyp-beagle", "plot")

    results_dir = ''

    suffix = 'BANGS'

    MN_suffix = '_BANGS_MNstats.dat'

    param_file = ''

to:

class BeagleDirectories(object):

    pypbeagle_data = os.path.join("pyp-beagle", "data")
    pypbeagle_plot = os.path.join("pyp-beagle", "plot")

    results_dir = ''

    suffix = 'BEAGLE'

    MN_suffix = '_BEAGLE_MNstats.dat'

    param_file = ''

At this point:

$ python test_photometry.py -r ../../BEAGLE-general-master/results/21129_data_test_4 -p fit_21129_test_4.param -v

...

Traceback (most recent call last):
  File "test_photometry.py", line 97, in <module>
    my_photometry.plot_marginal(obj_ID, replot=True)
  File "../PyP-BEAGLE/beagle_photometry.py", line 363, in plot_marginal
    transparent=False, bbox_inches="tight", pad_inches=0.1)
  File "/Users/knh/Ureka/python/lib/python2.7/site-packages/matplotlib-1.3.1-py2.7-macosx-10.6-x86_64.egg/matplotlib/figure.py", line 1421, in savefig
    self.canvas.print_figure(*args, **kwargs)
  File "/Users/knh/Ureka/python/lib/python2.7/site-packages/matplotlib-1.3.1-py2.7-macosx-10.6-x86_64.egg/matplotlib/backend_bases.py", line 2167, in print_figure
    **kwargs)
  File "/Users/knh/Ureka/python/lib/python2.7/site-packages/matplotlib-1.3.1-py2.7-macosx-10.6-x86_64.egg/matplotlib/backend_bases.py", line 1952, in print_pdf
    return pdf.print_pdf(*args, **kwargs)
  File "/Users/knh/Ureka/python/lib/python2.7/site-packages/matplotlib-1.3.1-py2.7-macosx-10.6-x86_64.egg/matplotlib/backends/backend_pdf.py", line 2352, in print_pdf
    self.figure.draw(renderer)
  File "/Users/knh/Ureka/python/lib/python2.7/site-packages/matplotlib-1.3.1-py2.7-macosx-10.6-x86_64.egg/matplotlib/artist.py", line 55, in draw_wrapper
    draw(artist, renderer, *args, **kwargs)
  File "/Users/knh/Ureka/python/lib/python2.7/site-packages/matplotlib-1.3.1-py2.7-macosx-10.6-x86_64.egg/matplotlib/figure.py", line 1034, in draw
    func(*args)
  File "/Users/knh/Ureka/python/lib/python2.7/site-packages/matplotlib-1.3.1-py2.7-macosx-10.6-x86_64.egg/matplotlib/artist.py", line 55, in draw_wrapper
    draw(artist, renderer, *args, **kwargs)
  File "/Users/knh/Ureka/python/lib/python2.7/site-packages/matplotlib-1.3.1-py2.7-macosx-10.6-x86_64.egg/matplotlib/axes.py", line 2086, in draw
    a.draw(renderer)
  File "/Users/knh/Ureka/python/lib/python2.7/site-packages/matplotlib-1.3.1-py2.7-macosx-10.6-x86_64.egg/matplotlib/artist.py", line 55, in draw_wrapper
    draw(artist, renderer, *args, **kwargs)
  File "/Users/knh/Ureka/python/lib/python2.7/site-packages/matplotlib-1.3.1-py2.7-macosx-10.6-x86_64.egg/matplotlib/axis.py", line 1105, in draw
    self.label.draw(renderer)
  File "/Users/knh/Ureka/python/lib/python2.7/site-packages/matplotlib-1.3.1-py2.7-macosx-10.6-x86_64.egg/matplotlib/artist.py", line 55, in draw_wrapper
    draw(artist, renderer, *args, **kwargs)
  File "/Users/knh/Ureka/python/lib/python2.7/site-packages/matplotlib-1.3.1-py2.7-macosx-10.6-x86_64.egg/matplotlib/text.py", line 547, in draw
    bbox, info, descent = self._get_layout(renderer)
  File "/Users/knh/Ureka/python/lib/python2.7/site-packages/matplotlib-1.3.1-py2.7-macosx-10.6-x86_64.egg/matplotlib/text.py", line 329, in _get_layout
    ismath=ismath)
  File "/Users/knh/Ureka/python/lib/python2.7/site-packages/matplotlib-1.3.1-py2.7-macosx-10.6-x86_64.egg/matplotlib/backends/backend_pdf.py", line 1957, in get_text_width_height_descent
    self.mathtext_parser.parse(s, 72, prop)
  File "/Users/knh/Ureka/python/lib/python2.7/site-packages/matplotlib-1.3.1-py2.7-macosx-10.6-x86_64.egg/matplotlib/mathtext.py", line 3011, in parse
    box = self._parser.parse(s, font_output, fontsize, dpi)
  File "/Users/knh/Ureka/python/lib/python2.7/site-packages/matplotlib-1.3.1-py2.7-macosx-10.6-x86_64.egg/matplotlib/mathtext.py", line 2345, in parse
    str(err)]))
ValueError: 
\lambda_\textnormal{eff} / \textnormal{\AA}
          ^
Expected Re:('[^A-Za-z]') (at char 10), (line:1, col:11)

This is an error with printing the TeX on the figure plots. I fixed that by adding:

from pylab import *
rcParams['text.usetex'] = True

to the top of beagle_photometry.py. There may be another way, though.

And finally, I was able to make plots from my run of BEAGLE!

and

These are pretty different from what you sent me, so now I just need to figure out what the parameters you may have used to fit this object were.

[jacopo-chevallard]

That's great that you manage go get through it and make the plots !!

I'm a bit worried by the 5 hours time to fit 1 object... can you show me the settings of MCMC.param file? Also, how many likelihood evaluations were needed? (this information is printed on stdout by MultiNest)

As a rule of thumb, typically a likelihood evaluation for photometric data (when templates have been rebinned) should be around 0.05 s.

[jacopo-chevallard]

This discussion continues in jacopo-chevallard/BEAGLE-general#5, closing the issue as it has been resolved (you can make the plots)