problem accessing ferre model grids after download

[JMarieClarkCunningham]

Hi, I'm trying to use ferre to make model spectra for apogee, and I've managed to get everything installed in both python 2 and python 3. However, in both cases, the ferre model grids are downloaded to a temporary folder and then when I attempt to use it with ferre.interpolate() I get an error because it can't find a directory named, for example, ./tmprCQvN7. Can you provide any pointers? I'm working with @mrawls

I appreciate it.

[jobovy]

Could you give me a little more information about your configuration? What data release are you trying to use? If you do

import apogee.tools.path as appath
print(appath.ferreModelLibraryPath())

what do you get?

I don't think that the model grids are being downloaded to a temporary directory. During the download they get saved to a temporary location, but then they should get moved to the correct directory (the one given by the command above).

If you post the full error message (if it's long, please put it on pastebin), I might be able to help better.

[JMarieClarkCunningham]

Thank you for your rapid response! The data release we are trying to use currently is the DR12.

After doing: import apogee.tools.path as appath print(appath.ferreModelLibraryPath())

I get, /Users/halzoo/SDSS/DS12/dr12/apogee/spectro/redux/speclib/asset/kurucz_filled/solarisotopes/asGK_131216_lsfcombo5v6/p6_apsasGK_131216_lsfcombo5v6_w123.dat

And the full error message received after from apogee.modelspec import ferre mspec= ferre.interpolate(4750.,2.5,-0.1,0.1,0.,0.) is as follows:

from apogee.modelspec import ferre mspec= ferre.interpolate(4750.,2.5,-0.1,0.1,0.,0.) Traceback (most recent call last): File "<stdin>", line 1, in <module> File "apogee/tools/read.py", line 37, in output_wrapper out= func(*args,**kwargs) File "apogee/modelspec/ferre.py", line 40, in scalar_wrapper result= func(*args,**kwargs) File "apogee/modelspec/ferre.py", line 223, in interpolate run_ferre(tmpDir,verbose=verbose) File "apogee/modelspec/ferre.py", line 1084, in run_ferre raise Exception("Running FERRE instance in directory %s failed ..." % dir) Exception: Running FERRE instance in directory ./tmpbljqmt failed ...

Thanks again!

[jobovy]

Thanks. It seems like something goes wrong when you try to run FERRE.

Did you check that the file

/Users/halzoo/SDSS/DS12/dr12/apogee/spectro/redux/speclib/asset/kurucz_filled/solarisotopes/asGK_131216_lsfcombo5v6/p6_apsasGK_131216_lsfcombo5v6_w123.dat

exists?

Are you sure that FERRE was installed properly? (Installation instructions here) If you type ferre in a terminal window, something should run (and error, but it shouldn't say 'ferre not found' or similar).

You can also run the interpolate command with verbose=True, which might produce more information.

[JMarieClarkCunningham]

The file /Users/halzoo/SDSS/DS12/dr12/apogee/spectro/redux/speclib/asset/kurucz_filled/solarisotopes/ does indeed exist.

@mrawls and I agree that we think FERRE is installed properly and receive the same error when we run in Python 2 or Python 3. I have tried reinstalling with the ferre -noopenmp option and receive the same error, which is as follows:

---

             f e r r e          v4.6.6      

 starting on  8-11-2016   at   11:12:59.45

---

 load_control: WARNING
 nov=0, using Pure Interpolation mode
 load_control -> ndim=           6
 load_control -> nov =           0
 load_control -> snr =  -1.00000000  
 load_control -> nruns=           1
 load_control -> lsf=           0
 load_control -> nthreads=           1
At line 78 of file read_f.f90 (unit = 1, file = '/Users/halzoo/SDSS/DS12/dr12/apogee/spectro/redux/speclib/asset/kurucz_filled/solarisotopes/asGK_131216_lsfcombo5v6/p6_apsasGK_131216_lsfcombo5v6_w123.hdr')
Fortran runtime error: End of file

Error termination. Backtrace:
#0  0x109a24b99
#1  0x109a25855
#2  0x109a25fb9
#3  0x109aee15b
#4  0x109aef0f1
#5  0x109af174f
#6  0x109af18e8
#7  0x10976c210
#8  0x1097618bf
#9  0x109784f7e
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "apogee/tools/read.py", line 37, in output_wrapper
    out= func(_args,__kwargs)
  File "apogee/modelspec/ferre.py", line 40, in scalar_wrapper
    result= func(_args,**kwargs)
  File "apogee/modelspec/ferre.py", line 223, in interpolate
    run_ferre(tmpDir,verbose=verbose)
  File "apogee/modelspec/ferre.py", line 1084, in run_ferre
    raise Exception("Running FERRE instance in directory %s failed ..." % dir)
Exception: Running FERRE instance in directory ./tmprnmIbi failed ...```

[jobovy]

Seems like there's an issue with the p6_apsasGK_131216_lsfcombo5v6_w123.hdr file. Probably best if you remove both p6_apsasGK_131216_lsfcombo5v6_w123.hdr and p6_apsasGK_131216_lsfcombo5v6_w123.dat (there might be another similar file, remove that too) and redownload them. It may be that the download failed to complete properly the first time.

[JMarieClarkCunningham]

Ok! So it seems like the issue was indeed with the p6_apsasGK_131216_lsfcombo5v6_w123.hdr file. I deleted them and redownloaded them using Python 2, instead of Python 3 like I had used before. We had discovered that the StringIO module is not available for Python 3, which is what prompted us to change from io import StringIO which resulted in a different error, this led to using Python 2 in the first place. But after downloading the grids with Python 2 it seems to have resolved the issue.

I appreciate your help, Joni Marie Clark Cunningham

[jobovy]

Good that it works now! So is there an issue with downloading the library in Python 3? Or was this unrelated?

[mrawls]

Hi there, yep, it seems like download.pyand ferre.py have some string parsing that is not actually python 3 compatible. The download appears to complete, but this error is printed, which is apparently connected to some deeper string-y issues.

Python 3.5.1 |Continuum Analytics, Inc.| (default, Dec  7 2015, 11:24:55) 
[GCC 4.2.1 (Apple Inc. build 5577)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from apogee.tools import download
>>> download.ferreModelLibrary(lib='GK',pca=True,sixd=True,unf=False,dr=None,convertToBin=True)
--2016-11-07 10:28:47--  http://data.sdss3.org/sas/dr12/apogee/spectro/redux/speclib/asset/kurucz_filled/solarisotopes/asGK_131216_lsfcombo5v6/p6_apsasGK_131216_lsfcombo5v6_w123.dat
Resolving data.sdss3.org... 128.122.53.67
Connecting to data.sdss3.org|128.122.53.67|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 6498504162 (6.1G) [application/octet-stream]
Saving to: ‘/var/folders/gh/wk3w_d9d5qx_550341rf7gmh0000gn/T/tmptczgbyez’

/var/folders/gh/wk3w_d9d 100%[==================================>]   6.05G  5.26MB/s    in 45m 13s 

2016-11-07 11:14:01 (2.28 MB/s) - ‘/var/folders/gh/wk3w_d9d5qx_550341rf7gmh0000gn/T/tmptczgbyez’ saved [6498504162/6498504162]

Traceback (most recent call last):binary (can take a few minutes) ...           
  File "<stdin>", line 1, in <module>
  File "/Users/Meredith/anaconda/envs/astroconda/lib/python3.5/site-packages/apogee-1.-py3.5.egg/apogee/tools/download.py", line 388, in ferreModelLibrary
    p.stdin.write(os.path.basename(filePath)+'\n')
TypeError: a bytes-like object is required, not 'str'

[jobovy]

Okay, thanks. I think this should be fixed now and I fixed a few other similar problems, including another one with ferre. So you should be able to use the package with Python3 now. Please let me know if there are other Python3 related issues.

[mrawls]

Great! Those fixes almost certainly helped, but ferre.py still has a from StringIO import StringIO line, and I believe the py3 equivalent is just from io import StringIO. I'm working on re-downloading the grids to see if things will work as intended with that small edit, but the download is being exceptionally slow. Perhaps a related issue is the fact that for me, in py3, a plot window does not appear when I do the example given in the README in the "plotting" section. No errors are printed but no plot is created either, so that is a bit of a mystery.

[jobovy]

I changed the from StringIO import StringIO line and think it should work now, but haven't tested.

The plotting issue is probably unrelated to apogee. It's likely a matplotlib setup issue; make sure you're using the right backend.

[mrawls]

Thanks! Is there a specific backend recommended for apogee? I've been using Qt4Agg which works fine for simple matplotlib plots I regularly make with python 3 on macOS.

[jobovy]

It shouldn't matter, apogee just calls plot, so if that normally works, it should work for apogee I think. I think there's a macosx backend that I normally use on a Mac.

[mrawls]

No change with different backends. I seem to be getting a NoneType instead of a plotting object? (Apologies for dragging this issue way beyond the original scope!)

>>> import apogee.tools.read as apread
>>> import apogee.spec.plot as splot
>>> data= apread.rcsample()
>>> splot.waveregions(data[1000])
>>> 
>>> 
>>> test = splot.waveregions(data[1000])
>>> test
>>> 
>>> test.show()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'NoneType' object has no attribute 'show'
>>> test.plot()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'NoneType' object has no attribute 'plot'
>>> type(test)
<class 'NoneType'>
>>> print(test)
None

[jobovy]

Yes, splot.waveregions doesn't return a plotting object (perhaps it should), it just returns None.