WISE not in zenodo MIST

[smoh]

Hi Tim,

I was able to import with current master of isochrones, but was getting some tarball decompression error even after I did it manually in ~/.isochrones/mist/. I think this is because WISE is not in the current MIST model upload in zenodo, and the code keeps requiring it, trying to download and decompress the entire master tarball again.

Traceback:

> /Users/semyeong/projects/isochrones/isochrones/mist/grid.py(111)get_filenames()
    109         if not os.path.exists(d):
    110             if not os.path.exists(self.phot_tarball_file(phot, version=version)):
--> 111                 self.extract_master_tarball()
    112             self.extract_phot_tarball(phot, version=version)
    113 

ipdb> self.phot_tarball_file(phot,version=version)
'/Users/semyeong/.isochrones/mist/MIST_v1.0_WISE.tar.gz'

Just thought I'd let you know.

[nespinoza]

Hi smoh, Tim,

I was going to open another ticket with a problem related to this, so maybe I'll leave the error I'm getting here, which is related to the MIST tar file that is downloaded automatically by the package (and which makes almost every nosetest to fail).

Trying to install everything from scratch, I try to run the example on the webpage, say:

from isochrones.mist import MIST_Isochrone
 mist = MIST_Isochrone()
 mist.radius(1.0, 9.7, 0.0)

Which fails; the whole traceback is huge, but the important part I think is the following:

/usr/local/lib/python2.7/site-packages/isochrones/grid.pyc in extract_master_tarball(cls)
      140         with tarfile.open(os.path.join(ISOCHRONES, cls.master_tarball_file)) as tar:
      141             logging.info('Extracting {}...'.format(cls.master_tarball_file))
--> 142             tar.extractall(ISOCHRONES)
      143 
      144     @classmethod

The traceback continues, and ends pointing here:

/usr/local/Cellar/python/2.7.9/Frameworks/Python.framework/Versions/2.7/lib/python2.7/gzip.pyc in _read_eof(self)
    352         if crc32 != self.crc:
    353             raise IOError("CRC check failed %s != %s" % (hex(crc32),
--> 354                                                          hex(self.crc)))
    355         elif isize != (self.size & 0xffffffffL):
    356             raise IOError, "Incorrect length of data produced"

IOError: CRC check failed 0x113ea395 != 0xdbafde43L

So as smoh pointed out, it seems it is a problem with the tar extraction of the file. Note the MIST file is the only that is failing; the Darthmouth models work like a charm.

[timothydmorton]

Did you compare the md5 checksum of the file you downloaded with the one on zenodo? On Sun, Nov 13, 2016 at 20:21 Néstor Espinoza notifications@github.com wrote:

Hi smoh, Tim,

I was going to open another ticket with a problem related to this, so maybe I'll leave the error I'm getting here, which is related to the MIST tar file that is downloaded automatically by the package (and which makes almost every nosetest to fail).

Trying to install everything from scratch, I try to run the example on the webpage, say:

from isochrones.mist import MIST_Isochrone mist = MIST_Isochrone() mist.radius(1.0, 9.7, 0.0)

Which fails; the whole traceback is huge, but the important part I think is the following:

/usr/local/lib/python2.7/site-packages/isochrones/grid.pyc in extract_master_tarball(cls) 140 with tarfile.open(os.path.join(ISOCHRONES, cls.master_tarball_file)) as tar: 141 logging.info('Extracting {}...'.format(cls.master_tarball_file)) --> 142 tar.extractall(ISOCHRONES) 143 144 @classmethod

The traceback continues, and ends pointing here:

`/usr/local/Cellar/python/2.7.9/Frameworks/Python.framework/Versions/2.7/lib/python2.7/gzip.pyc in _read_eof(self) 352 if crc32 != self.crc: 353 raise IOError("CRC check failed %s != %s" % (hex(crc32), --> 354 hex(self.crc))) 355 elif isize != (self.size & 0xffffffffL): 356 raise IOError, "Incorrect length of data produced"

IOError: CRC check failed 0x113ea395 != 0xdbafde43L`

So it seems it as sooh pointed out, it seems it is a problem with the tar extraction of the file. Note the MIST file is the only that is failing; the Darthmouth models work like a charm.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/timothydmorton/isochrones/issues/42#issuecomment-260228590, or mute the thread https://github.com/notifications/unsubscribe-auth/ABzr25ZqSvirY7hd30OOi-k3GvpIUFi5ks5q97eRgaJpZM4KvM3D .

[timothydmorton]

It's possible that both of you have had problems downloading the big MIST tar file. This is what I see with this archive locally on my computer, and this md5 checksum (0deaaca2836c7148c27ce5ba5bbdfe59) matches with what is on Zenodo. If the automated download isn't working well (which it appears may be the case), download it directly from here with your browser. @smoh, I think you did this already, so not sure why you didn't get the WISE bands...

Timothys-MacBook-Pro-2:.isochrones tdm$ tar -tf mist.tgz 
mist/._MIST_v1.0_CFHT.tar.gz
mist/MIST_v1.0_CFHT.tar.gz
mist/._MIST_v1.0_DECam.tar.gz
mist/MIST_v1.0_DECam.tar.gz
mist/._MIST_v1.0_GALEX.tar.gz
mist/MIST_v1.0_GALEX.tar.gz
mist/._MIST_v1.0_JWST.tar.gz
mist/MIST_v1.0_JWST.tar.gz
mist/._MIST_v1.0_LSST.tar.gz
mist/MIST_v1.0_LSST.tar.gz
mist/._MIST_v1.0_PanSTARRS.tar.gz
mist/MIST_v1.0_PanSTARRS.tar.gz
mist/._MIST_v1.0_SDSS.tar.gz
mist/MIST_v1.0_SDSS.tar.gz
mist/._MIST_v1.0_SPITZER.tar.gz
mist/MIST_v1.0_SPITZER.tar.gz
mist/._MIST_v1.0_SkyMapper.tar.gz
mist/MIST_v1.0_SkyMapper.tar.gz
mist/._MIST_v1.0_UBVRIplus.tar.gz
mist/MIST_v1.0_UBVRIplus.tar.gz
mist/._MIST_v1.0_UKIDSS.tar.gz
mist/MIST_v1.0_UKIDSS.tar.gz
mist/._MIST_v1.0_WISE.tar.gz
mist/MIST_v1.0_WISE.tar.gz
Timothys-MacBook-Pro-2:.isochrones tdm$ md5 mist.tgz
MD5 (mist.tgz) = 0deaaca2836c7148c27ce5ba5bbdfe59

[smoh]

Yes, I see WISE in a freshly download tarball just now. Sorry, must have been my mistake.

[nespinoza]

Hi Tim,

Indeed, the checksum for the downloaded file was wrong. Downloaded the one with the link you shared and now everything works like a charm. Thanks again for your help!

[timothydmorton]

OK, great to hear. I need to improve the automated downloading function to prevent this problem from happening to others-- I've heard it now from a lot of people!