enstaller 4.7.5 crash

[riehle]

Unfortunately, rev 4.7.5 also crashes on my windows machine:

$ python bootstrap.py
Downloading https://s3.amazonaws.com/enstaller-assets/enstaller/enstaller-4.7.5-py2.7.egg
enstaller-4.7.5-py2.7.egg                            [removing egg]
enstaller-4.7.5-py2.7.egg                          [installing egg]
exit()

C:\Windows\System32>enpkg --version
C:\Python27\lib\site-packages\enstaller\vendor\keyring\core.py:166: UserWarning:
 Module argparse was already imported from C:\Python27\lib\argparse.pyc, but c:\
python27\lib\site-packages is being added to sys.path
  pkg_resources = __import__('pkg_resources')
enstaller version: 4.7.5

C:\Windows\System32>enpkg --whats-new
C:\Python27\lib\site-packages\enstaller\vendor\keyring\core.py:166: UserWarning: Module argparse was already imported from C:\Python27\lib\argparse.py, but c:\python27\lib\site-packages is being added to sys.path pkg_resources = __import__('pkg_resources')
/eggs/win-64/index.json                            [Fetching index]
   1.05 MB [......................................................]
enstaller: Error: enstaller crashed (uncaught exception <type 'exceptions.KeyError'>: KeyError('python',)).
Please report this on enstaller issue tracker:
    http://github.com/enthought/enstaller/issues
You can get a full traceback by setting the ENSTALLER_DEBUG environment variable

C:\Windows\System32> set ENSTALLER_DEBUG=TRUE
C:\Windows\System32>enpkg --whats-new
C:\Python27\lib\site-packages\enstaller\vendor\keyring\core.py:166: UserWarning: Module argparse was already imported from C:\Python27\lib\argparse.py, but c:\python27\lib\site-packages is being added to sys.path
  pkg_resources = __import__('pkg_resources')
/eggs/win-64/index.json                            [Fetching index]
   1.05 MB [......................................................]
Traceback (most recent call last):
  File "C:\Python27\Scripts\enpkg-script.py", line 10, in <module>
    sys.exit(main_noexc())
  File "C:\Python27\lib\site-packages\enstaller\main.py", line 873, in main_noexc 
    main(argv)
  File "C:\Python27\lib\site-packages\enstaller\main.py", line 755, in main
    enpkg = Enpkg(repository, downloader, prefixes, progress_bar_context)
  File "C:\Python27\lib\site-packages\enstaller\enpkg.py", line 289, in __init__
    self._installed_repository = Repository._from_prefixes(self.prefixes)
  File "C:\Python27\lib\site-packages\enstaller\repository.py", line 295, in _fr
om_prefixes
    repository._populate_from_prefixes(prefixes)
  File "C:\Python27\lib\site-packages\enstaller\repository.py", line 281, in _po
pulate_from_prefixes
    InstalledPackageMetadata.from_installed_meta_dict(info)
  File "C:\Python27\lib\site-packages\enstaller\repository.py", line 192, in fro
m_installed_meta_dict
    json_dict["python"], json_dict["ctime"],
KeyError: 'python'

[riehle]

I put a try/except around the from_installed_meta_dict call in repository that's causing trouble and see that the json_dict contains these keys:

C:\Windows\System32>enpkg --whats-new
C:\Python27\lib\site-packages\enstaller\vendor\keyring\core.py:166: UserWarning:
 Module argparse was already imported from C:\Python27\lib\argparse.pyc, but c:\
python27\lib\site-packages is being added to sys.path
  pkg_resources = __import__('pkg_resources')
/eggs/win-64/index.json                            [Fetching index]
   1.05 MB [......................................................]
--Return--
> c:\python27\lib\site-packages\enstaller\repository.py(196)from_installed_meta_dict()->None
-> import pdb; pdb.set_trace()
(Pdb) json_dict.keys()
[u'name', 'store_location', 'installed', u'hook', u'version', u'build', u'key', 'meta_dir', u'ctime']

Parsing the repository.py code I see the following keys referenced for the json_dict[]:

json_dict["build"]                - in my keylist
json_dict["ctime"]                - in my keylist
json_dict["md5"]                  # computed in "from_egg" method?
json_dict["name"]                - in my keylist
json_dict["packages"]          # computed in "from_egg" method?
json_dict["python"])              # missing
json_dict["size"]                    # computed in "from_egg"?
json_dict["store_location"])   # computed in "from_egg"?
json_dict["version"]                - in my keylist
json_dict["key"]                      - in my keylist

[riehle]

Okay, I replaced the "from_installed_meta_dict" method in repository.py with this:

@classmethod
def from_installed_meta_dict(cls, json_dict):
    try:
        return cls(json_dict["key"], json_dict["name"], json_dict["version"],
               json_dict["build"], json_dict.get("packages", []),
               json_dict["python"], json_dict["ctime"],
               json_dict.get("store_location", ""))
    except:
        json_dict["python"] = "DEBUG TEST"
        return cls(json_dict["key"], json_dict["name"], json_dict["version"],
               json_dict["build"], json_dict.get("packages", []),
               json_dict["python"], json_dict["ctime"],
               json_dict.get("store_location", ""))

enpkg now works.

[cournape]

Ok, this is starting to get embarrassing. Let me look into it.

[cournape]

@riehle could you give me the value of json_dict when the python key does not exist ? It looks like some old eggs we produced do not have a python key, but I want to make sure.

[riehle]

C:\Windows\System32>enpkg --whats-new
C:\Python27\lib\site-packages\enstaller\vendor\keyring\core.py:166: UserWarning:
 Module argparse was already imported from C:\Python27\lib\argparse.pyc, but c:\
python27\lib\site-packages is being added to sys.path
  pkg_resources = __import__('pkg_resources')
/eggs/win-64/index.json                            [Fetching index]
   1.05 MB [......................................................]
> c:\python27\lib\site-packages\enstaller\repository.py(197)from_installed_meta_dict()
-> json_dict["python"] = "DEBUG TEST"
(Pdb) json_dict
{u'name': u'appinst', 'store_location': 'C:\\Python27', 'installed': True, u'hook': False, u'version': u'2.1.2', u'build': 1, u'key': u'appinst-2.1.2-1.egg', 'meta_dir': 'C:\\Python27\\EGG-INFO\\appinst', u'ctime': u'Thu Feb 14 11:20:43 2013'}
(Pdb)

[riehle]

cat /c/Python27/EGG-INFO/appinst/_info.json
{
  "build": 1,
  "ctime": "Thu Feb 14 11:20:43 2013",
  "hook": false,
  "key": "appinst-2.1.2-1.egg",
  "name": "appinst",
  "version": "2.1.2"
}

Searching through my EGG-INFO directory for eggs without "python" in their _info.json files I found these:

  "name": "appinst",    "version": "2.1.2"
  "name": "cloud",    "version": "2.4.6"
  "name": "doclinks",    "version": "7.3"
  "name": "epd",    "version": "7.3"
  "name": "epydoc",    "version": "3.0.1"
  "name": "examples",    "version": "7.3"
  "name": "grin",    "version": "1.2.1"
  "name": "idle",    "version": "2.7.3"
  "name": "jsonpickle",    "version": "0.4.0"
  "name": "mkl",    "version": "10.3"
  "name": "ply",    "version": "3.4"
  "name": "pyglet",    "version": "1.1.4"
  "name": "registrypath",    "version": "1.0"
  "name": "scikits.learn",    "version": "0.8"
  "name": "scikits.statsmodels",    "version": "0.3.1"
  "name": "scite",    "version": "1.74"
  "name": "_registry_path",    "version": "1.0"

[riehle]

The same version egg on my Linux machine has the python key in its json data, along with many other keys:

~]$ cat /usr/local/python27/EGG-INFO/appinst/_info.json
{
  "arch": null,
  "build": 1,
  "ctime": "Thu May 29 09:56:31 2014",
  "hook": false,
  "key": "appinst-2.1.2-1.egg",
  "name": "appinst",
  "osdist": null,
  "packages": [],
  "platform": null,
  "python": "2.7",
  "type": "egg",
  "version": "2.1.2"
}

AND none of the other Linux eggs are missing the "python" key.

[cournape]

The code involved here is way before my time, so I am not sure I understand how enstaller every wrote an _info.json without 'python' key.

I will just be more defensive in the code here then. I should have a patch ready soonish.

[riehle]

The _info.json is not in the egg file, so I presume it's created at install time. Is my problem that my install is really old, and some of my eggs were installed before the enstaller wrote all the required keys to its json files?

If so, could I simply do a "enpkg --force appinst" to get the newer installer to rewrite the json?

[riehle]

Okay, so I tried reinstalling an older egg, one that lacked the newer keys, and this did the expected upgrading of the json info:

OLD:

cat C:\Python27\EGG-INFO\examples\_info.json
{
  "build": 1,
  "ctime": "Thu Feb 14 11:20:43 2013",
  "hook": false,
  "key": "Examples-7.3-1.egg",
  "name": "examples",
  "version": "7.3"
}

C:\Windows\System32>enpkg --force examples
C:\Python27\lib\site-packages\enstaller\vendor\keyring\core.py:166: UserWarning:
 Module argparse was already imported from C:\Python27\lib\argparse.pyc, but c:\
python27\lib\site-packages is being added to sys.path
  pkg_resources = __import__('pkg_resources')
/eggs/win-64/index.json                            [Fetching index]
   1.05 MB [......................................................]
Examples-7.3-1.egg                                   [removing egg]
  63.45 MB [......................................................]
Examples-7.3-1.egg                                 [installing egg]
  63.45 MB [......................................................]

NEW

cat C:\Python27\EGG-INFO\examples\_info.json
{
  "arch": null,
  "build": 1,
  "ctime": "Wed Oct 08 13:57:19 2014",
  "hook": false,
  "key": "Examples-7.3-1.egg",
  "md5": "5a32461d4ef795af7d2cb1aaeeac1c20",
  "mtime": 1337788858.0,
  "name": "examples",
  "osdist": null,
  "packages": [
    "appinst"
  ],
  "platform": null,
  "product": "free",
  "python": null,
  "size": 19717207,
  "type": "egg",
  "version": "7.3"
}

So, I guess the best approach for me might be to refresh all my really old eggs.

[riehle]

I manually reinstalled all the eggs in question, including "epd", which downgraded all my updated eggs to their EPD7.3.2 release versions. Then I ran an "update-all" and watched as all the updates where installed. At this point I re-ran your bootstrap installer to update enstaller. Finally, to check that everything really was updated I ran a final "whats-new":

enpkg --whats-new
C:\Python27\lib\site-packages\enstaller\vendor\keyring\core.py:166: UserWarning:
 Module argparse was already imported from C:\Python27\lib\argparse.py, but c:\p
ython27\lib\site-packages is being added to sys.path
  pkg_resources = __import__('pkg_resources')
Traceback (most recent call last):
  File "C:\Python27\Scripts\enpkg-script.py", line 8, in <module>  from enstaller.main import main_noexc
  File "C:\Python27\lib\site-packages\enstaller\main.py", line 38, in <module>  from enstaller.config import (ENSTALLER4RC_FILENAME, HOME_ENSTALLER4RC,
  File "C:\Python27\lib\site-packages\enstaller\config.py", line 20, in <module> from enstaller.vendor import keyring
  File "C:\Python27\lib\site-packages\enstaller\vendor\keyring\__init__.py", line 12, in <module>
    from .core import (set_keyring, get_keyring, set_password, get_password,
  File "C:\Python27\lib\site-packages\enstaller\vendor\keyring\core.py", line 180, in <module>
    init_backend()
  File "C:\Python27\lib\site-packages\enstaller\vendor\keyring\core.py", line 59, in init_backend
    set_keyring(load_config() or _get_best_keyring())
  File "C:\Python27\lib\site-packages\enstaller\vendor\keyring\core.py", line 67, in _get_best_keyring
    keyrings = backend.get_all_keyring()
  File "C:\Python27\lib\site-packages\enstaller\vendor\keyring\util\__init__.py", line 24, in wrapper
    func.always_returns = func(*args, **kwargs)
  File "C:\Python27\lib\site-packages\enstaller\vendor\keyring\backend.py", line 137, in get_all_keyring    _load_backends()
  File "C:\Python27\lib\site-packages\enstaller\vendor\keyring\backend.py", line 129, in _load_backends
    list(map(_load_backend, backends))
  File "C:\Python27\lib\site-packages\enstaller\vendor\keyring\backend.py", line 114, in _load_backend
    mod = importlib.import_module('.'+name, package)
  File "C:\Python27\lib\importlib\__init__.py", line 37, in import_module
    __import__(name)
  File "C:\Python27\lib\site-packages\enstaller\vendor\keyring\backends\Google.p
y", line 11, in <module>
    import gdata.docs.service
  File "C:\Python27\lib\site-packages\gdata\__init__.py", line 109, in <module>  class LinkFinder(atom.LinkFinder):
AttributeError: 'module' object has no attribute 'LinkFinder'

At which point I recalled that the "atom" egg breaks the enstaller, but is still installed to satisfy some dependences. After deleting atom enstaller is working again.

[cournape]

FWIW, both the atom issue and missing key issue are now handled in enstaller master. If you are interested, I can give you an egg built from the repo that contains those fixes for testing.

[riehle]

Please do, I would be happy to try it.

[cournape]

The egg itself is there: https://s3.amazonaws.com/enstaller-assets/enstaller/enstaller-4.8.0.dev1_a34d367-py2.7.egg (sha256: c5b52e63eca3f064d44dbc815765224a9200b76c7a24795c44dd235488491d3d)

To install it, fetch the egg + the file https://s3.amazonaws.com/enstaller-assets/enstaller/bootstrap.py, and do as follows:

# It will automatically remove the old one if required
$ python bootstrap.py enstaller-4.8.0.dev1_a34d367-py2.7.egg

You should be able to safely go back to the latest released of enstaller as follows:

# This will automatically fetch the latest enstaller egg, remove the previous enstaller if found, and
# install the new one. Checksums are used to ensure the correct code is used.
$ python bootstrap.py