aws-shell does not work with prompt_toolkit==0.54.0

[orome]

I get:

First run, creating autocomplete index...
Creating doc index in the background. It will be a few minutes before all documentation is available.
Traceback (most recent call last):
  File "/usr/local/bin/aws-shell", line 11, in <module>
    sys.exit(main())
  File "/usr/local/lib/python2.7/site-packages/awsshell/__init__.py", line 63, in main
    shell.run()
  File "/usr/local/lib/python2.7/site-packages/awsshell/app.py", line 200, in run
    document = self.cli.run()
  File "/usr/local/lib/python2.7/site-packages/awsshell/app.py", line 193, in cli
    self._cli = self.create_cli_interface(self.show_completion_columns)
  File "/usr/local/lib/python2.7/site-packages/awsshell/app.py", line 367, in create_cli_interface
    display_completions_in_columns)
  File "/usr/local/lib/python2.7/site-packages/awsshell/app.py", line 335, in create_application
    key_bindings_registry=self.key_manager.manager.registry,
  File "/usr/local/lib/python2.7/site-packages/prompt_toolkit/application.py", line 112, in __init__
    assert style is None or isinstance(style, Style)
AssertionError

On OS X 10.11.2; Homebrew Python 2.7.11

[donnemartin]

@orome, curious what version of prompt-toolkit you have. Can you send me the output of the following command?

 pip freeze

[orome]

prompt-toolkit==0.54

[jamesls]

Not sure how it's possible to install the aws-shell right now with a version > 0.52, but I did confirm that if I manually bump the prompt_toolkit version to 0.54 I get the error message.

It would be good to get this working with newer versions of prompt_toolkit.

Related comment from the announcement blog:

https://blogs.aws.amazon.com/cli/post/Tx1XV5WUCMWLMEJ/Super-Charge-Your-AWS-Command-Line-Experience-with-aws-shell#postCommentsTx1XV5WUCMWLMEJ

[donnemartin]

Ah, I suspected it was a version mismatch, the prompt_toolkit 0.52 source didn't match up at all with the traceback.

Not sure how it's possible to install the aws-shell right now with a version > 0.52

Not sure either, we do peg it: 'prompt-toolkit==0.52'

It would be good to get this working with newer versions of prompt_toolkit.

I agree, although it could be tricky to always keep up to date on the latest. @jonathanslenders does awesome work and pushes out updates quickly. The side effect is that the rapid pace sometimes breaks backwards compatibility, as seen here: https://github.com/donnemartin/saws/issues/29. Jonathan was very helpful in submitting a fix quickly: https://github.com/donnemartin/saws/pull/30. It seems 0.54 has backwards compatibility issues as well.

I think this is further complicated by our ui.py, which is a copy of a fair amount of the prompt_toolkit internals. I've had to make tweaks to our create_default_layout to get auto suggestions to work. There are more changes for the upcoming syntax highlighting/lexer feature. The updates involve basically copying the diffs found in later versions of prompt_toolkit into ui.py.

[jamesls]

Yeah, just to clarify I meant this more as a nice-to-have. I don't think it's possible to pull in 0.54 via pip install aws-shell so most people shouldn't run into this issue, though I would like to understand how this happened.

FWIW, this particular issue seems pretty straightforward. The CliStyle class we have in awsshell.style subclasses from pygments.Style, but looks like 0.54 requires you to subclass from its own version:

In: prompt_toolkit/application.py(92)__init__()
(Pdb) p Style
<class 'prompt_toolkit.styles.Style'>
(Pdb) p style
<class 'awsshell.style.CliStyle'>
(Pdb) p isinstance(style, Style)
False

Seems like a pretty straightforward fix (I think).

[donnemartin]

Sounds like a plan :+1:

If you're not already in motion I can try to tackle this in the next day or two.

[jamesls]

I'll take a look.

Looks like I didn't look carefully enough at the pdb output. The isinstance() check is ensuring that the passed in style is an instance of the prompt_toolkit.styles.Style class, but we're providing a style class not an instance.

Looking through the latest prompt_toolkit code, it looks like I need to do something like this:

style = PygmentsStyle(style)

[mangofoto]

Some packages are downgraded during the pip install of aws-shell. Here’s what I found.

My test instance: cat /etc/redhat-release Red Hat Enterprise Linux Server release 7.1 (Maipo) uname -r 3.10.0-229.11.1.el7.x86_64

sudo pip install aws-shell

Just showing the relevant bits and the downgrades

Installing collected packages: pyasn1, rsa, jmespath, six, python-dateutil, docutils, botocore, colorama, awscli, pygments, wcwidth, prompt-toolkit, futures, boto3, configobj, aws-shell

Dependencies

Collecting docutils>=0.10 (from awscli<2.0.0,>=1.8.9->aws-shell) Collecting six>=1.9.0 (from prompt-toolkit==0.52->aws-shell) Downloading six-1.10.0-py2.py3-none-any.whl

downgrades for my existing packages

Found existing installation: six 1.7.3 DEPRECATION: Uninstalling a distutils installed project (six) has been deprecated and will be removed in a future version. This is due to the fact that uninstalling a distutils project will only partially uninstall the project. Uninstalling six-1.7.3: Successfully uninstalled six-1.7.3 Found existing installation: python-dateutil 1.5 Uninstalling python-dateutil-1.5: Successfully uninstalled python-dateutil-1.5 Running setup.py install for docutils Running setup.py install for colorama Found existing installation: configobj 4.7.2 DEPRECATION: Uninstalling a distutils installed project (configobj) has been deprecated and will be removed in a future version. This is due to the fact that uninstalling a distutils project will only partially uninstall the project. Uninstalling configobj-4.7.2: Successfully uninstalled configobj-4.7.2 Running setup.py install for configobj Running setup.py install for aws-shell Successfully installed aws-shell-0.0.1 awscli-1.9.15 boto3-1.2.3 botocore-1.3.15 colorama-0.3.3 configobj-5.0.6 docutils-0.12 futures-3.0.3 jmespath-0.9.0 prompt-toolkit-0.52 pyasn1-0.1.9 pygments-2.0.2 python-dateutil-2.4.2 rsa-3.2.3 six-1.10.0 wcwidth-0.1.5

pip list You are using pip version 7.1.0, however version 7.1.2 is available. You should consider upgrading via the 'pip install --upgrade pip' command. aws-shell (0.0.1) awscli (1.9.15) backports.ssl-match-hostname (3.4.0.2) boto3 (1.2.3) botocore (1.3.15) chardet (2.2.1) cloud-init (0.7.6) colorama (0.3.3) configobj (5.0.6) decorator (3.4.0) docutils (0.12) ethtool (0.8) futures (3.0.3) iniparse (0.4) IPy (0.75) jmespath (0.9.0) jsonpatch (1.2) jsonpointer (1.0) kitchen (1.1.1) lxml (3.2.1) M2Crypto (0.21.1) pciutils (1.7.3) pip (7.1.0) policycoreutils-default-encoding (0.1) prettytable (0.7.2) prompt-toolkit (0.52) pyasn1 (0.1.9) pycurl (7.19.0) Pygments (2.0.2) pygobject (3.8.2) pygpgme (0.3) pyliblzma (0.5.3) pyOpenSSL (0.13.1) python-dateutil (2.4.2) python-dmidecode (3.10.13) pyudev (0.15) pyxattr (0.5.1) PyYAML (3.10) requests (2.6.0) rhnlib (2.5.65) rhsm (1.13.10) rsa (3.2.3) seobject (0.1) sepolicy (1.1) setuptools (0.9.8) six (1.10.0) urlgrabber (3.10) urllib3 (1.10.2) wcwidth (0.1.5) yum-metadata-parser (1.1.4)

[jamesls]

@mangofoto If I'm understanding the output correctly, the packges under the "##downgrades for my existing packages##" section are being upgraded. For example:

Found existing installation: six 1.7.3
...
Uninstalling six-1.7.3:
Successfully uninstalled six-1.7.3
...
Successfully installed six-1.10.0

Package	Existing Version	Version after aws-shell install
six	1.7.3	1.10.0
python-dateutil	1.5	2.4.2
configobj	4.7.2	5.0.6

Are the new versions of the packages causing issues for you?

[orome]

Still downgrading colorama (to 0.3.3) and prompt-toolkit (to 0.52).

[Fale]

Please, update to current prompt_toolkit (0.57)

[jamesls]

We're now using v1.0.0 of prompt_toolkit so I believe you should no longer be seeing any of these issues. I'm going to go ahead and close out this issue now, but if anyone's still seeing issues, please let us know.