Properly document steem-python packaging

[theoreticalbts]

This ticket is about the awful experiences I'm having with the Python package manager and steem-python.

Let's say I want to create a simple tool called mytool that leverages steem-python, running on Ubuntu 16.04. I create minimal source files by cd ~/src and then executing the following script:

#!/bin/bash                                                                                                                                                                                                         

mkdir -p mytool
cd mytool
mkdir -p mytool

cat >setup.py >>EOF
from setuptools import setup

setup(name="mytool",
      version          = 0.1,
      description      = "Use steem-python to do something useful",
      url              = "",
      author           = "Steemit",
      packages         = ["mytool"],
      install_requires = ["steem"],
      entry_points     = {"console_scripts" : [
                          "mytool=mytool.main:main",
                         ]}
    )

EOF

touch mytool/__init__.py

cat >mytool/main.py >>EOF

import steem

def main():
    print("hello steem world")

if __name__ == "__main__":
    main()
EOF

Then I build Python 3.6.3 locally from source with --prefix=$HOME/opt/python3.6.3 followed by sudo apt-get install virtualenv, followed by using pip to install the newly created tool:

virtualenv -p "$HOME/opt/python3.6.3/bin/python3" ~/ve/mytool
. ~/ve/mytool/bin/activate
pip install ~/src/mytool
mytool              # fails

All is well, but actually executing mytool command in the virtualenv causes the following error:

Traceback (most recent call last):
  File "/home/theoretical/ve/mytool/lib/python3.6/site-packages/pkg_resources/__init__.py", line 635, in _build_master
    ws.require(__requires__)
  File "/home/theoretical/ve/mytool/lib/python3.6/site-packages/pkg_resources/__init__.py", line 943, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/home/theoretical/ve/mytool/lib/python3.6/site-packages/pkg_resources/__init__.py", line 834, in resolve
    raise VersionConflict(dist, req).with_context(dependent_req)
pkg_resources.ContextualVersionConflict: (toml 0.9.3 (/home/theoretical/ve/mytool/lib/python3.6/site-packages), Requirement.parse('toml==0.9.3.1'), {'steem'})

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/theoretical/ve/mytool/bin/mytool", line 6, in <module>
    from pkg_resources import load_entry_point
  File "/home/theoretical/ve/mytool/lib/python3.6/site-packages/pkg_resources/__init__.py", line 2927, in <module>
    @_call_aside
  File "/home/theoretical/ve/mytool/lib/python3.6/site-packages/pkg_resources/__init__.py", line 2913, in _call_aside
    f(*args, **kwargs)
  File "/home/theoretical/ve/mytool/lib/python3.6/site-packages/pkg_resources/__init__.py", line 2940, in _initialize_master_working_set
    working_set = WorkingSet._build_master()
  File "/home/theoretical/ve/mytool/lib/python3.6/site-packages/pkg_resources/__init__.py", line 637, in _build_master
    return cls._build_from_requirements(__requires__)
  File "/home/theoretical/ve/mytool/lib/python3.6/site-packages/pkg_resources/__init__.py", line 650, in _build_from_requirements
    dists = ws.resolve(reqs, Environment())
  File "/home/theoretical/ve/mytool/lib/python3.6/site-packages/pkg_resources/__init__.py", line 829, in resolve
    raise DistributionNotFound(req, requirers)
pkg_resources.DistributionNotFound: The 'toml==0.9.3.1' distribution was not found and is required by steem

I usually follow this guide to create Python packages. If this process doesn't work, we need to either make it work, or figure out some alternative process that does work, and then document it.

I don't really understand how steem-python itself is packaged, it apparently uses a complex process involving Pipfile.lock, setup.py and package_meta.py. More in the next post!

[theoreticalbts]

So I did some Googling after writing most of the above post, and I'm learning all about pipenv here. First of all, I don't really like its installation process. I end up doing something long to try to keep things isolated in a virtualenv and not do things user-wide:

virtualenv -p $(which python3) ~/ve/pipsi
~/ve/pipsi/bin/pip install pipsi      # not installing pipsi with curl | python, I hate that anti-pattern
~/ve/pipsi/bin/pipsi --home ~/pipsi-ve --bin-dir ~/pipsi-bin install pew
~/ve/pipsi/bin/pipsi --home ~/pipsi-ve --bin-dir ~/pipsi-bin install pipenv
mkdir -p ~/bin
export PATH=$PATH:$HOME/bin     # also add this to .bashrc
ln -s ../pipsi-bin/pew ~/bin/pew
ln -s ../pipsi-bin/pipenv ~/bin/pipenv

Then I try running pipenv install in the directory for steem-python:

$ pipenv install --python ~/opt/python3.6.3/bin/python3.6
Creating a virtualenv for this project…
Using /home/theoretical/opt/python3.6.3/bin/python3.6 to create virtualenv…
⠋Running virtualenv with interpreter /home/theoretical/opt/python3.6.3/bin/python3.6
Using base prefix '/home/theoretical/opt/python3.6.3'
New python executable in /home/theoretical/.local/share/virtualenvs/steem-python-Zkm9LBqG/bin/python3.6
Also creating executable in /home/theoretical/.local/share/virtualenvs/steem-python-Zkm9LBqG/bin/python
Installing setuptools, pip, wheel...done.

Virtualenv location: /home/theoretical/.local/share/virtualenvs/steem-python-Zkm9LBqG                                                       Installing dependencies from Pipfile.lock (30d256)…
An error occurred while installing ruamel.yaml==0.15.34! Will try again.
  🐍   ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 31/31 — 00:01:00
An error occurred while installing pendulum==1.3.0! Will try again.
Installing initially–failed dependencies…
Collecting ruamel.yaml==0.15.34 ▉▉▉▉▉ 0/2 — 00:00:00
  Using cached ruamel.yaml-0.15.34-cp36-cp36m-manylinux1_x86_64.whl

THESE PACKAGES DO NOT MATCH THE HASHES FROM Pipfile.lock!. If you have updated the package versions, please update the hashes. Otherwise, examine the package contents carefully; someone may have tampered with them.
    ruamel.yaml==0.15.34 from https://pypi.python.org/packages/88/54/4efcae595c6437a0e548ced166ea44a0d7aa200f903aaf7174d74952c5b3/ruamel.yaml-0.15.34-cp36-cp36m-manylinux1_x86_64.whl#md5=567097db4592e9e5b0581c5a379715be (from -r /tmp/pipenv-uy1s4cux-requirement.txt (line 1)):
        Expected sha256 d92d90c9bc0945223e47223a67808dd97ac9390ed914cc6871479b7ba489e607
             Got        2f28a3b6665697c20d841a4aee21cfb932bb0db91f293ff97daf845b914ddddb

  ☤  ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 0/2 — 00:00:00

[theoreticalbts]

@jnordberg @john-g-g I see yo uses steem as a package and doesn't have these issues, do you have any insights?

[holgern]

You can try my conda-forge version: https://github.com/conda-forge/steem-feedstock https://utopian.io/utopian-io/@holger80/install-steem-python-easily-by-conda-forge

[cyon1c]

These issues should be resolved with commit 1d7e7f3.