search

econ-ark / KrusellSmith

13 stars 10 forks source link

installation error #3

Open floswald opened 3 years ago

floswald commented 3 years ago

hi there

found this. any ideas? thanks

➜  KrusellSmith git:(master) ✗ ./reproduce.sh 
Requirement already satisfied: matplotlib==3.3.3 in /Users/74097/.pyenv/versions/3.6.1/lib/python3.6/site-packages (from -r requirements.txt (line 1)) (3.3.3)
Collecting econ-ark==0.10.7
  Downloading econ-ark-0.10.7.tar.gz (224 kB)
     |████████████████████████████████| 224 kB 18.9 MB/s 
  WARNING: Generating metadata for package econ-ark produced metadata for project name unknown. Fix your #egg=econ-ark fragments.
ERROR: Requested unknown from https://files.pythonhosted.org/packages/73/35/3fc0d840b4f7ffdc380005c3849c07ed8c4025f9762c490c69f9e25848ef/econ-ark-0.10.7.tar.gz#sha256=2e528c003e0ae2c70661ed2f0d1288b4b87414e6ecae387b7e5ecbc9048746bb (from -r requirements.txt (line 3)) has different name in metadata: 'UNKNOWN'
WARNING: You are using pip version 20.3; however, version 21.0.1 is available.
You should consider upgrading via the '/Users/74097/.pyenv/versions/3.6.1/bin/python3 -m pip install --upgrade pip' command.
Using matplotlib backend: MacOSX
---------------------------------------------------------------------------
ModuleNotFoundError                       Traceback (most recent call last)
~/git/KrusellSmith/Code/Python/KrusellSmith.py in <module>
    178 
    179 from copy import deepcopy
--> 180 from HARK.utilities import plotFuncs, plotFuncsDer, make_figs
    181 from HARK.distribution import DiscreteDistribution
    182 

ModuleNotFoundError: No module named 'HARK'
➜  KrusellSmith git:(master) ✗ python
➜  KrusellSmith git:(master) ✗ ls
Code             KrusellSmith.bib reproduce.sh
Figures          README.md        requirements.txt
➜  KrusellSmith git:(master) ✗ cat requirements.txt 
matplotlib==3.3.3
numpy==1.18
econ-ark==0.10.7
➜  KrusellSmith git:(master) ✗ python --version
Python 3.6.1
MridulS commented 3 years ago

thanks for the report. Can you try installing econ-ark in the current environment without using the reproduce script? 

$ pip install econ-ark==0.10.7

If that errors out too, can you check?

$ pip install econ-ark

Also there is another way of reproducing this is by using https://econ-ark.github.io/nbreproduce/ (you would need docker on your local machine).

You can install it with pip install nbreproduce and then just run $ nbreproduce in the local KrusellSmith directory. https://econ-ark.github.io/nbreproduce/scripts/#example

floswald commented 3 years ago 
74097@pst6977-1 ~/g/KrusellSmith (master)> pip install econ-ark==0.10.7
Collecting econ-ark==0.10.7
  Using cached econ-ark-0.10.7.tar.gz (224 kB)
  WARNING: Generating metadata for package econ-ark produced metadata for project name unknown. Fix your #egg=econ-ark fragments.
ERROR: Requested unknown from https://files.pythonhosted.org/packages/73/35/3fc0d840b4f7ffdc380005c3849c07ed8c4025f9762c490c69f9e25848ef/econ-ark-0.10.7.tar.gz#sha256=2e528c003e0ae2c70661ed2f0d1288b4b87414e6ecae387b7e5ecbc9048746bb has different name in metadata: 'UNKNOWN'
WARNING: You are using pip version 20.3; however, version 21.0.1 is available.
You should consider upgrading via the '/Users/74097/.pyenv/versions/3.6.1/bin/python3.6 -m pip install --upgrade pip' command.
74097@pst6977-1 ~/g/KrusellSmith (master) [1]> pip install econ-ark
Collecting econ-ark
  Downloading econ-ark-0.10.8.tar.gz (247 kB)
     |████████████████████████████████| 247 kB 21.7 MB/s 
  WARNING: Generating metadata for package econ-ark produced metadata for project name unknown. Fix your #egg=econ-ark fragments.
ERROR: Requested unknown from https://files.pythonhosted.org/packages/13/f3/8ec5fb5918e0682abf58c09410163124ec0c2b4cf302b79bfa6467b32a0c/econ-ark-0.10.8.tar.gz#sha256=f15bfd3a1a423f175cfaaabcb415f83182c844553cd9b6bfd45c816c8a242923 has different name in metadata: 'UNKNOWN'
WARNING: You are using pip version 20.3; however, version 21.0.1 is available.
You should consider upgrading via the '/Users/74097/.pyenv/versions/3.6.1/bin/python3.6 -m pip install --upgrade pip' command.

there is no way to run this without a local installation of docker?

MridulS commented 3 years ago

You just need econ-ark on your machine. pip install should just work but I am not able to figure out what's happening with the pip install. Everything looks okay when I try to install econ-ark on a fresh machine. Can you install other pip packages? Like pip install -U numpy or maybe something you don't have already locally?

Do you have conda available? You could try installing HARK with conda install econ-ark -c conda-forge.

floswald commented 3 years ago 
74097@pst6977-1 ~/g/KrusellSmith (master) [1]> pip install numpy
Requirement already satisfied: numpy in /Users/74097/.pyenv/versions/3.6.1/lib/python3.6/site-packages (1.19.5)
74097@pst6977-1 ~/g/KrusellSmith (master) [1]> pip install numba
Collecting numba
  Downloading numba-0.52.0-cp36-cp36m-macosx_10_14_x86_64.whl (2.2 MB)
     |████████████████████████████████| 2.2 MB 22.5 MB/s 
Collecting llvmlite<0.36,>=0.35.0
  Downloading llvmlite-0.35.0-cp36-cp36m-macosx_10_9_x86_64.whl (18.5 MB)
     |████████████████████████████████| 18.5 MB 15.5 MB/s 
Requirement already satisfied: numpy>=1.15 in /Users/74097/.pyenv/versions/3.6.1/lib/python3.6/site-packages (from numba) (1.19.5)
Requirement already satisfied: setuptools in /Users/74097/.pyenv/versions/3.6.1/lib/python3.6/site-packages (from numba) (28.8.0)
Installing collected packages: llvmlite, numba
Successfully installed llvmlite-0.35.0 numba-0.52.0

i think my pip is working fine. i'm using pyenv as you can see to manage different versions. do you think your install does not work with virtual envs? I am hesitant to install conda because it's a mess.

FWIW, the conda install works (running this inside julia), so i should be able to run it there, but that's not a great solution.

74097@pst6977-1 ~/g/KrusellSmith (master)> julia
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.5.3 (2020-11-09)
 _/ |\__'_|_|_|\__'_|  |  Official https://julialang.org/ release
|__/                   |

julia> using Conda

julia> Conda.add("econ-ark")
[ Info: Running `conda install -y econ-ark` in root environment
Collecting package metadata (current_repodata.json): done
Solving environment: done

## Package Plan ##

  environment location: /Users/74097/.julia/conda/3

  added / updated specs:
    - econ-ark

The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    blosc-1.21.0               |       h046ec9c_0         920 KB  conda-forge
    bzip2-1.0.8                |       hc929b4f_4         155 KB  conda-forge
    c-ares-1.17.1              |       hc929b4f_0          95 KB  conda-forge
    econ-ark-0.10.8            |     pyhd8ed1ab_1         196 KB  conda-forge
    hdf5-1.10.6                |nompi_hc5d9132_1114         3.1 MB  conda-forge
    interpolation-2.1.6        |             py_0          44 KB  conda-forge
    joblib-1.0.0               |     pyhd8ed1ab_0         206 KB  conda-forge
    libcurl-7.71.1             |       h9bf37e3_8         297 KB  conda-forge
    libev-4.33                 |       haf1e3a3_1          99 KB  conda-forge
    libllvm10-10.0.1           |       h009f743_3        20.8 MB  conda-forge
    libnghttp2-1.41.0          |       h7580e61_2         831 KB  conda-forge
    libssh2-1.9.0              |       h8a08a2b_5         214 KB  conda-forge
    llvmlite-0.35.0            |   py37hd32b0aa_1         224 KB  conda-forge
    mock-4.0.3                 |   py37hf985489_1          51 KB  conda-forge
    nose-1.3.7                 |          py_1006         118 KB  conda-forge
    numba-0.52.0               |   py37h010c265_0         3.5 MB  conda-forge
    numexpr-2.7.2              |   py37h010c265_0         130 KB  conda-forge
    pandas-1.2.1               |   py37h010c265_0        10.4 MB  conda-forge
    pytables-3.6.1             |   py37h6293439_3         1.4 MB  conda-forge
    pytz-2021.1                |     pyhd8ed1ab_0         239 KB  conda-forge
    quantecon-0.4.8            |   py37hc8dfbb8_0         324 KB  conda-forge
    scipy-1.6.0                |   py37h6bb7d36_0        18.4 MB  conda-forge
    tempita-0.5.3dev           |py37hc8dfbb8_1002          28 KB  conda-forge
    ------------------------------------------------------------
                                           Total:        61.5 MB

The following NEW packages will be INSTALLED:

  blosc              conda-forge/osx-64::blosc-1.21.0-h046ec9c_0
  bzip2              conda-forge/osx-64::bzip2-1.0.8-hc929b4f_4
  c-ares             conda-forge/osx-64::c-ares-1.17.1-hc929b4f_0
  econ-ark           conda-forge/noarch::econ-ark-0.10.8-pyhd8ed1ab_1
  hdf5               conda-forge/osx-64::hdf5-1.10.6-nompi_hc5d9132_1114
  interpolation      conda-forge/noarch::interpolation-2.1.6-py_0
  joblib             conda-forge/noarch::joblib-1.0.0-pyhd8ed1ab_0
  libcurl            conda-forge/osx-64::libcurl-7.71.1-h9bf37e3_8
  libev              conda-forge/osx-64::libev-4.33-haf1e3a3_1
  libllvm10          conda-forge/osx-64::libllvm10-10.0.1-h009f743_3
  libnghttp2         conda-forge/osx-64::libnghttp2-1.41.0-h7580e61_2
  libssh2            conda-forge/osx-64::libssh2-1.9.0-h8a08a2b_5
  llvmlite           conda-forge/osx-64::llvmlite-0.35.0-py37hd32b0aa_1
  mock               conda-forge/osx-64::mock-4.0.3-py37hf985489_1
  nose               conda-forge/noarch::nose-1.3.7-py_1006
  numba              conda-forge/osx-64::numba-0.52.0-py37h010c265_0
  numexpr            conda-forge/osx-64::numexpr-2.7.2-py37h010c265_0
  pandas             conda-forge/osx-64::pandas-1.2.1-py37h010c265_0
  pytables           conda-forge/osx-64::pytables-3.6.1-py37h6293439_3
  pytz               conda-forge/noarch::pytz-2021.1-pyhd8ed1ab_0
  quantecon          conda-forge/osx-64::quantecon-0.4.8-py37hc8dfbb8_0
  scipy              conda-forge/osx-64::scipy-1.6.0-py37h6bb7d36_0
  tempita            conda-forge/osx-64::tempita-0.5.3dev-py37hc8dfbb8_1002

Downloading and Extracting Packages
libllvm10-10.0.1     | 20.8 MB   | ############################################################################## | 100% 
bzip2-1.0.8          | 155 KB    | ############################################################################## | 100% 
numba-0.52.0         | 3.5 MB    | ############################################################################## | 100% 
interpolation-2.1.6  | 44 KB     | ############################################################################## | 100% 
libssh2-1.9.0        | 214 KB    | ############################################################################## | 100% 
tempita-0.5.3dev     | 28 KB     | ############################################################################## | 100% 
llvmlite-0.35.0      | 224 KB    | ############################################################################## | 100% 
econ-ark-0.10.8      | 196 KB    | ############################################################################## | 100% 
hdf5-1.10.6          | 3.1 MB    | ############################################################################## | 100% 
libnghttp2-1.41.0    | 831 KB    | ############################################################################## | 100% 
scipy-1.6.0          | 18.4 MB   | ############################################################################## | 100% 
pytables-3.6.1       | 1.4 MB    | ############################################################################## | 100% 
libcurl-7.71.1       | 297 KB    | ############################################################################## | 100% 
pandas-1.2.1         | 10.4 MB   | ############################################################################## | 100% 
libev-4.33           | 99 KB     | ############################################################################## | 100% 
mock-4.0.3           | 51 KB     | ############################################################################## | 100% 
nose-1.3.7           | 118 KB    | ############################################################################## | 100% 
blosc-1.21.0         | 920 KB    | ############################################################################## | 100% 
c-ares-1.17.1        | 95 KB     | ############################################################################## | 100% 
pytz-2021.1          | 239 KB    | ############################################################################## | 100% 
quantecon-0.4.8      | 324 KB    | ############################################################################## | 100% 
joblib-1.0.0         | 206 KB    | ############################################################################## | 100% 
numexpr-2.7.2        | 130 KB    | ############################################################################## | 100% 
Preparing transaction: done
Verifying transaction: done
Executing transaction: done

julia>
MridulS commented 3 years ago

looks like this could also be caused by an old version of setuptools, can you try running the following commands?

pip install setuptools --upgrade
pip install --no-cache-dir econ-ark

thanks!

floswald commented 3 years ago

bingo! thanks

74097@pst6977-1 ~> pip install setuptools --upgrade
Requirement already satisfied: setuptools in ./.pyenv/versions/3.6.1/lib/python3.6/site-packages (28.8.0)
Collecting setuptools
  Downloading setuptools-53.0.0-py3-none-any.whl (784 kB)
     |████████████████████████████████| 784 kB 22.4 MB/s 
Installing collected packages: setuptools
  Attempting uninstall: setuptools
    Found existing installation: setuptools 28.8.0
    Uninstalling setuptools-28.8.0:
      Successfully uninstalled setuptools-28.8.0
Successfully installed setuptools-53.0.0
74097@pst6977-1 ~> pip install --no-cache-dir econ-ark
Collecting econ-ark
  Downloading econ-ark-0.10.8.tar.gz (247 kB)
     |████████████████████████████████| 247 kB 23.8 MB/s 
Requirement already satisfied: numpy in ./.pyenv/versions/3.6.1/lib/python3.6/site-packages (from econ-ark) (1.19.5)
Requirement already satisfied: matplotlib in ./.pyenv/versions/3.6.1/lib/python3.6/site-packages (from econ-ark) (3.3.3)
Collecting scipy
  Downloading scipy-1.5.4-cp36-cp36m-macosx_10_9_x86_64.whl (28.8 MB)
     |████████████████████████████████| 28.8 MB 53.6 MB/s 
Collecting dill
  Downloading dill-0.3.3-py2.py3-none-any.whl (81 kB)
     |████████████████████████████████| 81 kB 75.2 MB/s 
Collecting joblib
  Downloading joblib-1.0.0-py3-none-any.whl (302 kB)
     |████████████████████████████████| 302 kB 73.3 MB/s 
Collecting quantecon
  Downloading quantecon-0.4.8-py3-none-any.whl (230 kB)
     |████████████████████████████████| 230 kB 80.2 MB/s 
Collecting interpolation
  Downloading interpolation-2.1.6-py3-none-any.whl (54 kB)
     |████████████████████████████████| 54 kB 51.4 MB/s 
Requirement already satisfied: numba in ./.pyenv/versions/3.6.1/lib/python3.6/site-packages (from econ-ark) (0.52.0)
Collecting tempita>=0.5.2
  Downloading Tempita-0.5.2.tar.gz (12 kB)
Requirement already satisfied: setuptools in ./.pyenv/versions/3.6.1/lib/python3.6/site-packages (from numba->econ-ark) (53.0.0)
Requirement already satisfied: llvmlite<0.36,>=0.35.0 in ./.pyenv/versions/3.6.1/lib/python3.6/site-packages (from numba->econ-ark) (0.35.0)
Requirement already satisfied: cycler>=0.10 in ./.pyenv/versions/3.6.1/lib/python3.6/site-packages (from matplotlib->econ-ark) (0.10.0)
Requirement already satisfied: python-dateutil>=2.1 in ./.pyenv/versions/3.6.1/lib/python3.6/site-packages (from matplotlib->econ-ark) (2.8.1)
Requirement already satisfied: kiwisolver>=1.0.1 in ./.pyenv/versions/3.6.1/lib/python3.6/site-packages (from matplotlib->econ-ark) (1.3.1)
Requirement already satisfied: pillow>=6.2.0 in ./.pyenv/versions/3.6.1/lib/python3.6/site-packages (from matplotlib->econ-ark) (8.1.0)
Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.3 in ./.pyenv/versions/3.6.1/lib/python3.6/site-packages (from matplotlib->econ-ark) (2.4.7)
Requirement already satisfied: six in ./.pyenv/versions/3.6.1/lib/python3.6/site-packages (from cycler>=0.10->matplotlib->econ-ark) (1.15.0)
Collecting requests
  Downloading requests-2.25.1-py2.py3-none-any.whl (61 kB)
     |████████████████████████████████| 61 kB 57.1 MB/s 
Collecting sympy
  Downloading sympy-1.7.1-py3-none-any.whl (5.9 MB)
     |████████████████████████████████| 5.9 MB 70.3 MB/s 
Requirement already satisfied: idna<3,>=2.5 in ./.pyenv/versions/3.6.1/lib/python3.6/site-packages (from requests->quantecon->econ-ark) (2.10)
Collecting chardet<5,>=3.0.2
  Downloading chardet-4.0.0-py2.py3-none-any.whl (178 kB)
     |████████████████████████████████| 178 kB 69.4 MB/s 
Collecting urllib3<1.27,>=1.21.1
  Downloading urllib3-1.26.3-py2.py3-none-any.whl (137 kB)
     |████████████████████████████████| 137 kB 71.2 MB/s 
Collecting certifi>=2017.4.17
  Downloading certifi-2020.12.5-py2.py3-none-any.whl (147 kB)
     |████████████████████████████████| 147 kB 68.9 MB/s 
Collecting mpmath>=0.19
  Downloading mpmath-1.1.0.tar.gz (512 kB)
     |████████████████████████████████| 512 kB 66.0 MB/s 
Using legacy 'setup.py install' for econ-ark, since package 'wheel' is not installed.
Using legacy 'setup.py install' for tempita, since package 'wheel' is not installed.
Using legacy 'setup.py install' for mpmath, since package 'wheel' is not installed.
Installing collected packages: urllib3, mpmath, chardet, certifi, tempita, sympy, scipy, requests, quantecon, joblib, interpolation, dill, econ-ark
    Running setup.py install for mpmath ... done
    Running setup.py install for tempita ... done
    Running setup.py install for econ-ark ... done
Successfully installed certifi-2020.12.5 chardet-4.0.0 dill-0.3.3 econ-ark-0.10.8 interpolation-2.1.6 joblib-1.0.0 mpmath-1.1.0 quantecon-0.4.8 requests-2.25.1 scipy-1.5.4 sympy-1.7.1 tempita-0.5.2 urllib3-1.26.3
MridulS commented 3 years ago

wohoooo 🎉🎉🎉

llorracc commented 3 years ago

@MridulS,

I'd prefer not to close an issue like this without a fuller understanding of what went awry and how we might be able to guard against it in the future.

Did our requirements.txt not specify a particular version of setuptools but it should have? Or, what else could we have done to prevent Florian's problems?

PS. This is another illustration of the point that I keep making that it's super important that the FIRST thing people try should be something that works off the bat -- so we should work harder to structure things so the overwhelmingly obvious option is for them to get launched into the guaranteed-to-work online docker-powered version of a REMARK. Then they'll see how great it is, and might be motivated to fiddle with their system to get all the dependencies right and can run it locally. Florian's such a trooper that he actually posted an issue, but for every person who posts an issue there are probably a hundred who tried for a while and gave up.

Florian, to help us design things better, it would be useful to know the path by which you came to this.

One way to get there is via

https://econ-ark.org

by going to "Materials" and finding the "KrusellSmith" item

Another is just to find the repo directly (say, by a Google search).

There are several others.

An annoying thing about GitHub is that at present its rendering of Jupyter notebooks is very unreliable, so we can't solve our navigational problems by just putting a clickable link at the top of the Jupyter notebook -- half the time, the link will not appear!

MridulS commented 3 years ago

Did our requirements.txt not specify a particular version of setuptools but it should have? Or, what else could we have done to prevent Florian's problems?

This was a local machine environment problem not an econ-ark installation one, we don't control setuptools installation with econ-ark.

but for every person who posts an issue there are probably a hundred who tried for a while and gave up.

Unfortunately its very easy for something in the python environment to go wrong, this issue seems like a usual one (https://stackoverflow.com/questions/55411139/cairocffi-install-failed-on-ubuntu-18-04). It would be great if we could somehow figure out how to encourage people to open up issues whenever they get stuck with these things and we could help them out.

floswald commented 3 years ago

I don't envy you for this issue @MridulS :wink:, I can see dependency hell can get quite hot here in python land. I feel there is going to be a divide between very experienced python users who can deal with this, and others less advanced (like myself) who won't. An easy suggestion would be to force user to use one particular environment (like: you must install anaconda for this) but this seems really heavy handed - I would refuse, because it messes up my system installation. so the real problem I think is that people show up with vastly different python envs as you say, and it's hard to place assumptions on that.

that said: what happens if you really just hard wire the setup-tools package to a higher version in your requirements.txt? would that break something else?

It's really frustrating to see that after all the work you guys have put into this, the user fails at this crucial step (how to run this??). @llorracc I've seen people putting badges with a link to a binder repo where the notebook then runs, and it seemed to be very robust. Is that not an option for you? Yes you don't want to rely on github rendering a notebook, half the time it won't (it's not built for serving stuff). another option for showing rendered notebooks could be https://nbviewer.jupyter.org . you probably went through most of those thoughts already, anyway, in case not maybe it could help.

floswald commented 3 years ago

oh - i found the repo by browsing your econ-ark.org website which is very nice!

MridulS commented 3 years ago

that said: what happens if you really just hard wire the setup-tools package to a higher version in your requirements.txt? would that break something else?

We (as a package) add a bit more to the dependency hell of python 🚀 To run HARK there isn't any direct dependency on setuptools, the pip install part uses setup-tools and in this case looks like something funky was happening there (pip install part). And we should absolutely not add a dependency which isn't required and the setuptools part is assumed to be offloaded to the local python installation.

BTW the "live interaction version" button on https://econ-ark.org/materials/krusellsmith takes you to mybinder. Should we be more explicit with something like "Interactive version (mybinder)" for people who do know about mybinder? I have added the "launch binder" button in this repo README too. Thanks for the suggestions!

We are also in the process of publishing a new website with more user actions like https://staging.econ-ark.org/materials/liqconstr.