QuantEcon / QuantEcon.jl

Julia implementation of QuantEcon routines
https://quantecon.org/quantecon-jl/
BSD 3-Clause "New" or "Revised" License
506 stars 303 forks source link

QuantEcon failing to load on Ubuntu LTS 16.04 #108

Closed jstac closed 8 years ago

jstac commented 8 years ago

This is what I'm getting. It's actually been failing for a few days, and since before I upgraded to LTS 16.04.

Julia is Version 0.4.5 (2016-03-18 00:58 UTC), downloaded as a PPA.

julia> Pkg.update()
INFO: Updating METADATA...
INFO: Updating cache of Combinatorics...
INFO: Updating cache of FixedPointNumbers...
INFO: Updating cache of Compat...
INFO: Computing changes...
INFO: Upgrading Combinatorics: v0.2.0 => v0.2.1
INFO: Upgrading Compat: v0.7.13 => v0.7.14
INFO: Upgrading FixedPointNumbers: v0.1.2 => v0.1.3

julia> using QuantEcon
INFO: Precompiling module QuantEcon...
INFO: Recompiling stale cache file /home/john/.julia/lib/v0.4/Compat.ji for module Compat.
INFO: Recompiling stale cache file /home/john/.julia/lib/v0.4/StatsFuns.ji for module StatsFuns.
INFO: Recompiling stale cache file /home/john/.julia/lib/v0.4/Distributions.ji for module Distributions.
INFO: Recompiling stale cache file /home/john/.julia/lib/v0.4/ArrayViews.ji for module ArrayViews.
INFO: Recompiling stale cache file /home/john/.julia/lib/v0.4/PDMats.ji for module PDMats.
INFO: Recompiling stale cache file /home/john/.julia/lib/v0.4/StatsBase.ji for module StatsBase.
INFO: Recompiling stale cache file /home/john/.julia/lib/v0.4/DSP.ji for module DSP.
INFO: Recompiling stale cache file /home/john/.julia/lib/v0.4/Polynomials.ji for module Polynomials.
INFO: Recompiling stale cache file /home/john/.julia/lib/v0.4/GZip.ji for module GZip.
INFO: Recompiling stale cache file /home/john/.julia/lib/v0.4/LightXML.ji for module LightXML.
INFO: Recompiling stale cache file /home/john/.julia/lib/v0.4/ParserCombinator.ji for module ParserCombinator.
INFO: Recompiling stale cache file /home/john/.julia/lib/v0.4/Clustering.ji for module Clustering.
INFO: Recompiling stale cache file /home/john/.julia/lib/v0.4/Combinatorics.ji for module Combinatorics.
INFO: Recompiling stale cache file /home/john/.julia/lib/v0.4/Iterators.ji for module Iterators.
WARNING: Method definition factorial(#T<:Integer, #T<:Integer) in module Base at combinatorics.jl:56 overwritten in module Combinatorics at /home/john/.julia/v0.4/Combinatorics/src/factorials.jl:18.
WARNING: Method definition factorial(Integer, Integer) in module Base at combinatorics.jl:66 overwritten in module Combinatorics at /home/john/.julia/v0.4/Combinatorics/src/factorials.jl:28.
WARNING: Method definition combinations(Any, Integer) in module Base at combinatorics.jl:182 overwritten in module Combinatorics at /home/john/.julia/v0.4/Combinatorics/src/combinations.jl:42.
WARNING: Method definition permutations(Any) in module Base at combinatorics.jl:219 overwritten in module Combinatorics at /home/john/.julia/v0.4/Combinatorics/src/permutations.jl:24.
WARNING: Method definition nthperm!(AbstractArray{T<:Any, 1}, Integer) in module Base at combinatorics.jl:70 overwritten in module Combinatorics at /home/john/.julia/v0.4/Combinatorics/src/permutations.jl:136.
WARNING: Method definition nthperm(AbstractArray{T<:Any, 1}, Integer) in module Base at combinatorics.jl:89 overwritten in module Combinatorics at /home/john/.julia/v0.4/Combinatorics/src/permutations.jl:157.
WARNING: Method definition nthperm(AbstractArray{#T<:Integer, 1}) in module Base at combinatorics.jl:92 overwritten in module Combinatorics at /home/john/.julia/v0.4/Combinatorics/src/permutations.jl:161.
WARNING: Method definition levicivita(AbstractArray{#T<:Integer, 1}) in module Base at combinatorics.jl:611 overwritten in module Combinatorics at /home/john/.julia/v0.4/Combinatorics/src/permutations.jl:188.
WARNING: Method definition parity(AbstractArray{#T<:Integer, 1}) in module Base at combinatorics.jl:642 overwritten in module Combinatorics at /home/john/.julia/v0.4/Combinatorics/src/permutations.jl:221.
WARNING: Method definition partitions(Integer) in module Base at combinatorics.jl:252 overwritten in module Combinatorics at /home/john/.julia/v0.4/Combinatorics/src/partitions.jl:26.
WARNING: Method definition partitions(Integer, Integer) in module Base at combinatorics.jl:318 overwritten in module Combinatorics at /home/john/.julia/v0.4/Combinatorics/src/partitions.jl:93.
WARNING: Method definition partitions(AbstractArray{T<:Any, 1}) in module Base at combinatorics.jl:380 overwritten in module Combinatorics at /home/john/.julia/v0.4/Combinatorics/src/partitions.jl:158.
WARNING: Method definition partitions(AbstractArray{T<:Any, 1}, Int64) in module Base at combinatorics.jl:447 overwritten in module Combinatorics at /home/john/.julia/v0.4/Combinatorics/src/partitions.jl:228.
WARNING: Method definition prevprod(Array{Int64, 1}, Any) in module Base at combinatorics.jl:565 overwritten in module Combinatorics at /home/john/.julia/v0.4/Combinatorics/src/partitions.jl:354.
WARNING: Method definition levicivita(AbstractArray{#T<:Integer, 1}) in module Base at combinatorics.jl:611 overwritten in module Combinatorics at /home/john/.julia/v0.4/Combinatorics/src/permutations.jl:188.
WARNING: Method definition nthperm(AbstractArray{#T<:Integer, 1}) in module Base at combinatorics.jl:92 overwritten in module Combinatorics at /home/john/.julia/v0.4/Combinatorics/src/permutations.jl:161.
WARNING: Method definition nthperm(AbstractArray{T<:Any, 1}, Integer) in module Base at combinatorics.jl:89 overwritten in module Combinatorics at /home/john/.julia/v0.4/Combinatorics/src/permutations.jl:157.
WARNING: Method definition partitions(Integer) in module Base at combinatorics.jl:252 overwritten in module Combinatorics at /home/john/.julia/v0.4/Combinatorics/src/partitions.jl:26.
WARNING: Method definition partitions(Integer, Integer) in module Base at combinatorics.jl:318 overwritten in module Combinatorics at /home/john/.julia/v0.4/Combinatorics/src/partitions.jl:93.
WARNING: Method definition partitions(AbstractArray{T<:Any, 1}) in module Base at combinatorics.jl:380 overwritten in module Combinatorics at /home/john/.julia/v0.4/Combinatorics/src/partitions.jl:158.
WARNING: Method definition partitions(AbstractArray{T<:Any, 1}, Int64) in module Base at combinatorics.jl:447 overwritten in module Combinatorics at /home/john/.julia/v0.4/Combinatorics/src/partitions.jl:228.
WARNING: Method definition combinations(Any, Integer) in module Base at combinatorics.jl:182 overwritten in module Combinatorics at /home/john/.julia/v0.4/Combinatorics/src/combinations.jl:42.
WARNING: Method definition nthperm!(AbstractArray{T<:Any, 1}, Integer) in module Base at combinatorics.jl:70 overwritten in module Combinatorics at /home/john/.julia/v0.4/Combinatorics/src/permutations.jl:136.
WARNING: Method definition permutations(Any) in module Base at combinatorics.jl:219 overwritten in module Combinatorics at /home/john/.julia/v0.4/Combinatorics/src/permutations.jl:24.
WARNING: Method definition factorial(#T<:Integer, #T<:Integer) in module Base at combinatorics.jl:56 overwritten in module Combinatorics at /home/john/.julia/v0.4/Combinatorics/src/factorials.jl:18.
WARNING: Method definition factorial(Integer, Integer) in module Base at combinatorics.jl:66 overwritten in module Combinatorics at /home/john/.julia/v0.4/Combinatorics/src/factorials.jl:28.
WARNING: Method definition parity(AbstractArray{#T<:Integer, 1}) in module Base at combinatorics.jl:642 overwritten in module Combinatorics at /home/john/.julia/v0.4/Combinatorics/src/permutations.jl:221.
WARNING: Method definition prevprod(Array{Int64, 1}, Any) in module Base at combinatorics.jl:565 overwritten in module Combinatorics at /home/john/.julia/v0.4/Combinatorics/src/partitions.jl:354.
ERROR: LoadError: error compiling __init__: could not load library "/home/john/.julia/v0.4/Blosc/src/../deps/libblosc"
/home/john/.julia/v0.4/Blosc/src/../deps/libblosc: cannot open shared object file: No such file or directory
 in include at ./boot.jl:261
 in include_from_node1 at ./loading.jl:320
 [inlined code] from none:2
 in anonymous at no file:0
 in process_options at ./client.jl:257
 in _start at ./client.jl:378
while loading /home/john/.julia/v0.4/Blosc/src/Blosc.jl, in expression starting on line 216
ERROR: LoadError: LoadError: LoadError: Failed to precompile Blosc to /home/john/.julia/lib/v0.4/Blosc.ji
 in error at ./error.jl:21
 in compilecache at loading.jl:400
 in require at ./loading.jl:240
 in include at ./boot.jl:261
 in include_from_node1 at ./loading.jl:320
 in include at ./boot.jl:261
 in include_from_node1 at ./loading.jl:320
 in include at ./boot.jl:261
 in include_from_node1 at ./loading.jl:320
 [inlined code] from none:2
 in anonymous at no file:0
 in process_options at ./client.jl:257
 in _start at ./client.jl:378
while loading /home/john/.julia/v0.4/HDF5/src/blosc_filter.jl, in expression starting on line 1
while loading /home/john/.julia/v0.4/HDF5/src/plain.jl, in expression starting on line 509
while loading /home/john/.julia/v0.4/HDF5/src/HDF5.jl, in expression starting on line 3
ERROR: LoadError: Failed to precompile HDF5 to /home/john/.julia/lib/v0.4/HDF5.ji
 in error at ./error.jl:21
 in compilecache at loading.jl:400
 in require at ./loading.jl:240
 in include at ./boot.jl:261
 in include_from_node1 at ./loading.jl:320
 [inlined code] from none:2
 in anonymous at no file:0
 in process_options at ./client.jl:257
 in _start at ./client.jl:378
while loading /home/john/.julia/v0.4/JLD/src/JLD.jl, in expression starting on line 8
ERROR: LoadError: LoadError: Failed to precompile JLD to /home/john/.julia/lib/v0.4/JLD.ji
 in error at ./error.jl:21
 in compilecache at loading.jl:400
 in require at ./loading.jl:240
 in include at ./boot.jl:261
 in include_from_node1 at ./loading.jl:320
 in include at ./boot.jl:261
 in include_from_node1 at ./loading.jl:320
 [inlined code] from none:2
 in anonymous at no file:0
 in process_options at ./client.jl:257
 in _start at ./client.jl:378
while loading /home/john/.julia/v0.4/LightGraphs/src/persistence/jld.jl, in expression starting on line 1
while loading /home/john/.julia/v0.4/LightGraphs/src/LightGraphs.jl, in expression starting on line 159
ERROR: LoadError: LoadError: Failed to precompile LightGraphs to /home/john/.julia/lib/v0.4/LightGraphs.ji
 in error at ./error.jl:21
 in compilecache at loading.jl:400
 in require at ./loading.jl:240
 in include at ./boot.jl:261
 in include_from_node1 at ./loading.jl:320
 in include at ./boot.jl:261
 in include_from_node1 at ./loading.jl:320
 [inlined code] from none:2
 in anonymous at no file:0
 in process_options at ./client.jl:257
 in _start at ./client.jl:378
while loading /home/john/.julia/v0.4/QuantEcon/src/markov/mc_tools.jl, in expression starting on line 13
while loading /home/john/.julia/v0.4/QuantEcon/src/QuantEcon.jl, in expression starting on line 120
ERROR: Failed to precompile QuantEcon to /home/john/.julia/lib/v0.4/QuantEcon.ji
 in error at ./error.jl:21
 in compilecache at loading.jl:400
 in require at ./loading.jl:266
sglyon commented 8 years ago

sometimes Blosc has given people issues on ubuntu.

Can you try Pkg.build("Blosc"); using Blosc in a fresh Julia session and post the output?

sglyon commented 8 years ago

On second thought. I actually think we should publish another release of QuantEcon.jl as I believe Blosc is no longer a dependency

sglyon commented 8 years ago

Ok so I was wrong, the latest released version does not have HDF5 or Blosc as a dependency. So these error shouldn't be coming up.

If you cd into the repo ~/.julia/v0.4/QuantEcon, does git status say the repo is clean or has changes? If it has changes you probably aren't getting the most recent QuantEcon.jl. We should clean out your local copy of the repo so you can be on the latest release.

jstac commented 8 years ago

Running git status gives

HEAD detached at fe2be6b
nothing to commit, working directory clean

I've tried removing and reinstalling. I'll try again.

jstac commented 8 years ago

Blosc:

julia> Pkg.build("Blosc")
INFO: Building Blosc
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0curl: (77) error setting certificate verify locations:
  CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
================================[ ERROR: Blosc ]================================

LoadError: failed process: Process(`curl -f -o c-blosc-1.7.0.tar.gz -L https://github.com/Blosc/c-blosc/archive/v1.7.0.tar.gz`, ProcessExited(77)) [77]
while loading /home/john/.julia/v0.4/Blosc/deps/build.jl, in expression starting on line 18

================================================================================

==========================================[ BUILD ERRORS ]===========================================

WARNING: Blosc had build errors.

 - packages with build errors remain installed in /home/john/.julia/v0.4
 - build the package(s) and all dependencies with `Pkg.build("Blosc")`
 - build a single package by running its `deps/build.jl` script

=====================================================================================================
sglyon commented 8 years ago

Excellent. I've seen this one.

@cc7768 might know a magic trick to get around this one...

cc7768 commented 8 years ago

Blosc problem probably has to do with BinDeps. My typical fix to this is to go into the BinDeps/src/BinDeps.jl and change downloadcmd = nothing to downloadcmd = :wget. Try this and let me know whether it works.

albop commented 8 years ago

This last one is probably related to bug: https://github.com/JuliaWeb/MbedTLS.jl/issues/10 . It is apparently related to where the distribution (Ubuntu I guess) stores its certificates. Another fix, without touching julia sources is: echo "cacert=/etc/ssl/certs/ca-certificates.crt" > ~/.curlrc

sglyon commented 8 years ago

Thanks @cc7768

And yes, thank you @albop. That's the fix I had in mind.

jstac commented 8 years ago

@spencerlyon2 @cc7768 @albop Thanks guys. I used @albop 's fix. Blosc builds and QuantEcon loads, albeit with a bunch of warnings.

Anything we can do for other users? Do we need a message here:

http://quant-econ.net/jl/getting_started.html#the-quantecon-library

or here:

http://quantecon.org/julia_index.html

sglyon commented 8 years ago

I don't see any warnings when build/load quantecon. Can you post what you're seeing?

Some instructions might be helpful.

We really need to do an overhaul of the qe.jl docs. They are really out of date and should probably have a users' guide component in addition to the docstring generated API docs we have right now.

jstac commented 8 years ago

Very happy to devote resources to such things over summer, assuming our coffers are replenished. (We'll know the outcome of the application in mid June but I would say the probability is above 80%.) Even better if you can recruit suitable people and run that show.

Here's my output:

julia> using QuantEcon
WARNING: Method definition levicivita(AbstractArray{#T<:Integer, 1}) in module Base at combinatorics.jl:611 overwritten in module Combinatorics at /home/john/.julia/v0.4/Combinatorics/src/permutations.jl:188.
WARNING: Method definition nthperm(AbstractArray{#T<:Integer, 1}) in module Base at combinatorics.jl:92 overwritten in module Combinatorics at /home/john/.julia/v0.4/Combinatorics/src/permutations.jl:161.
WARNING: Method definition nthperm(AbstractArray{T<:Any, 1}, Integer) in module Base at combinatorics.jl:89 overwritten in module Combinatorics at /home/john/.julia/v0.4/Combinatorics/src/permutations.jl:157.
WARNING: Method definition partitions(Integer) in module Base at combinatorics.jl:252 overwritten in module Combinatorics at /home/john/.julia/v0.4/Combinatorics/src/partitions.jl:26.
WARNING: Method definition partitions(Integer, Integer) in module Base at combinatorics.jl:318 overwritten in module Combinatorics at /home/john/.julia/v0.4/Combinatorics/src/partitions.jl:93.
WARNING: Method definition partitions(AbstractArray{T<:Any, 1}) in module Base at combinatorics.jl:380 overwritten in module Combinatorics at /home/john/.julia/v0.4/Combinatorics/src/partitions.jl:158.
WARNING: Method definition partitions(AbstractArray{T<:Any, 1}, Int64) in module Base at combinatorics.jl:447 overwritten in module Combinatorics at /home/john/.julia/v0.4/Combinatorics/src/partitions.jl:228.
WARNING: Method definition combinations(Any, Integer) in module Base at combinatorics.jl:182 overwritten in module Combinatorics at /home/john/.julia/v0.4/Combinatorics/src/combinations.jl:42.
WARNING: Method definition nthperm!(AbstractArray{T<:Any, 1}, Integer) in module Base at combinatorics.jl:70 overwritten in module Combinatorics at /home/john/.julia/v0.4/Combinatorics/src/permutations.jl:136.
WARNING: Method definition permutations(Any) in module Base at combinatorics.jl:219 overwritten in module Combinatorics at /home/john/.julia/v0.4/Combinatorics/src/permutations.jl:24.
WARNING: Method definition factorial(#T<:Integer, #T<:Integer) in module Base at combinatorics.jl:56 overwritten in module Combinatorics at /home/john/.julia/v0.4/Combinatorics/src/factorials.jl:18.
WARNING: Method definition factorial(Integer, Integer) in module Base at combinatorics.jl:66 overwritten in module Combinatorics at /home/john/.julia/v0.4/Combinatorics/src/factorials.jl:28.
WARNING: Method definition parity(AbstractArray{#T<:Integer, 1}) in module Base at combinatorics.jl:642 overwritten in module Combinatorics at /home/john/.julia/v0.4/Combinatorics/src/permutations.jl:221.
WARNING: Method definition prevprod(Array{Int64, 1}, Any) in module Base at combinatorics.jl:565 overwritten in module Combinatorics at /home/john/.julia/v0.4/Combinatorics/src/partitions.jl:354.
sglyon commented 8 years ago

That'd be great! Let's be on the lookout for people to work on that.

That's really strange that you are seeing those warnings as Combinatorics.jl is a new julia 0.5 only package. It simply contains the combinatorics pieces that are in the julia 0.4 standard library that were split out into their own package. According to the REQUIRE file in Combinatorics.jl you shouldn't even be allowed to install it under and julia 0.4.x

oyamad commented 8 years ago

I got similar warnings on OS X:

julia> using QuantEcon

...

WARNING: Method definition factorial(#T<:Integer, #T<:Integer) in module Base at combinatorics.jl:56 overwritten in module Combinatorics at /Volumes/HD1/Users/oyama/.julia/v0.4/Combinatorics/src/factorials.jl:18.

...

WARNING: Method definition partitions(AbstractArray{T<:Any, 1}, Int64) in module Base at combinatorics.jl:447 overwritten in module Combinatorics at /Volumes/HD1/Users/oyama/.julia/v0.4/Combinatorics/src/partitions.jl:228.
julia> versioninfo()
Julia Version 0.4.5
Commit 2ac304d (2016-03-18 00:58 UTC)
Platform Info:
  System: Darwin (x86_64-apple-darwin13.4.0)
  CPU: Intel(R) Xeon(R) CPU           W3580  @ 3.33GHz
  WORD_SIZE: 64
  BLAS: libopenblas (USE64BITINT DYNAMIC_ARCH NO_AFFINITY Nehalem)
  LAPACK: libopenblas64_
  LIBM: libopenlibm
  LLVM: libLLVM-3.3
cc7768 commented 8 years ago

Are people still running into problems? I think @spencerlyon2 made the changes needed to remove errors and warnings.

jstac commented 8 years ago

I'm fine, although I used Pablo's fix earlier.

But in that case why don't we close and reopen if people report further problems?

jstac commented 8 years ago

Still having problems. Just did Pkg.update() and using QuantEcon produces

INFO: Precompiling module QuantEcon...
ERROR: LoadError: LoadError: HDF5 not properly installed. Please run Pkg.build("HDF5")
 in error at ./error.jl:21
 in include at ./boot.jl:261
 in include_from_node1 at ./loading.jl:320
 in include at ./boot.jl:261
 in include_from_node1 at ./loading.jl:320
 [inlined code] from none:2
 in anonymous at no file:0
 in process_options at ./client.jl:257
 in _start at ./client.jl:378
while loading /home/john/.julia/v0.4/HDF5/src/plain.jl, in expression starting on line 23
while loading /home/john/.julia/v0.4/HDF5/src/HDF5.jl, in expression starting on line 3
ERROR: LoadError: Failed to precompile HDF5 to /home/john/.julia/lib/v0.4/HDF5.ji
 in error at ./error.jl:21
 in compilecache at loading.jl:400
 in require at ./loading.jl:240
 in include at ./boot.jl:261
 in include_from_node1 at ./loading.jl:320
 [inlined code] from none:2
 in anonymous at no file:0
 in process_options at ./client.jl:257
 in _start at ./client.jl:378
while loading /home/john/.julia/v0.4/JLD/src/JLD.jl, in expression starting on line 8
ERROR: LoadError: LoadError: Failed to precompile JLD to /home/john/.julia/lib/v0.4/JLD.ji
 in error at ./error.jl:21
 in compilecache at loading.jl:400
 in require at ./loading.jl:240
 in include at ./boot.jl:261
 in include_from_node1 at ./loading.jl:320
 in include at ./boot.jl:261
 in include_from_node1 at ./loading.jl:320
 [inlined code] from none:2
 in anonymous at no file:0
 in process_options at ./client.jl:257
 in _start at ./client.jl:378
while loading /home/john/.julia/v0.4/LightGraphs/src/persistence/jld.jl, in expression starting on line 1
while loading /home/john/.julia/v0.4/LightGraphs/src/LightGraphs.jl, in expression starting on line 159
ERROR: LoadError: LoadError: Failed to precompile LightGraphs to /home/john/.julia/lib/v0.4/LightGraphs.ji
 in error at ./error.jl:21
 in compilecache at loading.jl:400
 in require at ./loading.jl:240
 in include at ./boot.jl:261
 in include_from_node1 at ./loading.jl:320
 in include at ./boot.jl:261
 in include_from_node1 at ./loading.jl:320
 [inlined code] from none:2
 in anonymous at no file:0
 in process_options at ./client.jl:257
 in _start at ./client.jl:378
while loading /home/john/.julia/v0.4/QuantEcon/src/markov/mc_tools.jl, in expression starting on line 13
while loading /home/john/.julia/v0.4/QuantEcon/src/QuantEcon.jl, in expression starting on line 120

Here's version info

Julia Version 0.4.5
Commit 2ac304d (2016-03-18 00:58 UTC)
Platform Info:
  System: Linux (x86_64-linux-gnu)
  CPU: Intel(R) Core(TM) i7-4700HQ CPU @ 2.40GHz
  WORD_SIZE: 64
  BLAS: libopenblas (NO_LAPACKE DYNAMIC_ARCH NO_AFFINITY Haswell)
  LAPACK: liblapack.so.3
  LIBM: libopenlibm
  LLVM: libLLVM-3.3
oyamad commented 8 years ago

Regarding the warnings on my OS X, using QuantEcon no longer generates them after Pkg.update(). (Combinatorics disappeared from Pkg.status().)