Problem with mapclimgrid()

ghost commented 5 years ago

CONTEXT : I'm running on Fedora 29, I use anaconda3 as my python distribution and Julia 1.1 binaries inside of the Atom IDE. My .nc test files are from CMIP5, tasmax variable. I also reproduced this error on Windows 10.

PROBLEM : Whenever I try to call the mapclimgrid() function, I get the following stacktrace :

ERROR: LoadError: ArgumentError: ref of NULL PyObject Stacktrace: [1] getindex(::PyCall.PyObject, ::String) at /home/feanor/.julia/packages/PyCall/0jMpb/src/PyCall.jl:284 [2] getindex at /home/feanor/.julia/packages/PyCall/0jMpb/src/PyCall.jl:294 [inlined] [3] #mapclimgrid#37(::String, ::Bool, ::Array{Any,1}, ::Array{Any,1}, ::Array{Any,1}, ::Array{Any,1}, ::Function) at /home/feanor/.julia/packages/ClimateTools/W7g0z/src/mapping.jl:184 [4] mapclimgrid() at /home/feanor/.julia/packages/ClimateTools/W7g0z/src/mapping.jl:130 [5] top-level scope at none:0

I'm new to Julia and I can't solve or isolate the error. Thanks for the help and the super cool package :)

Balinus commented 5 years ago

Thanks for the report and comments! :)

does ] test ClimateTools passes all tests? (alternatively using Pkg;Pkg.test("ClimateTools") )

Meanwhile, can you point me to which CMIP5 models/files you are using? If you can upload it to Dropbox or something else I could test directly with the same file.

edit - And what version of ClimateTools are you using? Latest is v0.11.1.

ghost commented 5 years ago

Very interesting, I was not aware that one could test package like that. There was indeed an error during the testing, associated with Basemap (see the error below.) I used exactly the same MICO5 files as is suggested in the documentation, so I don't think they are the cause here. I will provide them if solving the error at hand does not work. Something of interest here is that I had to solve exactly this error "KeyError('PROJ_LIB')" on my anaconda3 distribution of python to get basemap to work. I was able to modify the setup.py of basemap in such a way to properly point it toward the anaconda3 folder, then using pip to reinstall over my current version. I have no idea how to carry out something similar within Julia though.

   Testing ClimateTools
    Status `/tmp/tmptfljZV/Manifest.toml`
  [dce04be8] ArgCheck v1.0.0
  [13072b0f] AxisAlgorithms v0.3.0
  [39de3d68] AxisArrays v0.3.0
  [9e28174c] BinDeps v0.8.10
  [4f4ee721] ClimateTools v0.11.1+ #master (https://github.com/Balinus/ClimateTools.jl.git)
  [3da002f7] ColorTypes v0.7.5
  [5ae59095] Colors v0.9.5
  [34da2185] Compat v1.5.1
  [8f4d0f93] Conda v1.2.0
  [a9693cdc] CondaBinDeps v0.1.0
  [53c48c17] FixedPointNumbers v0.5.3
  [59287772] Formatting v0.3.5
  [cf35fbd7] GeoInterface v0.4.0
  [a98d9a8b] Interpolations v0.11.1
  [8197267c] IntervalSets v0.3.1
  [c8e1da08] IterTools v1.1.1
  [682c06a0] JSON v0.20.0
  [b964fa9f] LaTeXStrings v1.0.3
  [1914dd2f] MacroTools v0.4.4
  [e1d29d7a] Missings v0.4.0
  [85f8d34a] NCDatasets v0.6.0
  [77ba4419] NaNMath v0.3.2
  [30363a11] NetCDF v0.7.2
  [6fe1bfb0] OffsetArrays v0.10.0
  [f27b6e38] Polynomials v0.5.2
  [92933f4c] ProgressMeter v0.9.0
  [438e738f] PyCall v1.18.5
  [d330b81b] PyPlot v2.7.0
  [b3c3ace0] RangeArrays v0.3.1
  [c84ed2f1] Ratios v0.3.0
  [3cdcf5f2] RecipesBase v0.6.0
  [189a3867] Reexport v0.2.0
  [8e980c4a] Shapefile v0.5.0
  [90137ffa] StaticArrays v0.10.2
  [30578b45] URIParser v0.4.0
  [1986cc42] Unitful v0.14.0
  [81def892] VersionParsing v1.1.3
  [efce3f68] WoodburyMatrices v0.4.1
  [2a0f44e3] Base64  [`@stdlib/Base64`]
  [ade2ca70] Dates  [`@stdlib/Dates`]
  [8bb1440f] DelimitedFiles  [`@stdlib/DelimitedFiles`]
  [8ba89e20] Distributed  [`@stdlib/Distributed`]
  [b77e0a4c] InteractiveUtils  [`@stdlib/InteractiveUtils`]
  [76f85450] LibGit2  [`@stdlib/LibGit2`]
  [8f399da3] Libdl  [`@stdlib/Libdl`]
  [37e2e46d] LinearAlgebra  [`@stdlib/LinearAlgebra`]
  [56ddb016] Logging  [`@stdlib/Logging`]
  [d6f4376e] Markdown  [`@stdlib/Markdown`]
  [a63ad114] Mmap  [`@stdlib/Mmap`]
  [44cfe95a] Pkg  [`@stdlib/Pkg`]
  [de0858da] Printf  [`@stdlib/Printf`]
  [3fa0cd96] REPL  [`@stdlib/REPL`]
  [9a3f8284] Random  [`@stdlib/Random`]
  [ea8e919c] SHA  [`@stdlib/SHA`]
  [9e88b42a] Serialization  [`@stdlib/Serialization`]
  [1a1011a3] SharedArrays  [`@stdlib/SharedArrays`]
  [6462fe0b] Sockets  [`@stdlib/Sockets`]
  [2f01184e] SparseArrays  [`@stdlib/SparseArrays`]
  [10745b16] Statistics  [`@stdlib/Statistics`]
  [8dfed614] Test  [`@stdlib/Test`]
  [cf7118a7] UUIDs  [`@stdlib/UUIDs`]
  [4ec0a83e] Unicode  [`@stdlib/Unicode`]
[ Info: Installing mpl_toolkits.basemap via the Conda basemap package...
[ Info: Running `conda install -y basemap` in root environment
Collecting package metadata: done
Solving environment: done

# All requested packages already installed.

ERROR: LoadError: InitError: PyError (PyImport_ImportModule) <class 'KeyError'>
KeyError('PROJ_LIB')
  File "/home/feanor/.julia/conda/3/lib/python3.7/site-packages/mpl_toolkits/basemap/__init__.py", line 155, in <module>
    pyproj_datadir = os.environ['PROJ_LIB']
  File "/home/feanor/.julia/conda/3/lib/python3.7/os.py", line 678, in __getitem__
    raise KeyError(key) from None

Stacktrace:
 [1] pyimport(::String) at /home/feanor/.julia/packages/PyCall/0jMpb/src/PyCall.jl:486
 [2] pyimport_conda(::String, ::String, ::String) at /home/feanor/.julia/packages/PyCall/0jMpb/src/PyCall.jl:656
 [3] pyimport_conda at /home/feanor/.julia/packages/PyCall/0jMpb/src/PyCall.jl:649 [inlined]
 [4] __init__() at /home/feanor/.julia/packages/ClimateTools/W7g0z/src/ClimateTools.jl:51
 [5] _include_from_serialized(::String, ::Array{Any,1}) at ./loading.jl:633
 [6] _require_search_from_serialized(::Base.PkgId, ::String) at ./loading.jl:713
 [7] _require(::Base.PkgId) at ./loading.jl:937
 [8] require(::Base.PkgId) at ./loading.jl:858
 [9] require(::Module, ::Symbol) at ./loading.jl:853
 [10] include at ./boot.jl:326 [inlined]
 [11] include_relative(::Module, ::String) at ./loading.jl:1038
 [12] include(::Module, ::String) at ./sysimg.jl:29
 [13] include(::String) at ./client.jl:403
 [14] top-level scope at none:0
during initialization of module ClimateTools
in expression starting at /home/feanor/.julia/packages/ClimateTools/W7g0z/test/runtests.jl:1
ERROR: Package ClimateTools errored during testing

Balinus commented 5 years ago

ok, I was expecting an error with Basemap (I encounter this kind of bahaviour each time I install ClimateTools on a new machine). This package is troublesome and ClimateTools needs to migrate towards Cartopy for mapping purpose (and for a more robust installation process!).

In your anaconda3 installation, you are able to load the following packages?

# python
import mpl_toolkits.basemap as basemap
import matplotlib.pyplot as plt
import cmocean as cm
import scipy as sc

The default python distribution in Julia is usually an "inside" python distribution. However, you can tell PyCall (the package responsible for calling python functions) which python distribution you want use. So, assuming that your own anaconda3 distribution is able to load Basemap, here's the steps to tell PyCall to use it :

# julia
using Pkg
ENV["PYTHON"] = "... path of the anaconda python program you want ..."
Pkg.build("PyCall")

Once this is done, close Julia and your workflow!

I'd be interested to know your workaround for basemap in anaconda3!

ghost commented 5 years ago

I confirm the python dependencies can be loaded on my machine. I was able to call my own python distro as you suggested and it seemed to help. Testing now progresses a bit further, but it hits an error because some number is indexed with a float...

   Testing ClimateTools
. . .
┌ Warning: Using Celsius can be problematic for arithmetic operations. Best practice is to keep Kelvin and only convert to Celsius at the end with the overloaded ClimateTools.uconvert function.
└ @ ClimateTools ~/.julia/packages/ClimateTools/W7g0z/src/extract.jl:243

(julia:32017): Gtk-WARNING **: 01:33:19.993: Impossible de trouver le moteur de thème dans module_path : « murrine »

(julia:32017): Gtk-WARNING **: 01:33:19.994: Impossible de trouver le moteur de thème dans module_path : « murrine »

(julia:32017): Gtk-WARNING **: 01:33:19.994: Impossible de trouver le moteur de thème dans module_path : « murrine »

(julia:32017): Gtk-WARNING **: 01:33:20.020: GModule (/usr/lib64/gtk-2.0/2.10.0/immodules/im-ibus.so) initialization check failed: GLib version too old (micro mismatch)

(julia:32017): Gtk-WARNING **: 01:33:20.020: Loading IM context type 'ibus' failed

(julia:32017): Gtk-WARNING **: 01:33:20.024: GModule (/usr/lib64/gtk-2.0/2.10.0/immodules/im-ibus.so) initialization check failed: GLib version too old (micro mismatch)

(julia:32017): Gtk-WARNING **: 01:33:20.024: Loading IM context type 'ibus' failed

(julia:32017): Gtk-WARNING **: 01:33:20.038: GModule (/usr/lib64/gtk-2.0/2.10.0/immodules/im-ibus.so) initialization check failed: GLib version too old (micro mismatch)

(julia:32017): Gtk-WARNING **: 01:33:20.038: Loading IM context type 'ibus' failed
/home/feanor/.local/lib/python3.6/site-packages/mpl_toolkits/basemap/__init__.py:3608: MatplotlibDeprecationWarning: The ishold function was deprecated in version 2.0.
  b = ax.ishold()
/home/feanor/.local/lib/python3.6/site-packages/mpl_toolkits/basemap/__init__.py:3675: MatplotlibDeprecationWarning: axes.hold is deprecated.
    See the API Changes document (http://matplotlib.org/api/api_changes.html)
    for more details.
  ax.hold(b)
Interface: Error During Test at /home/feanor/.julia/packages/ClimateTools/W7g0z/test/interface_test.jl:4
  Got exception outside of a @test
  PyError ($(Expr(:escape, :(ccall(#= /home/feanor/.julia/packages/PyCall/0jMpb/src/pyfncall.jl:44 =# @pysym(:PyObject_Call), PyPtr, (PyPtr, PyPtr, PyPtr), o, pyargsptr, kw))))) <class 'IndexError'>
  IndexError('only integers, slices (`:`), ellipsis (`...`), numpy.newaxis (`None`) and integer or boolean arrays are valid indices',)
    File "/home/feanor/.local/lib/python3.6/site-packages/mpl_toolkits/basemap/__init__.py", line 521, in with_transform
      return plotfunc(self,x,y,data,*args,**kwargs)
    File "/home/feanor/.local/lib/python3.6/site-packages/mpl_toolkits/basemap/__init__.py", line 3644, in contourf
      xx = x[x.shape[0]/2,:]

  Stacktrace:
   [1] pyerr_check at /home/feanor/.julia/packages/PyCall/0jMpb/src/exception.jl:60 [inlined]
   [2] pyerr_check at /home/feanor/.julia/packages/PyCall/0jMpb/src/exception.jl:64 [inlined]
   [3] macro expansion at /home/feanor/.julia/packages/PyCall/0jMpb/src/exception.jl:84 [inlined]
   [4] __pycall!(::PyCall.PyObject, ::Ptr{PyCall.PyObject_struct}, ::PyCall.PyObject, ::PyCall.PyObject) at /home/feanor/.julia/packages/PyCall/0jMpb/src/pyfncall.jl:44
   [5] _pycall!(::PyCall.PyObject, ::PyCall.PyObject, ::Tuple{Array{Float64,2},Array{Float64,2},Array{Float32,2},Int64}, ::Int64, ::PyCall.PyObject) at /home/feanor/.julia/packages/PyCall/0jMpb/src/pyfncall.jl:29
   [6] _pycall!(::PyCall.PyObject, ::PyCall.PyObject, ::Tuple{Array{Float64,2},Array{Float64,2},Array{Float32,2},Int64}, ::Base.Iterators.Pairs{Symbol,Any,Tuple{Symbol,Symbol,Symbol},NamedTuple{(:cmap, :vmin, :vmax),Tuple{ColorMap,Float32,Float32}}}) at /home/feanor/.julia/packages/PyCall/0jMpb/src/pyfncall.jl:11
   [7] #call#89(::Base.Iterators.Pairs{Symbol,Any,Tuple{Symbol,Symbol,Symbol},NamedTuple{(:cmap, :vmin, :vmax),Tuple{ColorMap,Float32,Float32}}}, ::PyCall.PyObject, ::Array{Float64,2}, ::Vararg{Any,N} where N) at /home/feanor/.julia/packages/PyCall/0jMpb/src/pyfncall.jl:89
   [8] (::getfield(PyCall, Symbol("#kw#PyObject")))(::NamedTuple{(:cmap, :vmin, :vmax),Tuple{ColorMap,Float32,Float32}}, ::PyCall.PyObject, ::Array{Float64,2}, ::Vararg{Any,N} where N) at ./none:0
   [9] #mapclimgrid#36(::String, ::Bool, ::Array{Any,1}, ::Int64, ::Array{Any,1}, ::Array{Any,1}, ::Tuple{Float64}, ::Tuple{Float64}, ::String, ::Symbol, ::String, ::Int64, ::Bool, ::String, ::String, ::typeof(mapclimgrid), ::ClimGrid{AxisArray{Float32,2,Array{Float32,2},Tuple{Axis{:lon,Array{Float64,1}},Axis{:lat,Array{Float64,1}}}}}) at /home/feanor/.julia/packages/ClimateTools/W7g0z/src/mapping.jl:95
   [10] mapclimgrid(::ClimGrid{AxisArray{Float32,2,Array{Float32,2},Tuple{Axis{:lon,Array{Float64,1}},Axis{:lat,Array{Float64,1}}}}}) at /home/feanor/.julia/packages/ClimateTools/W7g0z/src/mapping.jl:27
   [11] top-level scope at /home/feanor/.julia/packages/ClimateTools/W7g0z/test/interface_test.jl:21
   [12] top-level scope at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.1/Test/src/Test.jl:1083
   [13] top-level scope at /home/feanor/.julia/packages/ClimateTools/W7g0z/test/interface_test.jl:7
   [14] include at ./boot.jl:326 [inlined]
   [15] include_relative(::Module, ::String) at ./loading.jl:1038
   [16] include(::Module, ::String) at ./sysimg.jl:29
   [17] include(::String) at ./client.jl:403
   [18] top-level scope at none:0
   [19] include at ./boot.jl:326 [inlined]
   [20] include_relative(::Module, ::String) at ./loading.jl:1038
   [21] include(::Module, ::String) at ./sysimg.jl:29
   [22] include(::String) at ./client.jl:403
   [23] top-level scope at none:0
   [24] eval(::Module, ::Any) at ./boot.jl:328
   [25] exec_options(::Base.JLOptions) at ./client.jl:243
   [26] _start() at ./client.jl:436
Test Summary: | Pass  Error  Total
Interface     |    4      1      5
ERROR: LoadError: LoadError: Some tests did not pass: 4 passed, 0 failed, 1 errored, 0 broken.
in expression starting at /home/feanor/.julia/packages/ClimateTools/W7g0z/test/interface_test.jl:4
in expression starting at /home/feanor/.julia/packages/ClimateTools/W7g0z/test/runtests.jl:19
ERROR: Package ClimateTools errored during testing

I was able to make it progress significantly further by replacing said float with an Int (i.e. replacing /2 --> //2), but I find this very peculiar. I still get an error even with this "fix" when testing the package (see below), but the good news is that at this point, I can load a climgrid from my .nc files (yay!). Most functionnalities are broken though so I'm still looking for help.

   Testing ClimateTools
. . .
┌ Warning: Using Celsius can be problematic for arithmetic operations. Best practice is to keep Kelvin and only convert to Celsius at the end with the overloaded ClimateTools.uconvert function.
└ @ ClimateTools ~/.julia/packages/ClimateTools/W7g0z/src/extract.jl:243

(julia:32049): Gtk-WARNING **: 01:35:54.124: Impossible de trouver le moteur de thème dans module_path : « murrine »

(julia:32049): Gtk-WARNING **: 01:35:54.124: Impossible de trouver le moteur de thème dans module_path : « murrine »

(julia:32049): Gtk-WARNING **: 01:35:54.125: Impossible de trouver le moteur de thème dans module_path : « murrine »

(julia:32049): Gtk-WARNING **: 01:35:54.147: GModule (/usr/lib64/gtk-2.0/2.10.0/immodules/im-ibus.so) initialization check failed: GLib version too old (micro mismatch)

(julia:32049): Gtk-WARNING **: 01:35:54.147: Loading IM context type 'ibus' failed

(julia:32049): Gtk-WARNING **: 01:35:54.150: GModule (/usr/lib64/gtk-2.0/2.10.0/immodules/im-ibus.so) initialization check failed: GLib version too old (micro mismatch)

(julia:32049): Gtk-WARNING **: 01:35:54.150: Loading IM context type 'ibus' failed

(julia:32049): Gtk-WARNING **: 01:35:54.161: GModule (/usr/lib64/gtk-2.0/2.10.0/immodules/im-ibus.so) initialization check failed: GLib version too old (micro mismatch)

(julia:32049): Gtk-WARNING **: 01:35:54.161: Loading IM context type 'ibus' failed
/home/feanor/.local/lib/python3.6/site-packages/mpl_toolkits/basemap/__init__.py:3608: MatplotlibDeprecationWarning: The ishold function was deprecated in version 2.0.
  b = ax.ishold()
/home/feanor/.local/lib/python3.6/site-packages/mpl_toolkits/basemap/__init__.py:3677: MatplotlibDeprecationWarning: axes.hold is deprecated.
    See the API Changes document (http://matplotlib.org/api/api_changes.html)
    for more details.
  ax.hold(b)
Test Summary: | Pass  Broken  Total
Interface     |  165       1    166
Test Summary: | Pass  Total
Extraction    |   18     18
Test Summary: | Pass  Total
Exportation   |    1      1
Test Summary:   | Pass  Total
Bias correction |    4      4
Test Summary: | Pass  Total
Indices       |   26     26
Mapping: Error During Test at /home/feanor/.julia/packages/ClimateTools/W7g0z/test/mapping_test.jl:1
  Got exception outside of a @test
  PyError ($(Expr(:escape, :(ccall(#= /home/feanor/.julia/packages/PyCall/0jMpb/src/pyfncall.jl:44 =# @pysym(:PyObject_Call), PyPtr, (PyPtr, PyPtr, PyPtr), o, pyargsptr, kw))))) <class 'SystemError'>
  SystemError("<class 'RuntimeError'> returned a result with an error set",)
    File "/home/feanor/.local/lib/python3.6/site-packages/mpl_toolkits/basemap/__init__.py", line 1148, in __call__
      xout,yout = self.projtran(x,y,inverse=inverse)
    File "/home/feanor/.local/lib/python3.6/site-packages/mpl_toolkits/basemap/proj.py", line 286, in __call__
      outx,outy = self._proj4(x, y, inverse=inverse)
    File "/home/feanor/.local/lib/python3.6/site-packages/mpl_toolkits/basemap/pyproj.py", line 388, in __call__
      _proj.Proj._fwd(self, inx, iny, radians=radians, errcheck=errcheck)
    File "_proj.pyx", line 122, in _proj.Proj._fwd (src/_proj.c:1571)

  Stacktrace:
   [1] pyerr_check at /home/feanor/.julia/packages/PyCall/0jMpb/src/exception.jl:60 [inlined]
   [2] pyerr_check at /home/feanor/.julia/packages/PyCall/0jMpb/src/exception.jl:64 [inlined]
   [3] macro expansion at /home/feanor/.julia/packages/PyCall/0jMpb/src/exception.jl:84 [inlined]
   [4] __pycall!(::PyCall.PyObject, ::Ptr{PyCall.PyObject_struct}, ::PyCall.PyObject, ::Ptr{Nothing}) at /home/feanor/.julia/packages/PyCall/0jMpb/src/pyfncall.jl:44
   [5] _pycall!(::PyCall.PyObject, ::PyCall.PyObject, ::Tuple{Array{Float32,2},Array{Float32,2}}, ::Int64, ::Ptr{Nothing}) at /home/feanor/.julia/packages/PyCall/0jMpb/src/pyfncall.jl:29
   [6] _pycall!(::PyCall.PyObject, ::PyCall.PyObject, ::Tuple{Array{Float32,2},Array{Float32,2}}, ::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}) at /home/feanor/.julia/packages/PyCall/0jMpb/src/pyfncall.jl:11
   [7] #call#89(::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::PyCall.PyObject, ::Array{Float32,2}, ::Vararg{Array{Float32,2},N} where N) at /home/feanor/.julia/packages/PyCall/0jMpb/src/pyfncall.jl:89
   [8] (::PyCall.PyObject)(::Array{Float32,2}, ::Vararg{Array{Float32,2},N} where N) at /home/feanor/.julia/packages/PyCall/0jMpb/src/pyfncall.jl:89
   [9] #mapclimgrid#36(::String, ::Bool, ::Array{Any,1}, ::Int64, ::Array{Any,1}, ::Array{Any,1}, ::Tuple{Float64}, ::Tuple{Float64}, ::String, ::Symbol, ::String, ::Int64, ::Bool, ::String, ::String, ::typeof(mapclimgrid), ::ClimGrid{AxisArray{Unitful.Quantity{Float32,𝚯,Unitful.FreeUnits{(K,),𝚯,nothing}},3,Array{Unitful.Quantity{Float32,𝚯,Unitful.FreeUnits{(K,),𝚯,nothing}},3},Tuple{Axis{:lon,Array{Float32,1}},Axis{:lat,Array{Float32,1}},Axis{:time,Array{Union{Missing, DateTimeNoLeap},1}}}}}) at /home/feanor/.julia/packages/ClimateTools/W7g0z/src/mapping.jl:92
   [10] (::getfield(ClimateTools, Symbol("#kw##mapclimgrid")))(::NamedTuple{(:region,),Tuple{String}}, ::typeof(mapclimgrid), ::ClimGrid{AxisArray{Unitful.Quantity{Float32,𝚯,Unitful.FreeUnits{(K,),𝚯,nothing}},3,Array{Unitful.Quantity{Float32,𝚯,Unitful.FreeUnits{(K,),𝚯,nothing}},3},Tuple{Axis{:lon,Array{Float32,1}},Axis{:lat,Array{Float32,1}},Axis{:time,Array{Union{Missing, DateTimeNoLeap},1}}}}}) at ./none:0
   [11] top-level scope at /home/feanor/.julia/packages/ClimateTools/W7g0z/test/mapping_test.jl:38
   [12] top-level scope at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.1/Test/src/Test.jl:1083
   [13] top-level scope at /home/feanor/.julia/packages/ClimateTools/W7g0z/test/mapping_test.jl:4
   [14] include at ./boot.jl:326 [inlined]
   [15] include_relative(::Module, ::String) at ./loading.jl:1038
   [16] include(::Module, ::String) at ./sysimg.jl:29
   [17] include(::String) at ./client.jl:403
   [18] top-level scope at none:0
   [19] include at ./boot.jl:326 [inlined]
   [20] include_relative(::Module, ::String) at ./loading.jl:1038
   [21] include(::Module, ::String) at ./sysimg.jl:29
   [22] include(::String) at ./client.jl:403
   [23] top-level scope at none:0
   [24] eval(::Module, ::Any) at ./boot.jl:328
   [25] exec_options(::Base.JLOptions) at ./client.jl:243
   [26] _start() at ./client.jl:436
Test Summary: | Pass  Error  Total
Mapping       |    8      1      9
ERROR: LoadError: LoadError: Some tests did not pass: 8 passed, 0 failed, 1 errored, 0 broken.
in expression starting at /home/feanor/.julia/packages/ClimateTools/W7g0z/test/mapping_test.jl:1
in expression starting at /home/feanor/.julia/packages/ClimateTools/W7g0z/test/runtests.jl:23
ERROR: Package ClimateTools errored during testing

When this is all sorted all I'll post a walkthrough on how to get the package working (including the basemap "hack" :+1: )

Balinus commented 5 years ago

Do you have pyproj installed? I'm not sure if it's required.

Also, do you use PyCall for other tasks inside Julia? On my end, I only need Python calls for ClimateTools and to ensure that the right packages are installed I use a virtualenv for python. Here's what I'm doing (I guess I should suggest that approach in the documentation!).

Virtual Env

$ virtualenv --python=/usr/bin/python2 /path/to/venv
$ /path/to/venv/bin/python -m pip install numpy
$ /path/to/venv/bin/python -m pip install scipy
$ /path/to/venv/bin/python -m pip install matplotlib
$ /path/to/venv/bin/python -m pip install https://github.com/matplotlib/basemap/archive/v1.0.7rel.tar.gz
$ /path/to/venv/bin/python -m pip install git+https://github.com/matplotlib/cmocean

Testing Python installation

#bash 
$ /path/to/venv/bin/python # launch python
#python
>>> import mpl_toolkits.basemap as basemap
>>> import matplotlib.pyplot as plt
>>> import cmocean as cm
>>> import scipy as sc

Build PyCall with the new venv python

# julia
julia> ENV["PYTHON"] = "/path/to/venv/bin/python"
julia> using Pkg;Pkg.build("PyCall")
julia> exit
# re-enter julia
julia> using ClimateTools
julia> using Pkg; Pkg.test("ClimateTools")

Hopefully, this should work! i really need to get rid of basemap and use Cartopy. That's the main installation problem I have with ClimateTools.

ghost commented 5 years ago

I created a Venv like you suggested, and the test is successful now! Most functionalities seem to work as far as I'm concerned, although setting region = QuebecNSP returns an error. Thanks for helping me out, I'm looking forward to using ClimateTools.jl :+1:

Balinus commented 5 years ago

Nice! Bien content que cela te soit utile :)

what kind of error do you get when you use region="QuebecNSP"? On my end, I can produce the figure without an error (I do get a warning though).

file = "tasmax_day_CanESM2_rcp85_r1i1p1_20060101-21001231.nc"
"tasmax_day_CanESM2_rcp85_r1i1p1_20060101-21001231.nc"

julia> C = load(file, "tasmax", start_date=(2006, 01, 01), end_date=(2100,12, 31, 23))
ClimGrid struct with data:
   3-dimensional AxisArray{Unitful.Quantity{Float32,𝚯,Unitful.FreeUnits{(K,),𝚯,nothing}},3,...} with axes:
    :lon, [-180.0, -177.188, -174.375, -171.563, -168.75, -165.938, -163.125, -160.313, -157.5, -154.688  …  151.875, 154.688, 157.5, 160.313, 163.125, 165.938, 168.75, 171.563, 174.375, 177.188]
    :lat, [-87.8638, -85.0965, -82.3129, -79.5256, -76.7369, -73.9475, -71.1578, -68.3678, -65.5776, -62.7874  …  62.7874, 65.5776, 68.3678, 71.1578, 73.9475, 76.7369, 79.5256, 82.3129, 85.0965, 87.8638]
    :time, Union{Missing, DateTimeNoLeap}[DateTimeNoLeap(2006-01-01T12:00:00), DateTimeNoLeap(2006-01-02T12:00:00), DateTimeNoLeap(2006-01-03T12:00:00), DateTimeNoLeap(2006-01-04T12:00:00), DateTimeNoLeap(2006-01-05T12:00:00), DateTimeNoLeap(2006-01-06T12:00:00), DateTimeNoLeap(2006-01-07T12:00:00), DateTimeNoLeap(2006-01-08T12:00:00), DateTimeNoLeap(2006-01-09T12:00:00), DateTimeNoLeap(2006-01-10T12:00:00)  …  DateTimeNoLeap(2100-12-22T12:00:00), DateTimeNoLeap(2100-12-23T12:00:00), DateTimeNoLeap(2100-12-24T12:00:00), DateTimeNoLeap(2100-12-25T12:00:00), DateTimeNoLeap(2100-12-26T12:00:00), DateTimeNoLeap(2100-12-27T12:00:00), DateTimeNoLeap(2100-12-28T12:00:00), DateTimeNoLeap(2100-12-29T12:00:00), DateTimeNoLeap(2100-12-30T12:00:00), DateTimeNoLeap(2100-12-31T12:00:00)]
And data, a 128×64×34675 Array{Unitful.Quantity{Float32,𝚯,Unitful.FreeUnits{(K,),𝚯,nothing}},3}
Project: CMIP5
Institute: CCCma
Model: CanESM2
Experiment: rcp85
Run: r1i1p1
Variable: tasmax
Data units: K
Frequency: day
Global attributes: Dict{Any,Any} with 31 entries
Filename: tasmax_day_CanESM2_rcp85_r1i1p1_20060101-21001231.nc

julia> mapclimgrid(C, region="quebecnsp")
/home/proy/anaconda3/envs/pyjulia27/lib/python2.7/site-packages/mpl_toolkits/basemap/__init__.py:1623: MatplotlibDeprecationWarning: The get_axis_bgcolor function was deprecated in version 2.0. Use get_facecolor instead.
  fill_color = ax.get_axis_bgcolor()
Warning: Cannot label parallels on Near-Sided Perspective basemap'Warning: Cannot label meridians on full-disk
                Geostationary, Orthographic or Azimuthal equidistant basemap
/home/proy/anaconda3/envs/pyjulia27/lib/python2.7/site-packages/mpl_toolkits/basemap/__init__.py:3608: MatplotlibDeprecationWarning: The ishold function was deprecated in version 2.0.
  b = ax.ishold()
/home/proy/anaconda3/envs/pyjulia27/lib/python2.7/site-packages/mpl_toolkits/basemap/__init__.py:3677: MatplotlibDeprecationWarning: axes.hold is deprecated.
    See the API Changes document (http://matplotlib.org/api/api_changes.html)
    for more details.
  ax.hold(b)
(true, PyPlot.Figure(PyObject <matplotlib.figure.Figure object at 0x7ff62602f390>), PyObject <matplotlib.axes._subplots.AxesSubplot object at 0x7ff62606cb90>, PyObject <matplotlib.colorbar.Colorbar object at 0x7ff5d16f5210>)

Returns the following figure. quebecnsp

ghost commented 5 years ago

(Je garde la discussion en anglais pour des potentiels interessés à l'internationale, mais j'ai cru comprendre que tu es de l'UQàM? Bonjour de la part de l'UdeS!)

The function get_axis_bgcolor() is deprecated and will throw an error according to the forums, I'm not sure why you are not encountering this problem. In any case, it's very easy to solve, just replace all occurences in your basemap init file of get_axis_bgcolor() with ax.get_fc().

Now that everything is working and that I can reliably install it on other machines as well, I reckon this thread may be closed!

Balinus commented 5 years ago

Oui, j'ai étudié à l'UQAM et je travaille maintenant à Ouranos. Salutations!

So, I understand that you no longer have problems with region="Qeuebecnsp" once you edit the init file? Great!

I've updated the installation instruction. Do not hesitate to edit them if you see something! https://balinus.github.io/ClimateTools.jl/dev/installation/

If you develop functions or features using ClimateTools, I'll be more than happy to include them in the package.

JuliaClimate / ClimateTools.jl