help with CMIP6 / Google Cloud binder

[rabernat]

I am trying to prepare a binder to showcase reading CMIP6 data with Zarr.jl from Google Cloud. Here is the repo:

https://github.com/pangeo-data/pangeo-julia-examples/

I'm trying to just reproduce what is shown here: https://meggart.github.io/Zarr.jl/latest/s3examples/#Accessing-CMIP6-data-on-GCS-1

Here is the binder link: https://mybinder.org/v2/gh/pangeo-data/pangeo-julia-examples/master?filepath=CMIP6-GCS-Julia.ipynb

However, my kernel just hangs every time it comes to actually read data from GCS. I end up having to restart the kernel. So nothing really works.

My Julia skills are very limited, but this is very important to me. I really have no idea how to debug. I'd appreciate any advice you have about how to get this binder working. PR welcome!

[visr]

Cool! I tried running the binder quickly and see the same thing. A more minimal reproducer is using AWSCore; aws_config(). However trying this locally also gives problems:

julia> using AWSCore
[ Info: Precompiling AWSCore [4f1ea46c-232b-54a6-9b17-cc2d0f3e6598]

julia> aws_config()
ERROR: IOError(Base.IOError("connect: network is unreachable (ENETUNREACH)", -4062) during request(http://169.254.169.254/latest/meta-data/iam/info))

Stacktrace:
 [1] wait_connected(::Sockets.TCPSocket) at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.3\Sockets\src\Sockets.jl:520
 [2] connect at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.3\Sockets\src\Sockets.jl:555 [inlined]
 [3] #getconnection#14(::Bool, ::Int64, ::Base.Iterators.Pairs{Symbol,Any,Tuple{Symbol,Symbol,Symbol},NamedTuple{(:require_ssl_verification, :iofunction, :verbose),Tuple{Bool,Nothing,Int64}}}, ::typeof(HTTP.ConnectionPool.getconnection), ::Type{Sockets.TCPSocket}, ::SubString{String}, ::SubString{String}) at C:\Users\visser_mn\.julia\packages\HTTP\lZVI1\src\ConnectionPool.jl:568
 [4] #getconnection#11(::Int64, ::Int64, ::Int64, ::Int64, ::Bool, ::Base.Iterators.Pairs{Symbol,Union{Nothing, Int64},Tuple{Symbol,Symbol},NamedTuple{(:iofunction, :verbose),Tuple{Nothing,Int64}}}, ::typeof(HTTP.ConnectionPool.getconnection), ::Type{HTTP.ConnectionPool.Transaction{Sockets.TCPSocket}}, ::SubString{String}, ::SubString{String}) at .\none:0
 [5] (::HTTP.ConnectionPool.var"#kw##getconnection")(::NamedTuple{(:reuse_limit, :iofunction, :verbose, :require_ssl_verification),Tuple{Int64,Nothing,Int64,Bool}}, ::typeof(HTTP.ConnectionPool.getconnection), ::Type{HTTP.ConnectionPool.Transaction{Sockets.TCPSocket}}, ::SubString{String}, ::SubString{String}) at .\none:0
 [6] #request#1(::Nothing, ::Type, ::Int64, ::Base.Iterators.Pairs{Symbol,Any,Tuple{Symbol,Symbol,Symbol},NamedTuple{(:iofunction, :verbose, :require_ssl_verification),Tuple{Nothing,Int64,Bool}}}, ::typeof(HTTP.request), ::Type{HTTP.ConnectionRequest.ConnectionPoolLayer{HTTP.StreamRequest.StreamLayer{Union{}}}}, ::HTTP.URIs.URI, ::HTTP.Messages.Request, ::Array{UInt8,1}) at C:\Users\visser_mn\.julia\packages\HTTP\lZVI1\src\ConnectionRequest.jl:41
 [7] (::HTTP.var"#kw##request")(::NamedTuple{(:iofunction, :verbose, :require_ssl_verification),Tuple{Nothing,Int64,Bool}}, ::typeof(HTTP.request), ::Type{HTTP.ConnectionRequest.ConnectionPoolLayer{HTTP.StreamRequest.StreamLayer{Union{}}}}, ::HTTP.URIs.URI, ::HTTP.Messages.Request, ::Array{UInt8,1}) at .\none:0
 [8] #request#1(::Base.Iterators.Pairs{Symbol,Any,Tuple{Symbol,Symbol,Symbol},NamedTuple{(:iofunction, :verbose, :require_ssl_verification),Tuple{Nothing,Int64,Bool}}}, ::typeof(HTTP.request), ::Type{HTTP.ExceptionRequest.ExceptionLayer{HTTP.ConnectionRequest.ConnectionPoolLayer{HTTP.StreamRequest.StreamLayer{Union{}}}}}, ::HTTP.URIs.URI, ::Vararg{Any,N} where N) at C:\Users\visser_mn\.julia\packages\HTTP\lZVI1\src\ExceptionRequest.jl:19
 [9] #request at .\none:0 [inlined]
 [10] #request#1(::VersionNumber, ::String, ::Nothing, ::Nothing, ::Base.Iterators.Pairs{Symbol,Integer,Tuple{Symbol,Symbol},NamedTuple{(:verbose, :require_ssl_verification),Tuple{Int64,Bool}}}, ::typeof(HTTP.request), ::Type{HTTP.MessageRequest.MessageLayer{HTTP.ExceptionRequest.ExceptionLayer{HTTP.ConnectionRequest.ConnectionPoolLayer{HTTP.StreamRequest.StreamLayer{Union{}}}}}}, ::String, ::HTTP.URIs.URI, ::Array{Pair{SubString{String},SubString{String}},1}, ::Array{UInt8,1}) at C:\Users\visser_mn\.julia\packages\HTTP\lZVI1\src\MessageRequest.jl:51
 [11] (::HTTP.var"#kw##request")(::NamedTuple{(:verbose, :require_ssl_verification),Tuple{Int64,Bool}}, ::typeof(HTTP.request), ::Type{HTTP.MessageRequest.MessageLayer{HTTP.ExceptionRequest.ExceptionLayer{HTTP.ConnectionRequest.ConnectionPoolLayer{HTTP.StreamRequest.StreamLayer{Union{}}}}}}, ::String, ::HTTP.URIs.URI, ::Array{Pair{SubString{String},SubString{String}},1}, ::Array{UInt8,1}) at .\none:0
 [12] http_request(::Dict{Symbol,Any}) at C:\Users\visser_mn\.julia\packages\AWSCore\nRYAp\src\http.jl:42
 [13] http_get(::String) at C:\Users\visser_mn\.julia\packages\AWSCore\nRYAp\src\http.jl:80
 [14] macro expansion at C:\Users\visser_mn\.julia\packages\Mocking\hFQQ4\src\mock.jl:29 [inlined]
 [15] _ec2_metadata(::String) at C:\Users\visser_mn\.julia\packages\AWSCore\nRYAp\src\AWSCredentials.jl:263
 [16] ec2_instance_credentials() at C:\Users\visser_mn\.julia\packages\AWSCore\nRYAp\src\AWSCredentials.jl:282
 [17] #AWSCredentials#3(::Nothing, ::Type{AWSCredentials}) at C:\Users\visser_mn\.julia\packages\AWSCore\nRYAp\src\AWSCredentials.jl:146
 [18] Type at .\none:0 [inlined]
 [19] aws_config() at C:\Users\visser_mn\.julia\packages\AWSCore\nRYAp\src\AWSCore.jl:115
 [20] top-level scope at REPL[3]:1

Although I have to say I haven't worked with AWSCore before, maybe @meggart knows more?

[meggart]

Yes I am having a look at it although I am not very familiar with binder yet. @visr I think this is unrelated, can you try aws_config(creds=nothing) ? Or do you have an AWS credential file somewhere on your local disc?

[meggart]

Found it, you were right, it happens here because it is evaluating all arguments before call something and the last call to aws_core is causing the timeout. I already have a fix that made the notebook work on binder. I think the reason it was working locally for me and @rabernat was that we have some AWS credential file which was found before trying to connect to some EC2 server.

[meggart]

So this should be fixed now from the Zarr.jl side, although one might think about making an issue in AWSCore, because the timeout when calling aws_config is quite annoying. @rabernat I just tried this on binder and I could read the data.

Only the plots did not work because PyPlot is not installed in the local conda environment. One fix would be to run

run(`conda install matplotlib`)

before loading the PyPlot package or to use a different plotting package like GR, VegaLite, PlotlyJS, ECharts etc. Also, the plotting packages do not recognize Dates from the CFTime package so they have to be converted to the Julia DateTime type. You can do this e.g. with

"Convert cftime datetime to stdlib datetime"
cftime2datetime(t) = DateTime(year(t), month(t), day(t), hour(t), minute(t), second(t))
newtime = cftime2datetime.(time)

However, once you confirm that the notebook is running for you, I can make a PR fixing the datetime issue or using another plotting package.

[rabernat]

However, once you confirm that the notebook is running for you, I can make a PR fixing the datetime issue or using another plotting package.

Fantastic! I just rebuilt the binder and yes, the GCS calls do work now. Thanks so much!

one might think about making an issue in AWSCore, because the timeout when calling aws_config is quite annoying

I agree. You would be in a better position to open that issue than I am, but I'd be happy to 👍 it.

to use a different plotting package like GR, VegaLite, PlotlyJS, ECharts etc.

I would like to do whatever is the accepted best practice for Julia users. Happy to take your recommendation here.

the plotting packages do not recognize Dates from the CFTime package so they have to be converted to the Julia DateTime type. You can do this e.g. with

I actually opened an issue about this here: https://github.com/JuliaGeo/CFTime.jl/issues/4 Your fix sounds like an ideal workaround.

[rabernat]

FYI, I need to get this finished today, so I'm going to try to implement your suggestions myself.

[meggart]

Ok, I was about to do this and just realized you already beat me to it. Regarding multithreaded reading, I think I will try if a naive threaded loop over chunks in Zarr.jl would speed things up. Otherwise it might help to use:

import Blosc
Blosc.set_num_threads(some_number)

to speed up decompression, but I think in this case we are more or less limited by data transfer rate.

Otherwise I think this would be better implemented in downstream packages like ESDL.jl or if someone wants to couple this to Dagger.jl. However, this will definitely not be done by tomorrow :smiley:

[meggart]

Can we close this or should we leave this open for future CMIP6-related questions?