search

JuliaClimate / CDSAPI.jl

Julia API to the Climate Data Store (a.k.a. CDS)
https://cds.climate.copernicus.eu
MIT License
23 stars 6 forks source link

Replace base download() with Http.download() #29

Closed LakshyaKhatri closed 4 years ago

LakshyaKhatri commented 4 years ago

Fixes #23 With this PR we also get a better presentation of the download process:

┌ Info: Downloading
│   source = "http://136.156.133.39/cache-compute-0012/cache/data3/adaptor.mars.internal-1594123368.684603-13971-30-3ad460cb-9d27-4d94-8240-92d7f1a8e239.grib"
│   dest = "test/data/era5.grib"
│   progress = 0.1094
│   time_taken = "1.0 s"
│   time_remaining = "8.15 s"
│   average_speed = "221.627 KiB/s"
│   downloaded = "221.849 KiB"
│   remaining = "1.764 MiB"
└   total = "1.980 MiB"
┌ Info: Downloading
│   source = "http://136.156.133.39/cache-compute-0012/cache/data3/adaptor.mars.internal-1594123368.684603-13971-30-3ad460cb-9d27-4d94-8240-92d7f1a8e239.grib"
│   dest = "test/data/era5.grib"
│   progress = 0.346
│   time_taken = "2.0 s"
│   time_remaining = "3.78 s"
│   average_speed = "350.492 KiB/s"
│   downloaded = "701.685 KiB"
│   remaining = "1.295 MiB"
└   total = "1.980 MiB"
┌ Info: Downloading
│   source = "http://136.156.133.39/cache-compute-0012/cache/data3/adaptor.mars.internal-1594123368.684603-13971-30-3ad460cb-9d27-4d94-8240-92d7f1a8e239.grib"
│   dest = "test/data/era5.grib"
│   progress = 0.4937
│   time_taken = "3.38 s"
│   time_remaining = "3.46 s"
│   average_speed = "296.577 KiB/s"
│   downloaded = "1001.243 KiB"
│   remaining = "1.003 MiB"
└   total = "1.980 MiB"
┌ Info: Downloading
│   source = "http://136.156.133.39/cache-compute-0012/cache/data3/adaptor.mars.internal-1594123368.684603-13971-30-3ad460cb-9d27-4d94-8240-92d7f1a8e239.grib"
│   dest = "test/data/era5.grib"
│   progress = 0.8179
│   time_taken = "4.38 s"
│   time_remaining = "0.97 s"
│   average_speed = "378.946 KiB/s"
│   downloaded = "1.620 MiB"
│   remaining = "369.284 KiB"
└   total = "1.980 MiB"
┌ Info: Downloading
│   source = "http://136.156.133.39/cache-compute-0012/cache/data3/adaptor.mars.internal-1594123368.684603-13971-30-3ad460cb-9d27-4d94-8240-92d7f1a8e239.grib"
│   dest = "test/data/era5.grib"
│   progress = 1.0
│   time_taken = "5.25 s"
│   time_remaining = "0.0 s"
│   average_speed = "386.567 KiB/s"
│   downloaded = "1.980 MiB"
│   remaining = "0 bytes"
└   total = "1.980 MiB"

But if the directory is not present, we will get a nasty error (this could be much simpler, I guess):

ERROR: SystemError: opening file "yoman/ewq.zip": No such file or directory
Stacktrace:
 [1] systemerror(::String, ::Int32; extrainfo::Nothing) at ./error.jl:168
 [2] #systemerror#50 at ./error.jl:167 [inlined]
 [3] systemerror at ./error.jl:167 [inlined]
 [4] open(::String; read::Nothing, write::Nothing, create::Nothing, truncate::Bool, append::Nothing) at ./iostream.jl:254
 [5] open(::String, ::String) at ./iostream.jl:310
 [6] open(::HTTP.var"#21#28"{HTTP.Streams.Stream{HTTP.Messages.Response,HTTP.ConnectionPool.Transaction{Sockets.TCPSocket}},HTTP.var"#report_callback#27"{Float64,Dates.DateTime,String,HTTP.var"#format_progress#22",HTTP.var"#format_bytes#23",HTTP.var"#format_seconds#24",HTTP.var"#format_bytes_per_second#25"{HTTP.var"#format_bytes#23"}},Int64}, ::String, ::Vararg{String,N} where N; kwargs::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}) at ./io.jl:296
 [7] open(::Function, ::String, ::String) at ./io.jl:296
 [8] (::HTTP.var"#20#26"{Int64,String,String,HTTP.var"#format_progress#22",HTTP.var"#format_bytes#23",HTTP.var"#format_seconds#24",HTTP.var"#format_bytes_per_second#25"{HTTP.var"#format_bytes#23"}})(::HTTP.Streams.Stream{HTTP.Messages.Response,HTTP.ConnectionPool.Transaction{Sockets.TCPSocket}}) at /home/aries/.julia/packages/HTTP/BOJmV/src/download.jl:132
 [9] macro expansion at /home/aries/.julia/packages/HTTP/BOJmV/src/StreamRequest.jl:69 [inlined]
 [10] macro expansion at ./task.jl:334 [inlined]
 [11] request(::Type{HTTP.StreamRequest.StreamLayer{Union{}}}, ::HTTP.ConnectionPool.Transaction{Sockets.TCPSocket}, ::HTTP.Messages.Request, ::Nothing; response_stream::Nothing, iofunction::HTTP.var"#20#26"{Int64,String,String,HTTP.var"#format_progress#22",HTTP.var"#format_bytes#23",HTTP.var"#format_seconds#24",HTTP.var"#format_bytes_per_second#25"{HTTP.var"#format_bytes#23"}}, verbose::Int64, kw::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}) at /home/aries/.julia/packages/HTTP/BOJmV/src/StreamRequest.jl:56
 [12] request(::Type{HTTP.ConnectionRequest.ConnectionPoolLayer{HTTP.StreamRequest.StreamLayer{Union{}}}}, ::HTTP.URIs.URI, ::HTTP.Messages.Request, ::Nothing; proxy::Nothing, socket_type::Type{T} where T, reuse_limit::Int64, kw::Base.Iterators.Pairs{Symbol,HTTP.var"#20#26"{Int64,String,String,HTTP.var"#format_progress#22",HTTP.var"#format_bytes#23",HTTP.var"#format_seconds#24",HTTP.var"#format_bytes_per_second#25"{HTTP.var"#format_bytes#23"}},Tuple{Symbol},NamedTuple{(:iofunction,),Tuple{HTTP.var"#20#26"{Int64,String,String,HTTP.var"#format_progress#22",HTTP.var"#format_bytes#23",HTTP.var"#format_seconds#24",HTTP.var"#format_bytes_per_second#25"{HTTP.var"#format_bytes#23"}}}}}) at /home/aries/.julia/packages/HTTP/BOJmV/src/ConnectionRequest.jl:89
 [13] request(::Type{HTTP.ExceptionRequest.ExceptionLayer{HTTP.ConnectionRequest.ConnectionPoolLayer{HTTP.StreamRequest.StreamLayer{Union{}}}}}, ::HTTP.URIs.URI, ::Vararg{Any,N} where N; kw::Base.Iterators.Pairs{Symbol,HTTP.var"#20#26"{Int64,String,String,HTTP.var"#format_progress#22",HTTP.var"#format_bytes#23",HTTP.var"#format_seconds#24",HTTP.var"#format_bytes_per_second#25"{HTTP.var"#format_bytes#23"}},Tuple{Symbol},NamedTuple{(:iofunction,),Tuple{HTTP.var"#20#26"{Int64,String,String,HTTP.var"#format_progress#22",HTTP.var"#format_bytes#23",HTTP.var"#format_seconds#24",HTTP.var"#format_bytes_per_second#25"{HTTP.var"#format_bytes#23"}}}}}) at /home/aries/.julia/packages/HTTP/BOJmV/src/ExceptionRequest.jl:19
 [14] (::Base.var"#58#60"{Base.var"#58#59#61"{ExponentialBackOff,HTTP.RetryRequest.var"#2#3"{Bool,HTTP.Messages.Request},typeof(HTTP.request)}})(::Type{T} where T, ::Vararg{Any,N} where N; kwargs::Base.Iterators.Pairs{Symbol,HTTP.var"#20#26"{Int64,String,String,HTTP.var"#format_progress#22",HTTP.var"#format_bytes#23",HTTP.var"#format_seconds#24",HTTP.var"#format_bytes_per_second#25"{HTTP.var"#format_bytes#23"}},Tuple{Symbol},NamedTuple{(:iofunction,),Tuple{HTTP.var"#20#26"{Int64,String,String,HTTP.var"#format_progress#22",HTTP.var"#format_bytes#23",HTTP.var"#format_seconds#24",HTTP.var"#format_bytes_per_second#25"{HTTP.var"#format_bytes#23"}}}}}) at ./error.jl:288
 [15] #request#1 at /home/aries/.julia/packages/HTTP/BOJmV/src/RetryRequest.jl:44 [inlined]
 [16] request(::Type{HTTP.MessageRequest.MessageLayer{HTTP.RetryRequest.RetryLayer{HTTP.ExceptionRequest.ExceptionLayer{HTTP.ConnectionRequest.ConnectionPoolLayer{HTTP.StreamRequest.StreamLayer{Union{}}}}}}}, ::String, ::HTTP.URIs.URI, ::Array{Pair{SubString{String},SubString{String}},1}, ::Nothing; http_version::VersionNumber, target::String, parent::Nothing, iofunction::Function, kw::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}) at /home/aries/.julia/packages/HTTP/BOJmV/src/MessageRequest.jl:51
 [17] request(::Type{HTTP.BasicAuthRequest.BasicAuthLayer{HTTP.MessageRequest.MessageLayer{HTTP.RetryRequest.RetryLayer{HTTP.ExceptionRequest.ExceptionLayer{HTTP.ConnectionRequest.ConnectionPoolLayer{HTTP.StreamRequest.StreamLayer{Union{}}}}}}}}, ::String, ::HTTP.URIs.URI, ::Array{Pair{SubString{String},SubString{String}},1}, ::Nothing; kw::Base.Iterators.Pairs{Symbol,HTTP.var"#20#26"{Int64,String,String,HTTP.var"#format_progress#22",HTTP.var"#format_bytes#23",HTTP.var"#format_seconds#24",HTTP.var"#format_bytes_per_second#25"{HTTP.var"#format_bytes#23"}},Tuple{Symbol},NamedTuple{(:iofunction,),Tuple{HTTP.var"#20#26"{Int64,String,String,HTTP.var"#format_progress#22",HTTP.var"#format_bytes#23",HTTP.var"#format_seconds#24",HTTP.var"#format_bytes_per_second#25"{HTTP.var"#format_bytes#23"}}}}}) at /home/aries/.julia/packages/HTTP/BOJmV/src/BasicAuthRequest.jl:28
 [18] request(::Type{HTTP.RedirectRequest.RedirectLayer{HTTP.BasicAuthRequest.BasicAuthLayer{HTTP.MessageRequest.MessageLayer{HTTP.RetryRequest.RetryLayer{HTTP.ExceptionRequest.ExceptionLayer{HTTP.ConnectionRequest.ConnectionPoolLayer{HTTP.StreamRequest.StreamLayer{Union{}}}}}}}}}, ::String, ::HTTP.URIs.URI, ::Array{Pair{SubString{String},SubString{String}},1}, ::Nothing; redirect_limit::Int64, forwardheaders::Bool, kw::Base.Iterators.Pairs{Symbol,HTTP.var"#20#26"{Int64,String,String,HTTP.var"#format_progress#22",HTTP.var"#format_bytes#23",HTTP.var"#format_seconds#24",HTTP.var"#format_bytes_per_second#25"{HTTP.var"#format_bytes#23"}},Tuple{Symbol},NamedTuple{(:iofunction,),Tuple{HTTP.var"#20#26"{Int64,String,String,HTTP.var"#format_progress#22",HTTP.var"#format_bytes#23",HTTP.var"#format_seconds#24",HTTP.var"#format_bytes_per_second#25"{HTTP.var"#format_bytes#23"}}}}}) at /home/aries/.julia/packages/HTTP/BOJmV/src/RedirectRequest.jl:24
 [19] request(::String, ::String, ::Array{Pair{SubString{String},SubString{String}},1}, ::Nothing; headers::Array{Pair{SubString{String},SubString{String}},1}, body::Nothing, query::Nothing, kw::Base.Iterators.Pairs{Symbol,HTTP.var"#20#26"{Int64,String,String,HTTP.var"#format_progress#22",HTTP.var"#format_bytes#23",HTTP.var"#format_seconds#24",HTTP.var"#format_bytes_per_second#25"{HTTP.var"#format_bytes#23"}},Tuple{Symbol},NamedTuple{(:iofunction,),Tuple{HTTP.var"#20#26"{Int64,String,String,HTTP.var"#format_progress#22",HTTP.var"#format_bytes#23",HTTP.var"#format_seconds#24",HTTP.var"#format_bytes_per_second#25"{HTTP.var"#format_bytes#23"}}}}}) at /home/aries/.julia/packages/HTTP/BOJmV/src/HTTP.jl:314
 [20] #open#6 at /home/aries/.julia/packages/HTTP/BOJmV/src/HTTP.jl:348 [inlined]
 [21] open at /home/aries/.julia/packages/HTTP/BOJmV/src/HTTP.jl:348 [inlined]
 [22] #download#19 at /home/aries/.julia/packages/HTTP/BOJmV/src/download.jl:101 [inlined]
 [23] download at /home/aries/.julia/packages/HTTP/BOJmV/src/download.jl:93 [inlined] (repeats 2 times)
 [24] retrieve(::String, ::Dict{String,Any}, ::String) at /home/aries/.julia/dev/CDSAPI/src/CDSAPI.jl:42
 [25] top-level scope at REPL[28]:1
 [26] eval(::Module, ::Any) at ./boot.jl:331
 [27] eval_user_input(::Any, ::REPL.REPLBackend) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.4/REPL/src/REPL.jl:86
 [28] run_backend(::REPL.REPLBackend) at /home/aries/.julia/packages/Revise/BqeJF/src/Revise.jl:1184
 [29] top-level scope at REPL[6]:0

Let me know what you people think will be right :thinking: :smile:

michiboo commented 4 years ago

@LakshyaKhatri I think you should first check if the directory is available if not then it will automatically create such directory

LakshyaKhatri commented 4 years ago

We can create new directories if they are not present. The only doubt is "what do the standards suggest? (should we create new directories or throw an error when the directory is not present) "

juliohm commented 4 years ago

Should we just let the user deal with directory creation? I am guessing that for most use cases the user will just download the data in a preexisting directory (project dir), but feel free to create a directory recursively if you prefer. I am merging this one right away. :100: