search

JuliaCloud / AWS.jl

Julia interface to AWS
MIT License
160 stars 62 forks source link

Missing retry for "resolve host" error with Downloads backend? #541

Open ericphanson opened 2 years ago

ericphanson commented 2 years ago

Just got this error on two workers (different keys, same bucket) at about the same time:

ERROR: LoadError: Could not resolve host: s3.us-east-2.amazonaws.com while requesting https://s3.us-east-2.amazonaws.com/MY_BUCKET/MY_KEY
Stacktrace:
  [1] (::Downloads.var"#9#18"{Base.BufferStream, Base.DevNull, String, OrderedCollections.LittleDict{String, String, Vector{String}, Vector{String}}, Float64, Nothing, Bool, Bool, String, Int64, Bool, Bool})(easy::Downloads.Curl.Easy)
    @ Downloads /usr/local/julia/share/julia/stdlib/v1.7/Downloads/src/Downloads.jl:369
  [2] with_handle(f::Downloads.var"#9#18"{Base.BufferStream, Base.DevNull, String, OrderedCollections.LittleDict{String, String, Vector{String}, Vector{String}}, Float64, Nothing, Bool, Bool, String, Int64, Bool, Bool}, handle::Downloads.Curl.Easy)
    @ Downloads.Curl /usr/local/julia/share/julia/stdlib/v1.7/Downloads/src/Curl/Curl.jl:64
  [3] #8
    @ /usr/local/julia/share/julia/stdlib/v1.7/Downloads/src/Downloads.jl:311 [inlined]
  [4] arg_write(f::Downloads.var"#8#17"{Base.DevNull, String, OrderedCollections.LittleDict{String, String, Vector{String}, Vector{String}}, Float64, Nothing, Bool, Bool, String, Int64, Bool, Bool}, arg::Base.BufferStream)
    @ ArgTools /usr/local/julia/share/julia/stdlib/v1.7/ArgTools/src/ArgTools.jl:112
  [5] #7
    @ /usr/local/julia/share/julia/stdlib/v1.7/Downloads/src/Downloads.jl:310 [inlined]
  [6] arg_read
    @ /usr/local/julia/share/julia/stdlib/v1.7/ArgTools/src/ArgTools.jl:61 [inlined]
  [7] request(url::String; input::Nothing, output::Base.BufferStream, method::String, headers::OrderedCollections.LittleDict{String, String, Vector{String}, Vector{String}}, timeout::Float64, progress::Nothing, verbose::Bool, throw::Bool, downloader::Downloads.Downloader)
    @ Downloads /usr/local/julia/share/julia/stdlib/v1.7/Downloads/src/Downloads.jl:309
  [8] macro expansion
    @ ~/.julia/packages/Mocking/MsKoy/src/mock.jl:29 [inlined]
  [9] macro expansion
    @ ~/.julia/packages/AWS/5goc1/src/utilities/downloads_backend.jl:72 [inlined]
 [10] macro expansion
    @ ~/.julia/packages/Retry/vS1bg/src/repeat_try.jl:192 [inlined]
 [11] _http_request(backend::AWS.DownloadsBackend, request::AWS.Request, response_stream::IOBuffer)
...

This is with code that just executed several thousand times as part of the same job over the past few days without issue.

ericphanson commented 2 years ago

This should be fixed by https://github.com/JuliaCloud/AWS.jl/pull/543 because this is a RequestError (ref https://github.com/JuliaLang/Downloads.jl/pull/194#issue-1233090017) and we already have a retry; hopefully bumping the limit will help.