Sync failure starting March 10, 2021

[skyzh]

└ @ StorageMirrorServer ~/.julia/packages/StorageMirrorServer/XCiPW/src/mirror_tarball.jl:108
ERROR: LoadError: stop mirroring.
Stacktrace:
 [1] error(::String) at ./error.jl:33
 [2] mirror_tarball(::StorageMirrorServer.RegistryMeta, ::Array{String,1}, ::String; http_parameters::Dict{Symbol,Any}, packages::Nothing, registry_hash::Nothing, skip_duration::Int64, show_progress::Bool) at /root/.julia/packages/StorageMirrorServer/XCiPW/src/mirror_tarball.jl:109
 [3] #mirror_tarball#56 at /root/.julia/packages/StorageMirrorServer/XCiPW/src/mirror_tarball.jl:26 [inlined]
 [4] top-level scope at /worker-script/zhiyuan/worker-script/julia.jl:55
 [5] include(::Function, ::Module, ::String) at ./Base.jl:380
 [6] include(::Module, ::String) at ./Base.jl:368
 [7] exec_options(::Base.JLOptions) at ./client.jl:296
 [8] _start() at ./client.jl:506
in expression starting at /worker-script/zhiyuan/worker-script/julia.jl:54

Not sure what caused the failure.

[skyzh]

We're using julia -e 'using Pkg; pkg"add StorageMirrorServer@v0.2.1"'

[johnnychen94]

This seems caused by the fact that the upstream storage server stopped providing /registry/$uuid/$hash files.

# this works by switching to https://us-east.pkg.julialang.org
bash-3.2$ server=https://us-east.storage.julialang.org

bash-3.2$ wget $server$(curl -sSfL $server/registries)
--2021-03-11 23:38:28--  https://us-east.storage.julialang.org/registry/23338594-aafe-5451-b93e-139f81909106/5bc87d3fd5facd9877ad0adfd9feba7c8caaee9b
Resolving us-east.storage.julialang.org (us-east.storage.julialang.org)... 198.18.7.136
Connecting to us-east.storage.julialang.org (us-east.storage.julialang.org)|198.18.7.136|:443... connected.
HTTP request sent, awaiting response... 301 Moved Permanently
Location: https://us-east.storage.juliahub.com/registry/23338594-aafe-5451-b93e-139f81909106/5bc87d3fd5facd9877ad0adfd9feba7c8caaee9b [following]
--2021-03-11 23:38:29--  https://us-east.storage.juliahub.com/registry/23338594-aafe-5451-b93e-139f81909106/5bc87d3fd5facd9877ad0adfd9feba7c8caaee9b
Resolving us-east.storage.juliahub.com (us-east.storage.juliahub.com)... 198.18.7.133
Connecting to us-east.storage.juliahub.com (us-east.storage.juliahub.com)|198.18.7.133|:443... connected.
HTTP request sent, awaiting response... 500 Internal Server Error
2021-03-11 23:38:30 ERROR 500: Internal Server Error.

---

It seems that pkg servers still provide them, so using https://us-east.pkg.julialang.org as the upstream might fix the issue. More specifically, changing upstreams in https://github.com/johnnychen94/StorageMirrorServer.jl/blob/e2d2d491fa3d7a468a8a447257e414948a845f90/examples/gen_static_full.example.jl#L23-L27 to

upstreams = [
    "https://us-east.pkg.julialang.org",
    "https://kr.pkg.julialang.org"
]

should work right now but I'm not sure if this solution will work in the future. Need confirmation from Julialang(@staticfloat @StefanKarpinski). I didn't notice any Pkg changes wrt this, is this a servers issue?

cc: USTC (@taoky) TUNA(@jiegec @z4yx)

[johnnychen94]

It seems to be a server end issue that failed to handle Content-Encoding: identity header correctly. Thanks for the quick fix @staticfloat !

For mirror sites (users of this package), nothing needs to be changed.