base64encode segfault on 1.8.2 aarch64

[IanButterworth]

This is happening on Julia 1.8.2 on an aarch64 machine

signal (11): Segmentation fault
in expression starting at /opt/actions-runner/_work/{private package}/run/run.jl:2
close at /cache/build/default-armageddon-0/julialang/julia-release-1-dot-8/usr/share/julia/stdlib/v1.8/Base64/src/encode.jl:115
#base64encode#5 at /cache/build/default-armageddon-0/julialang/julia-release-1-dot-8/usr/share/julia/stdlib/v1.8/Base64/src/encode.jl:213
base64encode##kw at /cache/build/default-armageddon-0/julialang/julia-release-1-dot-8/usr/share/julia/stdlib/v1.8/Base64/src/encode.jl:205 [inlined]
#base64encode#6 at /cache/build/default-armageddon-0/julialang/julia-release-1-dot-8/usr/share/julia/stdlib/v1.8/Base64/src/encode.jl:216 [inlined]
base64encode at /cache/build/default-armageddon-0/julialang/julia-release-1-dot-8/usr/share/julia/stdlib/v1.8/Base64/src/encode.jl:216 [inlined]
sign_aws4! at /home/ubuntu/.julia/packages/AWS/Rj4FY/src/utilities/sign.jl:72
#sign!#50 at /home/ubuntu/.julia/packages/AWS/Rj4FY/src/utilities/sign.jl:5
sign! at /home/ubuntu/.julia/packages/AWS/Rj4FY/src/utilities/sign.jl:1 [inlined]
#33 at /home/ubuntu/.julia/packages/AWS/Rj4FY/src/utilities/request.jl:127
#34 at /home/ubuntu/.julia/packages/AWS/Rj4FY/src/utilities/request.jl:144
#88#89 at ./error.jl:296
#88 at ./error.jl:291
unknown function (ip: 0xfffddfafa0f7)

The segfault is reproducible in some private code that takes ~40 mins to get to this point.

This is a direct copy of the input and call that segfaults, but I can't get it to error in isolation

julia> using Base64

julia> content = UInt8[0x56, 0x94, 0xd0, 0x82, 0x60, 0xc3, 0x68, 0xe3, 0xac, 0x4a, 0x97, 0xc5, 0x24, 0xdb, 0xbf, 0x3e];

julia> base64encode(content)
"VpTQgmDDaOOsSpfFJNu/Pg=="

The code this happens in uses CUDA & plotting libraries, so rr cannot be used without that being stripped out

The segfault is happening at https://github.com/JuliaLang/julia/blob/625aed1c2b6d59e4a6de2b4db2991a76597469b0/stdlib/Base64/src/encode.jl#L115-L119

[PallHaraldsson]

Is the code multi-threaded? I think that could be the issue, I don't see anything wrong there, but likely at write (which is likely inlined):

https://github.com/JuliaLang/julia/blob/625aed1c2b6d59e4a6de2b4db2991a76597469b0/stdlib/Base64/src/encode.jl#L103-L106

[IanButterworth]

There is threading although this call is happening on thread 1. Are IOBuffers not threadsafe?

[vtjnash]

Unless explicitly specified, any mutating function is not threadsafe. IOBuffer is explicitly MT-unsafe

[IanButterworth]

I guess I mean more whether creating and writing to an IOBuffer can cause a segfault due to something happening on another thread, say if multiple IOBuffers are created and written to at the same time on different threads.

Here the IOBuffer isn't exposed to other threads.

I'm working to strip out all the CUDA and Plotting libraries so that an rr trace can be collected.

[PallHaraldsson]

Here the IOBuffer isn't exposed to other threads.

Then i believe threading is not the issue. And many shouldn't be either if only one thread accesses each.

[IanButterworth]

Another segfault after some minor code changes in the private package.

signal (11): Segmentation fault
in expression starting at /opt/actions-runner/_work/{private package}/run/run.jl:2
getalladdrinfo at /cache/build/default-armageddon-0/julialang/julia-release-1-dot-8/usr/share/julia/stdlib/v1.8/Sockets/src/addrinfo.jl:70
getalladdrinfo at /cache/build/default-armageddon-0/julialang/julia-release-1-dot-8/usr/share/julia/stdlib/v1.8/Sockets/src/addrinfo.jl:122 [inlined]
#getconnection#7 at /home/ubuntu/.julia/packages/HTTP/fthyG/src/ConnectionPool.jl:403
getconnection##kw at /home/ubuntu/.julia/packages/HTTP/fthyG/src/ConnectionPool.jl:393 [inlined]
#getconnection#14 at /home/ubuntu/.julia/packages/HTTP/fthyG/src/ConnectionPool.jl:457
getconnection##kw at /home/ubuntu/.julia/packages/HTTP/fthyG/src/ConnectionPool.jl:450 [inlined]
#5 at /home/ubuntu/.julia/packages/HTTP/fthyG/src/ConnectionPool.jl:361 [inlined]
acquire at /home/ubuntu/.julia/packages/HTTP/fthyG/src/connectionpools.jl:116
#acquire#3 at /home/ubuntu/.julia/packages/HTTP/fthyG/src/connectionpools.jl:249 [inlined]
acquire##kw at /home/ubuntu/.julia/packages/HTTP/fthyG/src/connectionpools.jl:245 [inlined]
#newconnection#4 at /home/ubuntu/.julia/packages/HTTP/fthyG/src/ConnectionPool.jl:356
newconnection##kw at /home/ubuntu/.julia/packages/HTTP/fthyG/src/ConnectionPool.jl:349
unknown function (ip: 0xfffdc9d7b4cb)
_jl_invoke at /cache/build/default-armageddon-0/julialang/julia-release-1-dot-8/src/gf.c:2367 [inlined]
ijl_apply_generic at /cache/build/default-armageddon-0/julialang/julia-release-1-dot-8/src/gf.c:2549
#1#2 at /home/ubuntu/.julia/packages/HTTP/fthyG/src/clientlayers/ConnectionRequest.jl:78
unknown function (ip: 0xfffdc9d6901b)
_jl_invoke at /cache/build/default-armageddon-0/julialang/julia-release-1-dot-8/src/gf.c:2367 [inlined]
ijl_apply_generic at /cache/build/default-armageddon-0/julialang/julia-release-1-dot-8/src/gf.c:2549
1#7##kw at /home/ubuntu/.julia/packages/HTTP/fthyG/src/clientlayers/ConnectionRequest.jl:58
#1#2 at /home/ubuntu/.julia/packages/HTTP/fthyG/src/clientlayers/CanonicalizeRequest.jl:17
1#3##kw at /home/ubuntu/.julia/packages/HTTP/fthyG/src/clientlayers/CanonicalizeRequest.jl:13 [inlined]
#1#2 at /home/ubuntu/.julia/packages/HTTP/fthyG/src/clientlayers/RetryRequest.jl:25
1#4##kw at /home/ubuntu/.julia/packages/HTTP/fthyG/src/clientlayers/RetryRequest.jl:22 [inlined]
#1#2 at /home/ubuntu/.julia/packages/HTTP/fthyG/src/clientlayers/CookieRequest.jl:42
1#5##kw at /home/ubuntu/.julia/packages/HTTP/fthyG/src/clientlayers/CookieRequest.jl:19 [inlined]
#1#2 at /home/ubuntu/.julia/packages/HTTP/fthyG/src/clientlayers/ContentTypeRequest.jl:23
1#3##kw at /home/ubuntu/.julia/packages/HTTP/fthyG/src/clientlayers/ContentTypeRequest.jl:14 [inlined]
#1#2 at /home/ubuntu/.julia/packages/HTTP/fthyG/src/clientlayers/BasicAuthRequest.jl:22
1#3##kw at /home/ubuntu/.julia/packages/HTTP/fthyG/src/clientlayers/BasicAuthRequest.jl:14 [inlined]
#1#2 at /home/ubuntu/.julia/packages/HTTP/fthyG/src/clientlayers/DefaultHeadersRequest.jl:47
1#3##kw at /home/ubuntu/.julia/packages/HTTP/fthyG/src/clientlayers/DefaultHeadersRequest.jl:13 [inlined]
#1#2 at /home/ubuntu/.julia/packages/HTTP/fthyG/src/clientlayers/RedirectRequest.jl:17
1#4##kw at /home/ubuntu/.julia/packages/HTTP/fthyG/src/clientlayers/RedirectRequest.jl:14 [inlined]
#1#2 at /home/ubuntu/.julia/packages/HTTP/fthyG/src/clientlayers/DebugRequest.jl:23 [inlined]
1#4##kw at /home/ubuntu/.julia/packages/HTTP/fthyG/src/clientlayers/DebugRequest.jl:16 [inlined]
#1#2 at /home/ubuntu/.julia/packages/HTTP/fthyG/src/clientlayers/MessageRequest.jl:17
1#3##kw at /home/ubuntu/.julia/packages/HTTP/fthyG/src/clientlayers/MessageRequest.jl:15 [inlined]
#request#33 at /home/ubuntu/.julia/packages/HTTP/fthyG/src/HTTP.jl:417
unknown function (ip: 0xfffdc9d671fb)
_jl_invoke at /cache/build/default-armageddon-0/julialang/julia-release-1-dot-8/src/gf.c:2367 [inlined]
ijl_apply_generic at /cache/build/default-armageddon-0/julialang/julia-release-1-dot-8/src/gf.c:2549
request##kw at /home/ubuntu/.julia/packages/HTTP/fthyG/src/HTTP.jl:415
#request#16 at /home/ubuntu/.julia/packages/HTTP/fthyG/src/HTTP.jl:277 [inlined]
request##kw at /home/ubuntu/.julia/packages/HTTP/fthyG/src/HTTP.jl:275
unknown function (ip: 0xfffe241b0b07)
_jl_invoke at /cache/build/default-armageddon-0/julialang/julia-release-1-dot-8/src/gf.c:2367 [inlined]
ijl_apply_generic at /cache/build/default-armageddon-0/julialang/julia-release-1-dot-8/src/gf.c:2549
macro expansion at /home/ubuntu/.julia/packages/Mocking/GhDrf/src/mock.jl:29 [inlined]
#40 at /home/ubuntu/.julia/packages/AWS/Rj4FY/src/utilities/request.jl:220
#88#89 at ./error.jl:296
#88 at ./error.jl:291
unknown function (ip: 0xfffe241af4a7)
_jl_invoke at /cache/build/default-armageddon-0/julialang/julia-release-1-dot-8/src/gf.c:2367 [inlined]
ijl_apply_generic at /cache/build/default-armageddon-0/julialang/julia-release-1-dot-8/src/gf.c:2549
_http_request at /home/ubuntu/.julia/packages/AWS/Rj4FY/src/utilities/request.jl:245
macro expansion at /home/ubuntu/.julia/packages/Mocking/GhDrf/src/mock.jl:29 [inlined]
#33 at /home/ubuntu/.julia/packages/AWS/Rj4FY/src/utilities/request.jl:129
#34 at /home/ubuntu/.julia/packages/AWS/Rj4FY/src/utilities/request.jl:144
#88#89 at ./error.jl:296
#88 at ./error.jl:291
unknown function (ip: 0xfffe241a9f07)
_jl_invoke at /cache/build/default-armageddon-0/julialang/julia-release-1-dot-8/src/gf.c:2367 [inlined]
ijl_apply_generic at /cache/build/default-armageddon-0/julialang/julia-release-1-dot-8/src/gf.c:2549
#submit_request#32 at /home/ubuntu/.julia/packages/AWS/Rj4FY/src/utilities/request.jl:195
submit_request##kw at /home/ubuntu/.julia/packages/AWS/Rj4FY/src/utilities/request.jl:102
_jl_invoke at /cache/build/default-armageddon-0/julialang/julia-release-1-dot-8/src/gf.c:2367 [inlined]
ijl_apply_generic at /cache/build/default-armageddon-0/julialang/julia-release-1-dot-8/src/gf.c:2549
#_#79 at /home/ubuntu/.julia/packages/AWS/Rj4FY/src/AWS.jl:428
Any##kw at /home/ubuntu/.julia/packages/AWS/Rj4FY/src/AWS.jl:399 [inlined]
#execute_statement#10 at /home/ubuntu/.julia/packages/AWS/Rj4FY/src/services/rds_data.jl:314
execute_statement##kw at /home/ubuntu/.julia/packages/AWS/Rj4FY/src/services/rds_data.jl:307 [inlined]

[IanButterworth]

linux aarch64 has been pretty robust for us on 1.7. Were there big changes in that regard on 1.8?

[gbaraldi]

It had a REPL test failure that jameson fixed, but outside of it seems to pass CI pretty regularly. I don't know how much it's tested outside of CI however.

[giordano]

@IanButterworth any chance you can test again on master?