search

aws / aws-cli

Universal Command Line Interface for Amazon Web Services
Other
15.1k stars 4.01k forks source link

Segmentation Fault on ArmV8 with Crypto Extensions #8713

Closed kubrickfr closed 1 month ago

kubrickfr commented 1 month ago

Describe the bug

Uploading thousands of 512MiB files to S3 by piping them to the aws cli always eventually crashes.

Expected Behavior

Uploads should complete, or, if something bad happens, return an error code and not crash.

Current Behavior

If doing it thousands of time (which is costing me a fortune btw), aws cli crashes with segmentation fault, sometimes printing: An error occurred (BadDigest) when calling the UploadPart operation: The Content-MD5 you specified did not match what we received. but not always.

Here's a backtrace:

#0  0x0000ffff82e7bce8 in aes_v8_ctr32_encrypt_blocks () from /usr/local/aws-cli/v2/2.15.58/dist/libcrypto.so.1.1
#1  0x0000ffff82f62e24 in CRYPTO_gcm128_encrypt_ctr32 () from /usr/local/aws-cli/v2/2.15.58/dist/libcrypto.so.1.1
#2  0x0000ffff82f3ff58 in aes_gcm_cipher () from /usr/local/aws-cli/v2/2.15.58/dist/libcrypto.so.1.1
#3  0x0000ffff82f4a92c in evp_EncryptDecryptUpdate () from /usr/local/aws-cli/v2/2.15.58/dist/libcrypto.so.1.1
#4  0x0000ffff82d28c24 in tls13_enc () from /usr/local/aws-cli/v2/2.15.58/dist/libssl.so.1.1
#5  0x0000ffff82d23410 in do_ssl3_write.localalias () from /usr/local/aws-cli/v2/2.15.58/dist/libssl.so.1.1
#6  0x0000ffff82d23da0 in ssl3_write_bytes () from /usr/local/aws-cli/v2/2.15.58/dist/libssl.so.1.1
#7  0x0000ffff82d37218 in ssl_write_internal () from /usr/local/aws-cli/v2/2.15.58/dist/libssl.so.1.1
#8  0x0000ffff82d3735c in SSL_write_ex () from /usr/local/aws-cli/v2/2.15.58/dist/libssl.so.1.1
#9  0x0000ffff82dc76d8 in _ssl__SSLSocket_write_impl (b=0xffff7d60d5c0, b=0xffff7d60d5c0, self=0xffff7f5ff060) at /tmp/Python-3.11.8/Modules/_ssl.c:2391
#10 _ssl__SSLSocket_write (self=0xffff7f5ff060, arg=<optimized out>) at /tmp/Python-3.11.8/Modules/clinic/_ssl.c.h:237
#11 0x0000ffff840fc068 in _PyEval_EvalFrameDefault (tstate=0xffff6c0012a0, frame=0xffff84995558, throwflag=192) at Python/ceval.c:5221
#12 0x0000ffff8424b1cc in _PyEval_EvalFrame (throwflag=0, frame=0xffff84995070, tstate=0xffff6c0012a0) at ./Include/internal/pycore_ceval.h:73
#13 _PyEval_Vector (tstate=0xffff6c0012a0, func=<optimized out>, locals=<optimized out>, args=<optimized out>, argcount=<optimized out>, kwnames=<optimized out>) at Python/ceval.c:6434
#14 0x0000ffff84157018 in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=6, args=0xffff7f79e280, callable=0xffff8315d1c0, tstate=0xffff6c0012a0) at ./Include/internal/pycore_call.h:92
#15 method_vectorcall (method=<optimized out>, args=0xffff7f7687d8, nargsf=<optimized out>, kwnames=0x0) at Objects/classobject.c:89
#16 0x0000ffff8415385c in _PyVectorcall_Call (kwargs=0xffff7f76d9c0, tuple=0xffff7f7687c0, callable=0xffff7f7c0a00, func=0xffff84156f60 <method_vectorcall>, tstate=0xffff6c0012a0) at Objects/call.c:245
#17 _PyObject_Call (tstate=0xffff6c0012a0, callable=0xffff7f7c0a00, args=0xffff7f7687c0, kwargs=0xffff7f76d9c0) at Objects/call.c:328
#18 0x0000ffff840fbdd0 in do_call_core (use_tracing=<optimized out>, kwdict=<optimized out>, callargs=<optimized out>, func=0xffff84995050, tstate=<optimized out>) at Python/ceval.c:7349
#19 _PyEval_EvalFrameDefault (tstate=0xffff6c0012a0, frame=0xffff84994fb0, throwflag=192) at Python/ceval.c:5376
#20 0x0000ffff8424b1cc in _PyEval_EvalFrame (throwflag=0, frame=0xffff84994e58, tstate=0xffff6c0012a0) at ./Include/internal/pycore_ceval.h:73
#21 _PyEval_Vector (tstate=0xffff6c0012a0, func=<optimized out>, locals=<optimized out>, args=<optimized out>, argcount=<optimized out>, kwnames=<optimized out>) at Python/ceval.c:6434
#22 0x0000ffff84157098 in _PyObject_VectorcallTstate (kwnames=0xffff7f7a2000, nargsf=3, args=0xffff7f79e250, callable=0xffff82c04180, tstate=0xffff6c0012a0) at ./Include/internal/pycore_call.h:92
#23 method_vectorcall (method=<optimized out>, args=0xffff7f79e258, nargsf=<optimized out>, kwnames=0xffff7f7a2000) at Objects/classobject.c:59
#24 0x0000ffff841538cc in _PyVectorcall_Call (kwargs=<optimized out>, tuple=<optimized out>, callable=0xffff7f7c0280, func=0xffff84156f60 <method_vectorcall>, tstate=0xffff6c0012a0) at Objects/call.c:257
#25 _PyObject_Call (tstate=0xffff6c0012a0, callable=0xffff7f7c0280, args=<optimized out>, kwargs=<optimized out>) at Objects/call.c:328
#26 0x0000ffff840fbdd0 in do_call_core (use_tracing=<optimized out>, kwdict=<optimized out>, callargs=<optimized out>, func=0xffff84994e10, tstate=<optimized out>) at Python/ceval.c:7349
#27 _PyEval_EvalFrameDefault (tstate=0xffff6c0012a0, frame=0xffff84994d50, throwflag=192) at Python/ceval.c:5376
#28 0x0000ffff8424b1cc in _PyEval_EvalFrame (throwflag=0, frame=0xffff849944e8, tstate=0xffff6c0012a0) at ./Include/internal/pycore_ceval.h:73
#29 _PyEval_Vector (tstate=0xffff6c0012a0, func=<optimized out>, locals=<optimized out>, args=<optimized out>, argcount=<optimized out>, kwnames=<optimized out>) at Python/ceval.c:6434
#30 0x0000ffff84157098 in _PyObject_VectorcallTstate (kwnames=0xffff7f769210, nargsf=1, args=0xffff7f79dd40, callable=0xffff7f8f8220, tstate=0xffff6c0012a0) at ./Include/internal/pycore_call.h:92
#31 method_vectorcall (method=<optimized out>, args=0xffff7f79dd48, nargsf=<optimized out>, kwnames=0xffff7f769210) at Objects/classobject.c:59
#32 0x0000ffff841538cc in _PyVectorcall_Call (kwargs=<optimized out>, tuple=<optimized out>, callable=0xffff7f7b9680, func=0xffff84156f60 <method_vectorcall>, tstate=0xffff6c0012a0) at Objects/call.c:257
#33 _PyObject_Call (tstate=0xffff6c0012a0, callable=0xffff7f7b9680, args=<optimized out>, kwargs=<optimized out>) at Objects/call.c:328
#34 0x0000ffff840fbdd0 in do_call_core (use_tracing=<optimized out>, kwdict=<optimized out>, callargs=<optimized out>, func=0xffff849944b0, tstate=<optimized out>) at Python/ceval.c:7349
#35 _PyEval_EvalFrameDefault (tstate=0xffff6c0012a0, frame=0xffff849943e0, throwflag=192) at Python/ceval.c:5376
#36 0x0000ffff8424b1cc in _PyEval_EvalFrame (throwflag=0, frame=0xffff849943e0, tstate=0xffff6c0012a0) at ./Include/internal/pycore_ceval.h:73
#37 _PyEval_Vector (tstate=0xffff6c0012a0, func=<optimized out>, locals=<optimized out>, args=<optimized out>, argcount=<optimized out>, kwnames=<optimized out>) at Python/ceval.c:6434
#38 0x0000ffff84157098 in _PyObject_VectorcallTstate (kwnames=0xffff831e0d00, nargsf=1, args=0xffff7f7bbb70, callable=0xffff80c8d940, tstate=0xffff6c0012a0) at ./Include/internal/pycore_call.h:92
#39 method_vectorcall (method=<optimized out>, args=0xffff7f7bbb78, nargsf=<optimized out>, kwnames=0xffff831e0d00) at Objects/classobject.c:59
#40 0x0000ffff841538cc in _PyVectorcall_Call (kwargs=<optimized out>, tuple=<optimized out>, callable=0xffff7f7b8a40, func=0xffff84156f60 <method_vectorcall>, tstate=0xffff6c0012a0) at Objects/call.c:257
#41 _PyObject_Call (tstate=0xffff6c0012a0, callable=0xffff7f7b8a40, args=<optimized out>, kwargs=<optimized out>) at Objects/call.c:328
#42 0x0000ffff840fbdd0 in do_call_core (use_tracing=<optimized out>, kwdict=<optimized out>, callargs=<optimized out>, func=0xffff849943c0, tstate=<optimized out>) at Python/ceval.c:7349
#43 _PyEval_EvalFrameDefault (tstate=0xffff6c0012a0, frame=0xffff84994340, throwflag=192) at Python/ceval.c:5376
#44 0x0000ffff8424b1cc in _PyEval_EvalFrame (throwflag=0, frame=0xffff849942b0, tstate=0xffff6c0012a0) at ./Include/internal/pycore_ceval.h:73
#45 _PyEval_Vector (tstate=0xffff6c0012a0, func=<optimized out>, locals=<optimized out>, args=<optimized out>, argcount=<optimized out>, kwnames=<optimized out>) at Python/ceval.c:6434
#46 0x0000ffff84153c8c in _PyObject_FastCallDictTstate (tstate=tstate@entry=0xffff6c0012a0, callable=callable@entry=0xffff80c45440, args=args@entry=0xffff7d60e318, nargsf=nargsf@entry=1, kwargs=kwargs@entry=0xffff7f79b180) at Objects/call.c:141
#47 0x0000ffff84153f70 in _PyObject_Call_Prepend (tstate=tstate@entry=0xffff6c0012a0, callable=callable@entry=0xffff80c45440, obj=obj@entry=0xffff7f78e190, args=args@entry=0xffff8455b720 <_PyRuntime+58920>, kwargs=kwargs@entry=0xffff7f79b180) at Objects/call.c:482
#48 0x0000ffff841c6c38 in slot_tp_call (self=0xffff7f78e190, args=0xffff8455b720 <_PyRuntime+58920>, kwds=0xffff7f79b180) at Objects/typeobject.c:7623
#49 0x0000ffff841537f8 in _PyObject_Call (kwargs=0xffff7f79b180, args=0xffff8455b720 <_PyRuntime+58920>, callable=0xffff7f78e190, tstate=0xffff6c0012a0) at Objects/call.c:343
#50 _PyObject_Call (tstate=0xffff6c0012a0, callable=0xffff7f78e190, args=0xffff8455b720 <_PyRuntime+58920>, kwargs=0xffff7f79b180) at Objects/call.c:313
#51 0x0000ffff840fbdd0 in do_call_core (use_tracing=<optimized out>, kwdict=<optimized out>, callargs=<optimized out>, func=0xffff84994298, tstate=<optimized out>) at Python/ceval.c:7349
#52 _PyEval_EvalFrameDefault (tstate=0xffff6c0012a0, frame=0xffff84994228, throwflag=192) at Python/ceval.c:5376
#53 0x0000ffff8424b1cc in _PyEval_EvalFrame (throwflag=0, frame=0xffff84994188, tstate=0xffff6c0012a0) at ./Include/internal/pycore_ceval.h:73
#54 _PyEval_Vector (tstate=0xffff6c0012a0, func=<optimized out>, locals=<optimized out>, args=<optimized out>, argcount=<optimized out>, kwnames=<optimized out>) at Python/ceval.c:6434
#55 0x0000ffff8415385c in _PyVectorcall_Call (kwargs=0xffff7f73a080, tuple=0xffff808442c0, callable=0xffff81634180, func=0xffff841539e8 <_PyFunction_Vectorcall>, tstate=0xffff6c0012a0) at Objects/call.c:245
#56 _PyObject_Call (tstate=0xffff6c0012a0, callable=0xffff81634180, args=0xffff808442c0, kwargs=0xffff7f73a080) at Objects/call.c:328
#57 0x0000ffff840fbdd0 in do_call_core (use_tracing=<optimized out>, kwdict=<optimized out>, callargs=<optimized out>, func=0xffff84994170, tstate=<optimized out>) at Python/ceval.c:7349
#58 _PyEval_EvalFrameDefault (tstate=0xffff6c0012a0, frame=0xffff84994110, throwflag=192) at Python/ceval.c:5376
#59 0x0000ffff8424b1cc in _PyEval_EvalFrame (throwflag=0, frame=0xffff84994020, tstate=0xffff6c0012a0) at ./Include/internal/pycore_ceval.h:73
#60 _PyEval_Vector (tstate=0xffff6c0012a0, func=<optimized out>, locals=<optimized out>, args=<optimized out>, argcount=<optimized out>, kwnames=<optimized out>) at Python/ceval.c:6434
#61 0x0000ffff84157140 in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=1, args=0xffff7d60e820, callable=0xffff839bce00, tstate=0xffff6c0012a0) at ./Include/internal/pycore_call.h:92
#62 method_vectorcall (method=<optimized out>, args=0xffff8455b738 <_PyRuntime+58944>, nargsf=<optimized out>, kwnames=0x0) at Objects/classobject.c:67
#63 0x0000ffff8415385c in _PyVectorcall_Call (kwargs=0x0, tuple=0xffff8455b720 <_PyRuntime+58920>, callable=0xffff7f75c840, func=0xffff84156f60 <method_vectorcall>, tstate=0xffff6c0012a0) at Objects/call.c:245
#64 _PyObject_Call (tstate=0xffff6c0012a0, callable=0xffff7f75c840, args=0xffff8455b720 <_PyRuntime+58920>, kwargs=0x0) at Objects/call.c:328
#65 0x0000ffff84305d5c in thread_run (boot_raw=0xffff6c001270) at ./Modules/_threadmodule.c:1124
#66 0x0000ffff8429f53c in pythread_wrapper (arg=<optimized out>) at Python/thread_pthread.h:241
#67 0x0000ffff847f597c in start_thread (arg=0xffff7ea0f8e0) at ./nptl/pthread_create.c:447
#68 0x0000ffff8485ba4c in thread_start () at ../sysdeps/unix/sysv/linux/aarch64/clone3.S:76

Sadly, I do not have the debug symbols, if you do, I can send you the core dump in private.

Reproduction Steps

Inside for loop: <something producing 512MiB of output> | aws s3 cp - s3://bucket/xx/xx/xx/$FILE --storage-class STANDARD

(also tried with GLACIER)

Possible Solution

No response

Additional Information/Context

No response

CLI version used

2.15.58

Environment details (OS name and version, etc.)

Ubuntu 22.04 on raspberry pi 5

kubrickfr commented 1 month ago

After having encountered other issues related to encryption on the same device, I do not believe it to be a aws cli issue.

github-actions[bot] commented 1 month ago

This issue is now closed. Comments on closed issues are hard for our team to see. If you need more assistance, please open a new issue that references this one.