`ftplib.cpython-312.pyc` does not appear to be reproducible

[raboof]

Bug report

Bug description:

Previously, when buiding cpython 3.11 twice on independent infrastructure (for example when building the binary package for the NixOS linux distribution), with the right parameters this process produced a bit-by-bit deterministic result.

For cpython 3.12 (tested with various versions up to 3.12.6), the result is almost deterministic:

It seems specifically ftplib.cpython-312.pyc, ftplib.cpython-312.opt-1.pyc and ftplib.cpython-312.opt-2.pyc may differ between rebuilds. This is strange, as I don't see anything special about the contents of ftplib.py or about the way it is built. Looking at the bytecode with pydisasm from xdis only shows differences in 'code object' addresses, not in 'actual' bytecode.

Does anyone have any idea where this might be coming from?

CPython versions tested on:

3.12

Operating systems tested on:

Linux

[picnixz]

1. Is it also the case in 3.13?
2. Does it happen with other files or is it really only this specific one?
3. When you say "rebuilds" do you mean something that does configure + make + py_compile?

There was a reproducibility issue in 3.13 because of __static_attributes__. Maybe a similar issue happens in 3.12? (__static_attributes__ is 3.13+ only).

[raboof]

1. No, I haven't seen this with 3.13 - 3.13.0rc3 seems fully reproducible
2. Strangely it seems to be specifically this file (in both plain pyc, opt-1 and opt-2 variants)
3. I'm comparing nix packages built from CI with the same package built locally - indeed all of configure+make+py_compile are part of that

Indeed on 3.13.0rc2 I saw what was likely the __static_attributes__ thing (#124442) - interestingly I didn't see that affect any of the the pyc files, just frozen.o and _bootstrap_python.o, and thus the (static and shared) libraries.

[hacscred]

If you know the specific byte code, you can track that to specific source code and then remove lines by bisecting until you find an offending line. Presumably you can then instrument and program the debugger to stop when generating that code and that should be non-deterministic then. Probably such a tool would be of general interest if nobody already built one.

[raboof]

Like I wrote in the summary:

Looking at the bytecode with pydisasm from xdis only shows differences in 'code object' addresses, not in 'actual' bytecode.

[hacscred]

I didn't make a distinction between actual byte code and their addresses.

Either way, one way of finding the issue is my approach. It's ofcourse possible you have a better idea.

[raboof]

I didn't make a distinction between actual byte code and their addresses.

Gotcha - those addresses are different pretty much 'across the board' in this file though, there's not any particular offending lines in the source code. There's also not much to bisect, this problem appears to exist since early 3.12 rc's.

For example:

 #    5: 1
 #    6: 21
 #    7: 8192
-#    8: <code object Error at 0x7f96f0cbda30, file "/nix/store/h3i0acpmr8mrjx07519xxmidv8mpax
4y-python3-3.12.5/lib/python3.12/ftplib.py", line 57>
+#    8: <code object Error at 0x7f059eeb1a30, file "/nix/store/h3i0acpmr8mrjx07519xxmidv8mpax
4y-python3-3.12.5/lib/python3.12/ftplib.py", line 57>
 #    9: 'Error'
-#   10: <code object error_reply at 0x7f96f0a72ce0, file "/nix/store/h3i0acpmr8mrjx07519xxmid
v8mpax4y-python3-3.12.5/lib/python3.12/ftplib.py", line 58>
+#   10: <code object error_reply at 0x7f059ec4ace0, file "/nix/store/h3i0acpmr8mrjx07519xxmid
v8mpax4y-python3-3.12.5/lib/python3.12/ftplib.py", line 58>
 #   11: 'error_reply'
-#   12: <code object error_temp at 0x7f96f0a72e80, file "/nix/store/h3i0acpmr8mrjx07519xxmidv
8mpax4y-python3-3.12.5/lib/python3.12/ftplib.py", line 59>
+#   12: <code object error_temp at 0x7f059ec4ae80, file "/nix/store/h3i0acpmr8mrjx07519xxmidv
8mpax4y-python3-3.12.5/lib/python3.12/ftplib.py", line 59>
 #   13: 'error_temp'

It's clear that these correspond to the classes Error, error_reply etc around line 57 of ftplib.py, but otherwise it doesn't seem too helpful.

the full diff

``` --- one.dis 2024-10-08 14:03:45.043342741 +0200 +++ other.dis 2024-10-08 14:05:03.785590515 +0200 @@ -21,29 +21,29 @@ # 5: 1 # 6: 21 # 7: 8192 -# 8: +# 8: # 9: 'Error' -# 10: +# 10: # 11: 'error_reply' -# 12: +# 12: # 13: 'error_temp' -# 14: +# 14: # 15: 'error_perm' -# 16: +# 16: # 17: 'error_proto' # 18: '\r\n' # 19: b'\r\n' -# 20: +# 20: # 21: 'FTP' -# 22: +# 22: # 23: 'FTP_TLS' -# 24: -# 25: -# 26: -# 27: -# 28: -# 29: -# 30: +# 24: +# 25: +# 26: +# 27: +# 28: +# 29: +# 30: # 31: '__main__' # 32: ('', 'I') # Names: @@ -122,7 +122,7 @@ 57: 54 PUSH_NULL 56 LOAD_BUILD_CLASS - 58 LOAD_CONST () + 58 LOAD_CONST () 60 MAKE_FUNCTION (No arguments) 62 LOAD_CONST ("Error") 64 LOAD_NAME (Exception) @@ -131,7 +131,7 @@ 58: 76 PUSH_NULL 78 LOAD_BUILD_CLASS - 80 LOAD_CONST () + 80 LOAD_CONST () 82 MAKE_FUNCTION (No arguments) 84 LOAD_CONST ("error_reply") 86 LOAD_NAME (Error) @@ -140,7 +140,7 @@ 59: 98 PUSH_NULL 100 LOAD_BUILD_CLASS - 102 LOAD_CONST () + 102 LOAD_CONST () 104 MAKE_FUNCTION (No arguments) 106 LOAD_CONST ("error_temp") 108 LOAD_NAME (Error) @@ -149,7 +149,7 @@ 60: 120 PUSH_NULL 122 LOAD_BUILD_CLASS - 124 LOAD_CONST () + 124 LOAD_CONST () 126 MAKE_FUNCTION (No arguments) 128 LOAD_CONST ("error_perm") 130 LOAD_NAME (Error) @@ -158,7 +158,7 @@ 61: 142 PUSH_NULL 144 LOAD_BUILD_CLASS - 146 LOAD_CONST () + 146 LOAD_CONST () 148 MAKE_FUNCTION (No arguments) 150 LOAD_CONST ("error_proto") 152 LOAD_NAME (Error) @@ -179,7 +179,7 @@ 74: 182 PUSH_NULL 184 LOAD_BUILD_CLASS - 186 LOAD_CONST () + 186 LOAD_CONST () 188 MAKE_FUNCTION (No arguments) 190 LOAD_CONST ("FTP") 192 CALL 2 @@ -198,7 +198,7 @@ 677: 236 PUSH_NULL 238 LOAD_BUILD_CLASS - 240 LOAD_CONST () + 240 LOAD_CONST () 242 MAKE_FUNCTION (No arguments) 244 LOAD_CONST ("FTP_TLS") 246 LOAD_NAME (FTP) @@ -222,35 +222,35 @@ 790: 324 LOAD_CONST (None) 326 STORE_GLOBAL (_150_re) -792: 328 LOAD_CONST () +792: 328 LOAD_CONST () 330 MAKE_FUNCTION (No arguments) 332 STORE_NAME (parse150) 810: 334 LOAD_CONST (None) 336 STORE_GLOBAL (_227_re) -812: 338 LOAD_CONST () +812: 338 LOAD_CONST () 340 MAKE_FUNCTION (No arguments) 342 STORE_NAME (parse227) -831: 344 LOAD_CONST () +831: 344 LOAD_CONST () 346 MAKE_FUNCTION (No arguments) 348 STORE_NAME (parse229) -852: 350 LOAD_CONST () +852: 350 LOAD_CONST () 352 MAKE_FUNCTION (No arguments) 354 STORE_NAME (parse257) -874: 356 LOAD_CONST () +874: 356 LOAD_CONST () 358 MAKE_FUNCTION (No arguments) 360 STORE_NAME (print_line) 879: 362 LOAD_CONST (('', 'I')) - 364 LOAD_CONST () + 364 LOAD_CONST () 366 MAKE_FUNCTION (default) 368 STORE_NAME (ftpcp) -901: 370 LOAD_CONST () +901: 370 LOAD_CONST () 372 MAKE_FUNCTION (No arguments) 374 STORE_NAME (test) @@ -427,47 +427,47 @@ # 6: False # 7: 'utf-8' # 8: ('encoding',) -# 9: -# 10: -# 11: -# 12: -# 13: -# 14: -# 15: -# 16: -# 17: -# 18: -# 19: -# 20: -# 21: -# 22: -# 23: -# 24: -# 25: -# 26: -# 27: -# 28: -# 29: -# 30: -# 31: -# 32: -# 33: -# 34: -# 35: -# 36: -# 37: -# 38: -# 39: -# 40: -# 41: -# 42: -# 43: -# 44: -# 45: -# 46: -# 47: -# 48: -# 49: +# 9: +# 10: +# 11: +# 12: +# 13: +# 14: +# 15: +# 16: +# 17: +# 18: +# 19: +# 20: +# 21: +# 22: +# 23: +# 24: +# 25: +# 26: +# 27: +# 28: +# 29: +# 30: +# 31: +# 32: +# 33: +# 34: +# 35: +# 36: +# 37: +# 38: +# 39: +# 40: +# 41: +# 42: +# 43: +# 44: +# 45: +# 46: +# 47: +# 48: +# 49: # 50: ('', 0, -999, None) # 51: (None,) # 52: ('', '', '') @@ -582,181 +582,181 @@ 109: 66 LOAD_CONST (('encoding',)) 68 BUILD_CONST_KEY_MAP 1 - 70 LOAD_CONST () + 70 LOAD_CONST () 72 MAKE_FUNCTION (default, keyword-only) 74 STORE_NAME (__init__) -125: 76 LOAD_CONST () +125: 76 LOAD_CONST () 78 MAKE_FUNCTION (No arguments) 80 STORE_NAME (__enter__) -129: 82 LOAD_CONST () +129: 82 LOAD_CONST () 84 MAKE_FUNCTION (No arguments) 86 STORE_NAME (__exit__) 139: 88 LOAD_CONST (('', 0, -999, None)) - 90 LOAD_CONST () + 90 LOAD_CONST () 92 MAKE_FUNCTION (default) 94 STORE_NAME (connect) -165: 96 LOAD_CONST () +165: 96 LOAD_CONST () 98 MAKE_FUNCTION (No arguments) 100 STORE_NAME (getwelcome) -172: 102 LOAD_CONST () +172: 102 LOAD_CONST () 104 MAKE_FUNCTION (No arguments) 106 STORE_NAME (set_debuglevel) 179: 108 LOAD_NAME (set_debuglevel) 110 STORE_NAME (debug) -181: 112 LOAD_CONST () +181: 112 LOAD_CONST () 114 MAKE_FUNCTION (No arguments) 116 STORE_NAME (set_pasv) -188: 118 LOAD_CONST () +188: 118 LOAD_CONST () 120 MAKE_FUNCTION (No arguments) 122 STORE_NAME (sanitize) -195: 124 LOAD_CONST () +195: 124 LOAD_CONST () 126 MAKE_FUNCTION (No arguments) 128 STORE_NAME (putline) -205: 130 LOAD_CONST () +205: 130 LOAD_CONST () 132 MAKE_FUNCTION (No arguments) 134 STORE_NAME (putcmd) -211: 136 LOAD_CONST () +211: 136 LOAD_CONST () 138 MAKE_FUNCTION (No arguments) 140 STORE_NAME (getline) -229: 142 LOAD_CONST () +229: 142 LOAD_CONST () 144 MAKE_FUNCTION (No arguments) 146 STORE_NAME (getmultiline) -243: 148 LOAD_CONST () +243: 148 LOAD_CONST () 150 MAKE_FUNCTION (No arguments) 152 STORE_NAME (getresp) -257: 154 LOAD_CONST () +257: 154 LOAD_CONST () 156 MAKE_FUNCTION (No arguments) 158 STORE_NAME (voidresp) -264: 160 LOAD_CONST () +264: 160 LOAD_CONST () 162 MAKE_FUNCTION (No arguments) 164 STORE_NAME (abort) -278: 166 LOAD_CONST () +278: 166 LOAD_CONST () 168 MAKE_FUNCTION (No arguments) 170 STORE_NAME (sendcmd) -283: 172 LOAD_CONST () +283: 172 LOAD_CONST () 174 MAKE_FUNCTION (No arguments) 176 STORE_NAME (voidcmd) -288: 178 LOAD_CONST () +288: 178 LOAD_CONST () 180 MAKE_FUNCTION (No arguments) 182 STORE_NAME (sendport) -298: 184 LOAD_CONST () +298: 184 LOAD_CONST () 186 MAKE_FUNCTION (No arguments) 188 STORE_NAME (sendeprt) -311: 190 LOAD_CONST () +311: 190 LOAD_CONST () 192 MAKE_FUNCTION (No arguments) 194 STORE_NAME (makeport) -324: 196 LOAD_CONST () +324: 196 LOAD_CONST () 198 MAKE_FUNCTION (No arguments) 200 STORE_NAME (makepasv) 336: 202 LOAD_CONST ((None,)) - 204 LOAD_CONST () + 204 LOAD_CONST () 206 MAKE_FUNCTION (default) 208 STORE_NAME (ntransfercmd) 391: 210 LOAD_CONST ((None,)) - 212 LOAD_CONST () + 212 LOAD_CONST () 214 MAKE_FUNCTION (default) 216 STORE_NAME (transfercmd) 395: 218 LOAD_CONST (('', '', '')) - 220 LOAD_CONST () + 220 LOAD_CONST () 222 MAKE_FUNCTION (default) 224 STORE_NAME (login) 421: 226 LOAD_CONST ((8192, None)) - 228 LOAD_CONST () + 228 LOAD_CONST () 230 MAKE_FUNCTION (default) 232 STORE_NAME (retrbinary) 444: 234 LOAD_CONST ((None,)) - 236 LOAD_CONST () + 236 LOAD_CONST () 238 MAKE_FUNCTION (default) 240 STORE_NAME (retrlines) 479: 242 LOAD_CONST ((8192, None, None)) - 244 LOAD_CONST () + 244 LOAD_CONST () 246 MAKE_FUNCTION (default) 248 STORE_NAME (storbinary) 505: 250 LOAD_CONST ((None,)) - 252 LOAD_CONST () + 252 LOAD_CONST () 254 MAKE_FUNCTION (default) 256 STORE_NAME (storlines) -536: 258 LOAD_CONST () +536: 258 LOAD_CONST () 260 MAKE_FUNCTION (No arguments) 262 STORE_NAME (acct) -541: 264 LOAD_CONST () +541: 264 LOAD_CONST () 266 MAKE_FUNCTION (No arguments) 268 STORE_NAME (nlst) -550: 270 LOAD_CONST () +550: 270 LOAD_CONST () 272 MAKE_FUNCTION (No arguments) 274 STORE_NAME (dir) 565: 276 LOAD_CONST ("") 278 BUILD_LIST 0 280 BUILD_TUPLE 2 - 282 LOAD_CONST () + 282 LOAD_CONST () 284 MAKE_FUNCTION (default) 286 STORE_NAME (mlsd) -593: 288 LOAD_CONST () +593: 288 LOAD_CONST () 290 MAKE_FUNCTION (No arguments) 292 STORE_NAME (rename) -600: 294 LOAD_CONST () +600: 294 LOAD_CONST () 296 MAKE_FUNCTION (No arguments) 298 STORE_NAME (delete) -608: 300 LOAD_CONST () +608: 300 LOAD_CONST () 302 MAKE_FUNCTION (No arguments) 304 STORE_NAME (cwd) -621: 306 LOAD_CONST () +621: 306 LOAD_CONST () 308 MAKE_FUNCTION (No arguments) 310 STORE_NAME (size) -629: 312 LOAD_CONST () +629: 312 LOAD_CONST () 314 MAKE_FUNCTION (No arguments) 316 STORE_NAME (mkd) -638: 318 LOAD_CONST () +638: 318 LOAD_CONST () 320 MAKE_FUNCTION (No arguments) 322 STORE_NAME (rmd) -642: 324 LOAD_CONST () +642: 324 LOAD_CONST () 326 MAKE_FUNCTION (No arguments) 328 STORE_NAME (pwd) -651: 330 LOAD_CONST () +651: 330 LOAD_CONST () 332 MAKE_FUNCTION (No arguments) 334 STORE_NAME (quit) -657: 336 LOAD_CONST () +657: 336 LOAD_CONST () 338 MAKE_FUNCTION (No arguments) 340 STORE_NAME (close) 342 RETURN_CONST (None) @@ -777,14 +777,14 @@ # 2: None # 3: 'utf-8' # 4: ('context', 'timeout', 'source_address', 'encoding') -# 5: -# 6: -# 7: -# 8: -# 9: -# 10: -# 11: -# 12: +# 5: +# 6: +# 7: +# 8: +# 9: +# 10: +# 11: +# 12: # 13: ('', '', '', '') # 14: ('', '', '', True) # 15: (None,) @@ -828,41 +828,41 @@ 28 BUILD_CONST_KEY_MAP 4 30 LOAD_CLOSURE (__class__) 32 BUILD_TUPLE 1 - 34 LOAD_CONST () + 34 LOAD_CONST () 36 MAKE_FUNCTION (default, keyword-only, closure) 38 STORE_NAME (__init__) 721: 40 LOAD_CONST (('', '', '', True)) 42 LOAD_CLOSURE (__class__) 44 BUILD_TUPLE 1 - 46 LOAD_CONST () + 46 LOAD_CONST () 48 MAKE_FUNCTION (default, closure) 50 STORE_NAME (login) -726: 52 LOAD_CONST () +726: 52 LOAD_CONST () 54 MAKE_FUNCTION (No arguments) 56 STORE_NAME (auth) -738: 58 LOAD_CONST () +738: 58 LOAD_CONST () 60 MAKE_FUNCTION (No arguments) 62 STORE_NAME (ccc) -746: 64 LOAD_CONST () +746: 64 LOAD_CONST () 66 MAKE_FUNCTION (No arguments) 68 STORE_NAME (prot_p) -762: 70 LOAD_CONST () +762: 70 LOAD_CONST () 72 MAKE_FUNCTION (No arguments) 74 STORE_NAME (prot_c) 770: 76 LOAD_CONST ((None,)) 78 LOAD_CLOSURE (__class__) 80 BUILD_TUPLE 1 - 82 LOAD_CONST () + 82 LOAD_CONST () 84 MAKE_FUNCTION (default, closure) 86 STORE_NAME (ntransfercmd) -777: 88 LOAD_CONST () +777: 88 LOAD_CONST () 90 MAKE_FUNCTION (No arguments) 92 STORE_NAME (abort) 94 LOAD_CLOSURE (__class__) ```

one way of finding the issue is my approach. It's ofcourse possible you have a better idea.

I'm not sure where to go from here, what would be your next step?

[hacscred]

What I meant is:

Identify one of the outputs as the desired output (that is a mapping from byte address to byte code)/a "file".

Open a terminal. In this terminal find the exact dynamic program point at which the differently generated value is generated in a debugger, like gdb.

So, that will give you two pairs of type: (address, value) with potentially completely different values. However, you should be able to construct a mathematical expression (and certainly in a time travelling debugger (not sure whether gdb has a good implementation of that)) computing just that one wrong tuple value. By your observations it is impossible for that to be deterministic. I suspect that there are data structures being used without deterministic properties or possibly (but less likely) even parallelism.

The mathematical expression can be obtained by going up some frames and writing down what happens in a separate file or on a piece of paper.

If you do the previous process in two separate terminals for two different executions resulting in two different outputs at the same time, you can compare frame by frame what's happening and when the first dynamic difference happens, because that's what you are looking for. Then it's just repeating the whole progress for every new "first" dynamic difference, until the problem is resolved.

So, the brute force method is writing down the whole mathematical expression and a slightly more informed one is looking into the general structure of the code generator (which is not exactly an example of engineering).

I don't think debuggers exist which can extract the expression I discussed automatically, which kind of shows how bad tools still are, but I think gdb could be programmed to output such an expression, if you are feeling adventurous.