search

esp-rs / esp-mbedtls

mbedtls for ESP32 bare-metal
Apache License 2.0
17 stars 7 forks source link

[esp32c3] Crash when using set_debug(i >=3) in server examples. #11

Closed AnthonyGrondin closed 1 year ago

AnthonyGrondin commented 1 year ago

When setting set_debug(3) with 3 or higher, there's a crash during the handshake for the server examples.

This has only been tested on esp32c3 because Xtensa uses a no debug build.

This happens for both async_server and sync_server, here's the output log for async_server with level 3:

Waiting for connection...
Connected...
Start tls connect
INFO - 2 /mnt/c/projects/esp/esp-mbedtls/build_mbedtls/tmpsrc/mbedtls/library/ssl_tls.c:3939 => handshake

INFO - 2 /mnt/c/projects/esp/esp-mbedtls/build_mbedtls/tmpsrc/mbedtls/library/ssl_msg.c:2124 => flush output

INFO - 2 /mnt/c/projects/esp/esp-mbedtls/build_mbedtls/tmpsrc/mbedtls/library/ssl_msg.c:2133 <= flush output

INFO - 2 /mnt/c/projects/esp/esp-mbedtls/build_mbedtls/tmpsrc/mbedtls/library/ssl_tls12_server.c:4228 server state: 0

INFO - 2 /mnt/c/projects/esp/esp-mbedtls/build_mbedtls/tmpsrc/mbedtls/library/ssl_msg.c:2124 => flush output

INFO - 2 /mnt/c/projects/esp/esp-mbedtls/build_mbedtls/tmpsrc/mbedtls/library/ssl_msg.c:2133 <= flush output

INFO - 2 /mnt/c/projects/esp/esp-mbedtls/build_mbedtls/tmpsrc/mbedtls/library/ssl_tls12_server.c:4228 server state: 1

INFO - 2 /mnt/c/projects/esp/esp-mbedtls/build_mbedtls/tmpsrc/mbedtls/library/ssl_tls12_server.c:914 => parse client hello

INFO - 2 /mnt/c/projects/esp/esp-mbedtls/build_mbedtls/tmpsrc/mbedtls/library/ssl_msg.c:1926 => fetch input

INFO - 2 /mnt/c/projects/esp/esp-mbedtls/build_mbedtls/tmpsrc/mbedtls/library/ssl_msg.c:2066 in_left: 0, nb_want: 5

INFO - 2 /mnt/c/projects/esp/esp-mbedtls/build_mbedtls/tmpsrc/mbedtls/library/ssl_msg.c:2086 in_left: 0, nb_want: 5

INFO - 2 /mnt/c/projects/esp/esp-mbedtls/build_mbedtls/tmpsrc/mbedtls/library/ssl_tls.c:3950 <= handshake

INFO - 2 /mnt/c/projects/esp/esp-mbedtls/build_mbedtls/tmpsrc/mbedtls/library/ssl_tls.c:3939 => handshake

INFO - 2 /mnt/c/projects/esp/esp-mbedtls/build_mbedtls/tmpsrc/mbedtls/library/ssl_msg.c:2124 => flush output

INFO - 2 /mnt/c/projects/esp/esp-mbedtls/build_mbedtls/tmpsrc/mbedtls/library/ssl_msg.c:2133 <= flush output

INFO - 2 /mnt/c/projects/esp/esp-mbedtls/build_mbedtls/tmpsrc/mbedtls/library/ssl_tls12_server.c:4228 server state: 1

INFO - 2 /mnt/c/projects/esp/esp-mbedtls/build_mbedtls/tmpsrc/mbedtls/library/ssl_tls12_server.c:914 => parse client hello

INFO - 2 /mnt/c/projects/esp/esp-mbedtls/build_mbedtls/tmpsrc/mbedtls/library/ssl_msg.c:1926 => fetch input

INFO - 2 /mnt/c/projects/esp/esp-mbedtls/build_mbedtls/tmpsrc/mbedtls/library/ssl_msg.c:2066 in_left: 0, nb_want: 5

INFO - 2 /mnt/c/projects/esp/esp-mbedtls/build_mbedtls/tmpsrc/mbedtls/library/ssl_msg.c:2086 in_left: 0, nb_want: 5

INFO - 2 /mnt/c/projects/esp/esp-mbedtls/build_mbedtls/tmpsrc/mbedtls/library/ssl_msg.c:2089 ssl->f_recv(_timeout)() returned 5 (-0xfffffffb)
0xfffffffb - _rtc_fast_bss_end
    at ??:??

INFO - 2 /mnt/c/projects/esp/esp-mbedtls/build_mbedtls/tmpsrc/mbedtls/library/ssl_msg.c:2111 <= fetch input

INFO - 3 /mnt/c/projects/esp/esp-mbedtls/build_mbedtls/tmpsrc/mbedtls/library/ssl_tls12_server.c:951 client hello, message type: 22

INFO - 3 /mnt/c/projects/esp/esp-mbedtls/build_mbedtls/tmpsrc/mbedtls/library/ssl_tls12_server.c:959 client hello, message len.: 512

INFO - 3 /mnt/c/projects/esp/esp-mbedtls/build_mbedtls/tmpsrc/mbedtls/library/ssl_tls12_server.c:962 client hello, protocol version: [3:1]

INFO - 2 /mnt/c/projects/esp/esp-mbedtls/build_mbedtls/tmpsrc/mbedtls/library/ssl_msg.c:1926 => fetch input

INFO - 2 /mnt/c/projects/esp/esp-mbedtls/build_mbedtls/tmpsrc/mbedtls/library/ssl_msg.c:2066 in_left: 5, nb_want: 517

INFO - 2 /mnt/c/projects/esp/esp-mbedtls/build_mbedtls/tmpsrc/mbedtls/library/ssl_msg.c:2086 in_left: 5, nb_want: 517

INFO - 2 /mnt/c/projects/esp/esp-mbedtls/build_mbedtls/tmpsrc/mbedtls/library/ssl_msg.c:2089 ssl->f_recv(_timeout)() returned 512 (-0xfffffe00)
0xfffffe00 - _rtc_fast_bss_end
    at ??:??

INFO - 2 /mnt/c/projects/esp/esp-mbedtls/build_mbedtls/tmpsrc/mbedtls/library/ssl_msg.c:2111 <= fetch input

INFO - 3 /mnt/c/projects/esp/esp-mbedtls/build_mbedtls/tmpsrc/mbedtls/library/ssl_tls12_server.c:1048 client hello v3, handshake type: 1

INFO - 3 /mnt/c/projects/esp/esp-mbedtls/build_mbedtls/tmpsrc/mbedtls/library/ssl_tls12_server.c:1056 client hello v3, handshake len.: 508

INFO - 3 /mnt/c/projects/esp/esp-mbedtls/build_mbedtls/tmpsrc/mbedtls/library/ssl_tls12_server.c:1158 dumping 'client hello, version' (2 bytes)

INFO - 3 /mnt/c/projects/esp/esp-mbedtls/build_mbedtls/tmpsrc/mbedtls/library/ssl_tls12_server.c:1158 00:  03 03                                            ..

INFO - 3 /mnt/c/projects/esp/esp-mbedtls/build_mbedtls/tmpsrc/mbedtls/library/ssl_tls12_server.c:1173 dumping 'client hello, random bytes' (32 bytes)

INFO - 3 /mnt/c/projects/esp/esp-mbedtls/build_mbedtls/tmpsrc/mbedtls/library/ssl_tls12_server.c:1173 00:  ec 8d 64 08 bb 38 2c 20 79 b9 db 56 38 1a 8d fb  ..d..8, y..V8...

INFO - 3 /mnt/c/projects/esp/esp-mbedtls/build_mbedtls/tmpsrc/mbedtls/library/ssl_tls12_server.c:1173 10:  62 09 18 8e 33 63 ea 55 8e c7 5f 9f de 8d cd cb  b...3c.U.._.....

INFO - 3 /mnt/c/projects/esp/esp-mbedtls/build_mbedtls/tmpsrc/mbedtls/library/ssl_tls12_server.c:1190 dumping 'client hello, session id' (32 bytes)

INFO - 3 /mnt/c/projects/esp/esp-mbedtls/build_mbedtls/tmpsrc/mbedtls/library/ssl_tls12_server.c:1190 00:  46 d3 37 3f fa 32 5e 7e f7 b0 8b e1 19 33 c0 bb  F.7?.2^~.....3..

INFO - 3 /mnt/c/projects/esp/esp-mbedtls/build_mbedtls/tmpsrc/mbedtls/library/ssl_tls12_server.c:1190 10:  1d 26 3e cc ff 62 4c a1 2f f6 55 f4 d2 5f 2c f7  .&>..bL./.U.._,.

INFO - 3 /mnt/c/projects/esp/esp-mbedtls/build_mbedtls/tmpsrc/mbedtls/library/ssl_tls12_server.c:1264 dumping 'client hello, ciphersuitelist' (36 bytes)

INFO - 3 /mnt/c/projects/esp/esp-mbedtls/build_mbedtls/tmpsrc/mbedtls/library/ssl_tls12_server.c:1264 00:  13 01 13 03 13 02 c0 2b c0 2f cc a9 cc a8 c0 2c  .......+./.....,

INFO - 3 /mnt/c/projects/esp/esp-mbedtls/build_mbedtls/tmpsrc/mbedtls/library/ssl_tls12_server.c:1264 10:  c0 30 c0 0a c0 09 c0 13 c0 14 00 9c 00 9d 00 2f  .0............./

INFO - 3 /mnt/c/projects/esp/esp-mbedtls/build_mbedtls/tmpsrc/mbedtls/library/ssl_tls12_server.c:1264 20:  00 35 00 0a                                      .5..

INFO - 3 /mnt/c/projects/esp/esp-mbedtls/build_mbedtls/tmpsrc/mbedtls/library/ssl_tls12_server.c:1286 dumping 'client hello, compression' (1 bytes)

INFO - 3 /mnt/c/projects/esp/esp-mbedtls/build_mbedtls/tmpsrc/mbedtls/library/ssl_tls12_server.c:1286 00:  00                                               .

INFO - 3 /mnt/c/projects/esp/esp-mbedtls/build_mbedtls/tmpsrc/mbedtls/library/ssl_tls12_server.c:1315 dumping 'client hello extensions' (399 bytes)

INFO - 3 /mnt/c/projects/esp/esp-mbedtls/build_mbedtls/tmpsrc/mbedtls/library/ssl_tls12_server.c:1315 00:  00 17 00 00 ff 01 00 01 00 00 0a 00 0e 00 0c 00  ................

INFO - 3 /mnt/c/projects/esp/esp-mbedtls/build_mbedtls/tmpsrc/mbedtls/library/ssl_tls12_server.c:1315 10:  1d 00 17 00 18 00 19 01 00 01 01 00 0b 00 02 01  ................

INFO - 3 /mnt/c/projects/esp/esp-mbedtls/build_mbedtls/tmpsrc/mbedtls/library/ssl_tls12_server.c:1315 20:  00 00 23 00 00 00 10 00 0e 00 0c 02 68 32 08 68  ..#.........h2.h

INFO - 3 /mnt/c/projects/esp/esp-mbedtls/build_mbedtls/tmpsrc/mbedtls/library/ssl_tls12_server.c:1315 30:  74 74 70 2f 31 2e 31 00 05 00 05 01 00 00 00 00  ttp/1.1.........

INFO - 3 /mnt/c/projects/esp/esp-mbedtls/build_mbedtls/tmpsrc/mbedtls/library/ssl_tls12_server.c:1315 40:  00 22 00 0a 00 08 04 03 05 03 06 03 02 03 00 33  .".............3

INFO - 3 /mnt/c/projects/esp/esp-mbedtls/build_mbedtls/tmpsrc/mbedtls/library/ssl_tls12_server.c:1315 50:  00 6b 00 69 00 1d 00 20 da b6 7b 44 6c 93 c9 9f  .k.i... ..{Dl...

INFO - 3 /mnt/c/projects/esp/esp-mbedtls/build_mbedtls/tmpsrc/mbedtls/library/ssl_tls12_server.c:1315 60:  02 4f 58 27 cf ce c0 8e d4 5b 86 a4 35 5d f7 7e  .OX'.....[..5].~

INFO - 3 /mnt/c/projects/esp/esp-mbedtls/build_mbedtls/tmpsrc/mbedtls/library/ssl_tls12_server.c:1315 70:  83 32 48 71 26 61 aa 6d 00 17 00 41 04 bb ee d9  .2Hq&a.m...A....

INFO - 3 /mnt/c/projects/esp/esp-mbedtls/build_mbedtls/tmpsrc/mbedtls/library/ssl_tls12_server.c:1315 80:  67 7b 73 78 01 b7 d9 a2 f0 a0 3e 67 36 ba 61 9f  g{sx......>g6.a.

INFO - 3 /mnt/c/projects/esp/esp-mbedtls/build_mbedtls/tmpsrc/mbedtls/library/ssl_tls12_server.c:1315 90:  87 70 23 66 89 01 13 11 98 0c 8f 0a eb a0 83 27  .p#f...........'

INFO - 3 /mnt/c/projects/esp/esp-mbedtls/build_mbedtls/tmpsrc/mbedtls/library/ssl_tls12_server.c:1315 a0:  c4 58 89 69 c8 7e 82 ff 0d ec 55 33 dd b7 33 5e  .X.i.~....U3..3^

INFO - 3 /mnt/c/projects/esp/esp-mbedtls/build_mbedtls/tmpsrc/mbedtls/library/ssl_tls12_server.c:1315 b0:  f1 cc d7 ce cb e7 05 05 c5 57 ac 47 9e 00 2b 00  .........W.G..+.

INFO - 3 /mnt/c/projects/esp/esp-mbedtls/build_mbedtls/tmpsrc/mbedtls/library/ssl_tls12_server.c:1315 c0:  09 08 03 04 03 03 03 02 03 01 00 0d 00 18 00 16  ................

INFO - 3 /mnt/c/projects/esp/esp-mbedtls/build_mbedtls/tmpsrc/mbedtls/library/ssl_tls12_server.c:1315 d0:  04 03 05 03 06 03 08 04 08 05 08 06 04 01 05 01  ................

INFO - 3 /mnt/c/projects/esp/esp-mbedtls/build_mbedtls/tmpsrc/mbedtls/library/ssl_tls12_server.c:1315 e0:  06 01 02 03 02 01 00 2d 00 02 01 01 00 1c 00 02  .......-........

INFO - 3 /mnt/c/projects/esp/esp-mbedtls/build_mbedtls/tmpsrc/mbedtls/library/ssl_tls12_server.c:1315 f0:  40 01 00 15 00 99 00 00 00 00 00 00 00 00 00 00  @...............

INFO - 3 /mnt/c/projects/esp/esp-mbedtls/build_mbedtls/tmpsrc/mbedtls/library/ssl_tls12_server.c:1315 100:  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................

INFO - 3 /mnt/c/projects/esp/esp-mbedtls/build_mbedtls/tmpsrc/mbedtls/library/ssl_tls12_server.c:1315 110:  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................

INFO - 3 /mnt/c/projects/esp/esp-mbedtls/build_mbedtls/tmpsrc/mbedtls/library/ssl_tls12_server.c:1315 120:  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................

INFO - 3 /mnt/c/projects/esp/esp-mbedtls/build_mbedtls/tmpsrc/mbedtls/library/ssl_tls12_server.c:1315 130:  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................

INFO - 3 /mnt/c/projects/esp/esp-mbedtls/build_mbedtls/tmpsrc/mbedtls/library/ssl_tls12_server.c:1315 140:  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................

INFO - 3 /mnt/c/projects/esp/esp-mbedtls/build_mbedtls/tmpsrc/mbedtls/library/ssl_tls12_server.c:1315 150:  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................

INFO - 3 /mnt/c/projects/esp/esp-mbedtls/build_mbedtls/tmpsrc/mbedtls/library/ssl_tls12_server.c:1315 160:  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................

INFO - 3 /mnt/c/projects/esp/esp-mbedtls/build_mbedtls/tmpsrc/mbedtls/library/ssl_tls12_server.c:1315 170:  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................

INFO - 3 /mnt/c/projects/esp/esp-mbedtls/build_mbedtls/tmpsrc/mbedtls/library/ssl_tls12_server.c:1315 180:  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00     ...............

INFO - 3 /mnt/c/projects/esp/esp-mbedtls/build_mbedtls/tmpsrc/mbedtls/library/ssl_tls12_server.c:1441 found extended master secret extension

INFO - 3 /mnt/c/projects/esp/esp-mbedtls/build_mbedtls/tmpsrc/mbedtls/library/ssl_tls12_server.c:1348 found renegotiation extension

INFO - 3 /mnt/c/projects/esp/esp-mbedtls/build_mbedtls/tmpsrc/mbedtls/library/ssl_tls12_server.c:1375 found supported elliptic curves extension

INFO - 3 /mnt/c/projects/esp/esp-mbedtls/build_mbedtls/tmpsrc/mbedtls/library/ssl_tls12_server.c:1384 found supported point formats extension

INFO - 3 /mnt/c/projects/esp/esp-mbedtls/build_mbedtls/tmpsrc/mbedtls/library/ssl_tls12_server.c:1452 found session ticket extension

INFO - 3 /mnt/c/projects/esp/esp-mbedtls/build_mbedtls/tmpsrc/mbedtls/library/ssl_tls12_server.c:1463 found alpn extension

INFO - 3 /mnt/c/projects/esp/esp-mbedtls/build_mbedtls/tmpsrc/mbedtls/library/ssl_tls12_server.c:1485 unknown extension found: 5 (ignoring)

INFO - 3 /mnt/c/projects/esp/esp-mbedtls/build_mbedtls/tmpsrc/mbedtls/library/ssl_tls12_server.c:1485 unknown extension found: 34 (ignoring)

INFO - 3 /mnt/c/projects/esp/esp-mbedtls/build_mbedtls/tmpsrc/mbedtls/library/ssl_tls12_server.c:1485 unknown extension found: 51 (ignoring)

INFO - 3 /mnt/c/projects/esp/esp-mbedtls/build_mbedtls/tmpsrc/mbedtls/library/ssl_tls12_server.c:1485 unknown extension found: 43 (ignoring)

INFO - 3 /mnt/c/projects/esp/esp-mbedtls/build_mbedtls/tmpsrc/mbedtls/library/ssl_tls12_server.c:1361 found signature_algorithms extension

INFO - 3 /mnt/c/projects/esp/esp-mbedtls/build_mbedtls/tmpsrc/mbedtls/library/ssl_tls12_server.c:1485 unknown extension found: 45 (ignoring)

INFO - 3 /mnt/c/projects/esp/esp-mbedtls/build_mbedtls/tmpsrc/mbedtls/library/ssl_tls12_server.c:1485 unknown extension found: 28 (ignoring)

INFO - 3 /mnt/c/projects/esp/esp-mbedtls/build_mbedtls/tmpsrc/mbedtls/library/ssl_tls12_server.c:1485 unknown extension found: 21 (ignoring)

Exception 'Load access fault' mepc=0x40058e94, mtval=0x00001303
0x40058e94 - r_lld_init_evt_end_type_check_state_get
    at ??:??
0x00001303 - 
    at ??:??
TrapFrame
PC=0x40058e94         RA/x1=0x4201562a      SP/x2=0x3fc84be0      GP/x3=0x3fcca130      TP/x4=0x00000000
0x40058e94 - r_lld_init_evt_end_type_check_state_get
    at ??:??
0x4201562a - core::ffi::c_str::CStr::from_ptr
    at /home/anthony/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ffi/c_str.rs:296
0x3fc84be0 - __global_pointer$
    at ??:??
0x3fcca130 - _heap_start
    at ??:??
0x00000000 - hmac_md5_vector
    at /home/bjoern/esp/esp-idf/components/wpa_supplicant/src/crypto/md5.c:107
T0/x5=0x000000df      T1/x6=0x3fcca160      T2/x7=0x3c0b8044      S0/FP/x8=0x3fcca7d0   S1/x9=0x00001303
0x000000df - hmac_md5_vector
    at /home/anthony/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/num/mod.rs:1167
0x3fcca160 - _heap_start
    at ??:??
0x3c0b8044 - .Lanon.a637d6179f1be8d572ea13cf18a8161c.17
    at ??:??
0x3fcca7d0 - _heap_start
    at ??:??
0x00001303 - 
    at ??:??
A0/x10=0x00001303     A1/x11=0x3fccaa08     A2/x12=0x3fcca182     A3/x13=0x00000200     A4/x14=0x00000003
0x00001303 - 
    at ??:??
0x3fccaa08 - _heap_start
    at ??:??
0x3fcca182 - _heap_start
    at ??:??
0x00000200 - _ZN17compiler_builtins5float3div8__divdf317h5e229b8a6a8eb12dE
    at ??:??
0x00000003 - hmac_md5_vector
    at /home/bjoern/esp/esp-idf/components/wpa_supplicant/src/crypto/md5.c:107
A5/x15=0x00001303     A6/x16=0x000000f0     A7/x17=0x00110000     S2/x18=0x00000002     S3/x19=0x3c0b7fc4
0x00001303 - 
    at ??:??
0x000000f0 - hmac_md5_vector
    at /home/anthony/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/bit.rs:790
0x00110000 - 
    at ??:??
0x00000002 - hmac_md5_vector
    at /home/bjoern/esp/esp-idf/components/wpa_supplicant/src/crypto/md5.c:107
0x3c0b7fc4 - .Lanon.a637d6179f1be8d572ea13cf18a8161c.0
    at ??:??
S4/x20=0x00000020     S5/x21=0x3fcca382     S6/x22=0x3fcca381     S7/x23=0x00000000     S8/x24=0x00000001
0x00000020 - hmac_md5_vector
    at /home/bjoern/esp/esp-idf/components/wpa_supplicant/src/crypto/md5-internal.c:103
0x3fcca382 - _heap_start
    at ??:??
0x3fcca381 - _heap_start
    at ??:??
0x00000000 - hmac_md5_vector
    at /home/bjoern/esp/esp-idf/components/wpa_supplicant/src/crypto/md5.c:107
0x00000001 - hmac_md5_vector
    at /home/bjoern/esp/esp-idf/components/wpa_supplicant/src/crypto/md5.c:107
S9/x25=0x3c0b8028     S10/x26=0x3c0b7f28    S11/x27=0x3fcca160    T3/x28=0x00000000     T4/x29=0x00000000
0x3c0b8028 - .Lanon.a637d6179f1be8d572ea13cf18a8161c.11
    at ??:??
0x3c0b7f28 - .LJTI16_0
    at ??:??
0x3fcca160 - _heap_start
    at ??:??
0x00000000 - hmac_md5_vector
    at /home/bjoern/esp/esp-idf/components/wpa_supplicant/src/crypto/md5.c:107
0x00000000 - hmac_md5_vector
    at /home/bjoern/esp/esp-idf/components/wpa_supplicant/src/crypto/md5.c:107
T5/x30=0x00000000     T6/x31=0x00000000
0x00000000 - hmac_md5_vector
    at /home/bjoern/esp/esp-idf/components/wpa_supplicant/src/crypto/md5.c:107
0x00000000 - hmac_md5_vector
    at /home/bjoern/esp/esp-idf/components/wpa_supplicant/src/crypto/md5.c:107

MSTATUS=0x00001881
0x00001881 - 
    at ??:??
MCAUSE=0x00000005
0x00000005 - hmac_md5_vector
    at /home/bjoern/esp/esp-idf/components/wpa_supplicant/src/crypto/md5.c:107
MTVAL=0x00001303
0x00001303 - 
    at ??:??

No backtrace available - make sure to force frame-pointers. (see https://crates.io/crates/esp-backtrace)
bjoernQ commented 1 year ago

Ok this one is caused by an unexpected formatting symbol. Fixable by changing

esp-mbedtls\src\compat.rs:41

to if c.is_numeric() || c == '-' || c == 'l' || c == 'z' || c == '#' {

We didn't expect # here