Unable To Append Two Emoji Strings On macOS Ventura's Swift

[brianmichel]

Description Appending two emoji strings together crashes on macOS Ventura

Steps to reproduce

1. Use macOS Ventura
2. Run the sample package's tests or
3. try to execute the code "⛓".appending("🦑") in the Swift repl

Expected behavior This code should run and execute fine producing a string that is equal to "⛓🦑"

Actual behavior This works correctly on macOS Monterey, but crashes on macOS Ventura with the message Swift/StringUTF16View.swift:144: Fatal error: String index is out of bounds

Environment

• Swift compiler version info -- swift-driver version: 1.62.15 Apple Swift version 5.7.2 (swiftlang-5.7.2.135.5 clang-1400.0.29.51) -- Target: arm64-apple-macosx13.0
• Xcode version info -- Xcode 14.2 -- Build version 14C18
• Deployment target: macOS 11

Sample Project FuzzyMatchingPackage.zip

[WowbaggersLiquidLunch]

I'm not able to reproduce the error. Could there be some bugs in your appending function? The standard library has only append functions.

[amonshiz]

appending is in the StringProtocol. https://developer.apple.com/documentation/swift/stringprotocol/appending(_:)

[danielpunkass]

FWIW I also am not able to reproduce the crash on Xcode Version 14.2 (14C18). Ventura 13.2.1 (22D68). (swiftlang-5.7.2.135.5 clang-1400.0.29.51)

[brianmichel]

I'm going to close this for now since it feels like we're missing a step somewhere in the repro (or some specific machine configuration).

[AnthonyLatsis]

appending is in the StringProtocol.

Yes, but regrettably the documentation does not mention that the function is declared in Foundation, not the Standard Library.

@brianmichel Does "⛓".appending("🦑") crash in the REPL by itself, or after importing Foundation?

[fischman-bcny]

@AnthonyLatsis Foundation needs to be imported first. This is what I see (note use of \u{} to make input 7-bit clean, and env -i usage to rule out locale-related env vars' impact):

~ $ env -i TERM=vt100 PATH=/usr/bin /usr/bin/swift repl
Welcome to Apple Swift version 5.7.2 (swiftlang-5.7.2.135.5 clang-1400.0.29.51).
Type :help for assistance.
  1> "\u{26D3}".appending("\u{1F991}")
expression failed to parse:
error: repl.swift:1:12: error: value of type 'String' has no member 'appending'
"\u{26D3}".appending("\u{1F991}")
~~~~~~~~~~ ^~~~~~~~~

  1> import Foundation
  2> "\u{26D3}".appending("\u{1F991}")
Swift/StringUTF16View.swift:144: Fatal error: String index is out of bounds
2023-02-15 09:34:51.301761-0800 repl_swift[17376:2992790] Swift/StringUTF16View.swift:144: Fatal error: String index is out of bounds
Execution interrupted. Enter code to recover and continue.
Enter LLDB commands to investigate (type :help for assistance.)

Note that adding an explicit NSString() to the mix makes the crash go away:

  3> NSString("\u{26D3}").appending("\u{1F991}")
$R0: String = "⛓🦑"

(this is on Ventura 13.2.1 (22D68) on a 2021 M1 Pro)

[danielpunkass]

When I follow precisely the same steps as @fischman-bcny, on the same OS, also with an M1 (Max) MacBook Pro, I don't see the same crash. Could we be getting different Foundations?

ProductName:        macOS
ProductVersion:     13.2.1
BuildVersion:       22D68
daniel@Taco ~/Sources/RSFoundation % env -i TERM=vt100 PATH=/usr/bin /usr/bin/swift repl
Welcome to Apple Swift version 5.7.2 (swiftlang-5.7.2.135.5 clang-1400.0.29.51).
Type :help for assistance.
  1>  "\u{26D3}".appending("\u{1F991}")
expression failed to parse:
error: repl.swift:1:13: error: value of type 'String' has no member 'appending'
 "\u{26D3}".appending("\u{1F991}")
 ~~~~~~~~~~ ^~~~~~~~~

  1> import Foundation
  2> "\u{26D3}".appending("\u{1F991}")
$R0: String = "⛓🦑"
  3>  

[AnthonyLatsis]

Thanks for clarifying. One more thing: Does it crash if you interpret it with swift file.swift?

// file.swift
import Foundation
let _ = "\u{26D3}".appending("\u{1F991}")

[AnthonyLatsis]

Could we be getting different Foundations?

Not if you both have a single Xcode installed and their versions match, but you can check for certain with print(NSFoundationVersionNumber). Maybe the crash is non-deterministic?

[fischman-bcny]

@AnthonyLatsis this is what I see:

$ cat file.swift ; echo; env -i PATH=/usr/bin /usr/bin/swift file.swift
import Foundation
print(NSFoundationVersionNumber)
let _ = "\u{26D3}".appending("\u{1F991}")

1953.3
Swift/StringUTF16View.swift:144: Fatal error: String index is out of bounds
Stack dump:
0.      Program arguments: /Applications/Xcode-14.2.0.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/swift-frontend -frontend -interpret file.swift -Xllvm -aarch64-use-tbi -enable-objc-interop -stack-check -sdk /Applications/Xcode-14.2.0.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX13.1.sdk -color-diagnostics -new-driver-path /Applications/Xcode-14.2.0.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/swift-driver -empty-abi-descriptor -resource-dir /Applications/Xcode-14.2.0.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/swift -module-name file -disable-clang-spi -target-sdk-version 13.1
1.      Apple Swift version 5.7.2 (swiftlang-5.7.2.135.5 clang-1400.0.29.51)
2.      Compiling with the current language version
3.      While running user code "file.swift"
Stack dump without symbol names (ensure you have llvm-symbolizer in your PATH or set the environment var `LLVM_SYMBOLIZER_PATH` to point to it):
0  swift-frontend           0x0000000104bb35b0 llvm::sys::PrintStackTrace(llvm::raw_ostream&, int) + 56
1  swift-frontend           0x0000000104bb25b4 llvm::sys::RunSignalHandlers() + 112
2  swift-frontend           0x0000000104bb3c34 SignalHandler(int) + 344
3  libsystem_platform.dylib 0x0000000196c702a4 _sigtramp + 56
4  libswiftCore.dylib       0x00000001a4a20fa0 $ss17_assertionFailure__4file4line5flagss5NeverOs12StaticStringV_A2HSus6UInt32VtFySRys5UInt8VGXEfU_yAMXEfU_yAMXEfU_ + 380
5  libswiftCore.dylib       0x00000001a4a20ce4 $ss17_assertionFailure__4file4line5flagss5NeverOs12StaticStringV_A2HSus6UInt32VtFySRys5UInt8VGXEfU_yAMXEfU_ + 200
6  libswiftCore.dylib       0x00000001a4a20adc $ss17_assertionFailure__4file4line5flagss5NeverOs12StaticStringV_A2HSus6UInt32VtFySRys5UInt8VGXEfU_ + 212
7  libswiftCore.dylib       0x00000001a4a2063c $ss17_assertionFailure__4file4line5flagss5NeverOs12StaticStringV_A2HSus6UInt32VtF + 236
8  libswiftCore.dylib       0x00000001a4a253b0 $sSKsE6_index_8offsetBy5IndexQzAD_SitFSS9UTF16ViewV_Tg5 + 1248
9  libswiftCore.dylib       0x00000001a4c0a084 $sSS9UTF16ViewV15_nativeGetIndex3forSS0E0VSi_tF + 272
10 libswiftCore.dylib       0x00000001a4c05860 $ss15__StringStorageC13getCharacters_5rangeySpys6UInt16VG_So13_SwiftNSRangeatF + 128
11 libswiftCore.dylib       0x00000001a4c059d4 $ss15__StringStorageC13getCharacters_5rangeySpys6UInt16VG_So13_SwiftNSRangeatFTo + 36
12 CoreFoundation           0x0000000196cca740 __CFStringEncodeByteStream + 2708
13 Foundation               0x0000000197bd52d4 -[NSString(NSStringOtherEncodings) getBytes:maxLength:usedLength:encoding:options:range:remainingRange:] + 260
14 Foundation               0x0000000197bda0cc _NSNewStringByAppendingStrings + 440
15 Foundation               0x0000000197bdab14 -[NSString stringByAppendingString:] + 76
16 Foundation               0x0000000197ff921c $sSy10FoundationE9appendingySSqd__SyRd__lF + 148
17 Foundation               0x000000010a2f0118 $sSy10FoundationE9appendingySSqd__SyRd__lF + 18446744071330295696
18 swift-frontend           0x000000010116b998 llvm::orc::runAsMain(int (*)(int, char**), llvm::ArrayRef<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >, llvm::Optional<llvm::StringRef>) + 1312
19 swift-frontend           0x0000000100449250 swift::RunImmediately(swift::CompilerInstance&, std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&, swift::IRGenOptions const&, swift::SILOptions const&, std::__1::unique_ptr<swift::SILModule, std::__1::default_delete<swift::SILModule> >&&) + 11268
20 swift-frontend           0x00000001001feccc performCompileStepsPostSILGen(swift::CompilerInstance&, std::__1::unique_ptr<swift::SILModule, std::__1::default_delete<swift::SILModule> >, llvm::PointerUnion<swift::ModuleDecl*, swift::SourceFile*>, swift::PrimarySpecificPaths const&, int&, swift::FrontendObserver*) + 2856
21 swift-frontend           0x0000000100200ce8 swift::performFrontend(llvm::ArrayRef<char const*>, char const*, void*, swift::FrontendObserver*) + 7284
22 swift-frontend           0x00000001001a1294 swift::mainEntry(int, char const**) + 3940
23 dyld                     0x0000000196917e50 start + 2544
Trace/BPT trap: 5

[AnthonyLatsis]

cc @parkera

[fischman-bcny]

Ping :) Am interested in any other suggestions that might help divine where my system is differing from others' on this thread (and whether that difference matters to the software we ship)

[compnerd]

CC: @lorentey - in case something about the bridging stands out to him

[lorentey]

StringProtocol.appending is going out of its way to bridge its input as many times as possible, for no good reason. It's long overdue for a rewrite (and/or deprecation), but it should never crash.

The crash may be due to a bug in one of these related String changes:

https://github.com/apple/swift/pull/62717 https://github.com/apple/swift/pull/63592 https://github.com/apple/swift/pull/63646

I'll take a closer look.

[lorentey]

One very reliable workaround is to avoid calling StringProtocol.appending, preferring to use the Swift native + operator instead. (There are no drawbacks to doing that. Besides investigating the potential stdlib issue, I'll also try to change the implementation of appending to do the same. However, that change will not apply retroactively, even if it ends up shipping. + does the right thing on all platforms.)

[lorentey]

This is reliably reproducible by overriding Core Foundation's default text encoding from Mac OS Roman to UTF-8:

$ cat foo.swift
import Foundation
let _ = "\u{26D3}".appending("\u{1F991}")
$ swiftc foo.swift
$ __CF_USER_TEXT_ENCODING="$(id -u):0x08000100:0" ./foo
Swift/StringUTF16View.swift:147: Fatal error: String index is out of bounds
Trace/BPT trap: 5

This switches concatenation to a different algorithm, which exhibits the issue. I'm continuing to investigate.

(UTF-8 is likely to be the default text encoding at least in some regions, but evidently not all.)

[fischman-bcny]

@lorentey the tip to use + worked a treat for the reliable crash in our app; thanks!

Re: encoding/locale: note that up-thread there are env -i-based repro recipes that should have isolated the results from the contents of my environment, yet others fail to reproduce the crash. Curious if you know of other places besides the environment where encoding/locale/etc info might be sneaking into the mix (that might differ in your/my systems from those of others' who have failed to repro the crash).

[lorentey]

@fischman-bcny If the environment variable isn't set, Core Foundation has a legacy fallback to get its default encoding from the file ~/.CFUserTextEncoding, which usually does exist on macOS. Perhaps that's where the diverging configurations come from.

(Defaulting to anything other than UTF-8 doesn't make much sense to me these days, but fwiw it appears my Mac (in the US region) is still configured to prefer Mac Roman in this context. 🤷🏼‍♂️)

[fischman-bcny]

@lorentey nice! Indeed, I have a ~/.CFUserTextEncoding file containing 0x08000100:0x08000100. When I rename it, the crash from https://github.com/apple/swift/issues/63664#issuecomment-1431915696 goes away, and when I put it back the crash is back.

[lorentey]

It turns out that the fault lies is outside of the Swift stdlib in this case, although it got triggered by Swift 5.7 in Ventura starting to trap on more out-of-bounds string accesses. We're tracking this issue separately (rdar://108765311); the fix will ship in a future OS release.

@fischman-bcny Do you happen to remember if you manually edited that file to set those contents, or if something configured it automatically?

[fischman-bcny]

@lorentey not sure, sorry.

[AnthonyLatsis]

It turns out that the fault lies is outside of the Swift stdlib in this case

@lorentey Is the issue with Foundation after all? If so, perhaps we should transfer the issue to the newly published Foundation repository?

[lorentey]

It is a Foundation issue, but it does not involve the code in swift-foundation. I think it's okay to keep the issue here for now.