Formatter: `string_processing` preview style

[MichaReiser]

Add support for Black's improved string processing.

Black will split long string literals and merge short ones. Parentheses are used where appropriate. When split, parts of f-strings that don’t need formatting are converted to plain strings. User-made splits are respected when they do not exceed the line length limit. Line continuation backslashes are converted into parenthesized strings. Unnecessary parentheses are stripped. The stability and status of this feature is tracked in this issue.

Examples

"a somewhat long string that doesn't fit the line length. " "Testing to see what black does keep it a bit longer and longer",
"a somewhat long string that doesn't fit the line length. " \
    "Testing to see what black does keep it a bit longer and longer",

if "a somewhat long string that doesn't fit the line length. " + "Testing to see what black does keep it a bit longer and longer longer":
    pass

"a somewhat long string that doesn't fit the line length. " + "Testing to see what black does keep it a bit longer and longer longer longer longer longer longer longer longer longer longer longer"

("a somewhat long string that doesn't fit the line length. " + "Testing to see what black does keep it a bit longer and longer longer longer longer longer longer longer longer longer longer longer")

"a somewhat long string that doesn't fit the line length. " + ["aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb", "Ccccccccccccccccccccccccccccccccccccccc cccccccccccccccccccccccccccccccccccccccccccccccccccccc"]

call(
    "a somewhat long string that doesn't fit the line length. "
    "Testing to see what black does keep it a bit longer and longer",
    "a second argument",
    3,
    4,
)

call(
    "a somewhat long string that doesn't fit the line length. Testing to see what black does keep "
    "it a bit longer and longer",
    "a second argument",
    3,
    4,
)

call(
    "a somewhat long string that doesn't fit the line length. "
    "shorter",
    "a second argument",
    3,
    4,
)

L1 = ["The is a short string", "This is a really long string that can't possibly be expected to fit all together on one line. Also it is inside a list literal, so it's expected to be wrapped in parens when spliting to avoid implicit str concatenation.", short_call("arg", {"key": "value"}), "This is another really really (not really) long string that also can't be expected to fit on one line and is, like the other string, inside a list literal.", ("parens should be stripped for short string in list")]

--- /home/micha/.config/JetBrains/PyCharmCE2023.2/scratches/scratch.py  2023-08-28 09:15:01.476535+00:00
+++ /home/micha/.config/JetBrains/PyCharmCE2023.2/scratches/scratch.py  2023-08-28 09:15:02.026782+00:00
@@ -1,17 +1,36 @@
-"a somewhat long string that doesn't fit the line length. " "Testing to see what black does keep it a bit longer and longer",
-"a somewhat long string that doesn't fit the line length. " \
-    "Testing to see what black does keep it a bit longer and longer",
+"a somewhat long string that doesn't fit the line length. "
+"Testing to see what black does keep it a bit longer and longer",
+"a somewhat long string that doesn't fit the line length. "
+"Testing to see what black does keep it a bit longer and longer",

-if "a somewhat long string that doesn't fit the line length. " + "Testing to see what black does keep it a bit longer and longer longer":
+if (
+    "a somewhat long string that doesn't fit the line length. "
+    + "Testing to see what black does keep it a bit longer and longer longer"
+):
     pass

-"a somewhat long string that doesn't fit the line length. " + "Testing to see what black does keep it a bit longer and longer longer longer longer longer longer longer longer longer longer longer"
+(
+    "a somewhat long string that doesn't fit the line length. "
+    + "Testing to see what black does keep it a bit longer and longer longer longer"
+    " longer longer longer longer longer longer longer longer"
+)

-("a somewhat long string that doesn't fit the line length. " + "Testing to see what black does keep it a bit longer and longer longer longer longer longer longer longer longer longer longer longer")
+(
+    "a somewhat long string that doesn't fit the line length. "
+    + "Testing to see what black does keep it a bit longer and longer longer longer"
+    " longer longer longer longer longer longer longer longer"
+)

-"a somewhat long string that doesn't fit the line length. " + ["aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb", "Ccccccccccccccccccccccccccccccccccccccc cccccccccccccccccccccccccccccccccccccccccccccccccccccc"]
+"a somewhat long string that doesn't fit the line length. " + [
+    "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa",
+    "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb",
+    (
+        "Ccccccccccccccccccccccccccccccccccccccc"
+        " cccccccccccccccccccccccccccccccccccccccccccccccccccccc"
+    ),
+]

 call(
     "a somewhat long string that doesn't fit the line length. "
     "Testing to see what black does keep it a bit longer and longer",
     "a second argument",
@@ -19,22 +38,35 @@
     4,
 )

 call(
-    "a somewhat long string that doesn't fit the line length. Testing to see what black does keep "
-    "it a bit longer and longer",
+    "a somewhat long string that doesn't fit the line length. Testing to see what black"
+    " does keep it a bit longer and longer",
     "a second argument",
     3,
     4,
 )

 call(
-    "a somewhat long string that doesn't fit the line length. "
-    "shorter",
+    "a somewhat long string that doesn't fit the line length. shorter",
     "a second argument",
     3,
     4,
 )

-L1 = ["The is a short string", "This is a really long string that can't possibly be expected to fit all together on one line. Also it is inside a list literal, so it's expected to be wrapped in parens when spliting to avoid implicit str concatenation.", short_call("arg", {"key": "value"}), "This is another really really (not really) long string that also can't be expected to fit on one line and is, like the other string, inside a list literal.", ("parens should be stripped for short string in list")]
\ No newline at end of file
+L1 = [
+    "The is a short string",
+    (
+        "This is a really long string that can't possibly be expected to fit all"
+        " together on one line. Also it is inside a list literal, so it's expected to"
+        " be wrapped in parens when spliting to avoid implicit str concatenation."
+    ),
+    short_call("arg", {"key": "value"}),
+    (
+        "This is another really really (not really) long string that also can't be"
+        " expected to fit on one line and is, like the other string, inside a list"
+        " literal."
+    ),
+    "parens should be stripped for short string in list",
+]

Observations:

• Binary operators break before the strings
• The formatting only applies to implicit concatenated strings, but not strings joined with the + operator

The strings must be parenthesized in some contexts, see https://github.com/psf/black/pull/3162/files

[MichaReiser]

My first reaction is that this is similar to Prettier's/Rome's JSX formatting and that we need to use BestFitting with three variants:

1. Join all string parts into a single string to make it fit
2. Keep the same string parts as in the source. Test if all lines fit
3. Use fill see prototype to fit as many words as possible on the line.

The main concern with this is performance:

• We should avoid normalizing the same strings multiple times. This is already one of the most expensive functions today
• We use BestFitting for the expression BestFit layout. Using BestFitting for strings would result in a nested BestFitting layout with exponential complexity. Can we remove the need to useBestFit` for Strings altogether?

[KotlinIsland]

Black no longer wraps implicitly concatenated strings used as func args in parens, due to this comment. There is an open issue to re-enable it: https://github.com/psf/black/issues/3955

[DetachHead]

fyi in case anyone was confused and thought this feature was removed in black version 24.1 like i did, the string processing now has to be enabled with --enable-unstable-feature=string_processing in addition to the --preview flag.

[MichaReiser]

Black considers to not move forward with the string_processing preview style (issue)

[KotlinIsland]

A simpler aspect of this larger problem is joining strings that get put onto the same line. It constantly annoys me and would love to see it resolved:

# Before
(
    ""
    ""
)
# After
("" "")
# Desired
("") 
# OR
""

• from #12057

[MichaReiser]

@KotlinIsland we agree. A solution for this is described in https://github.com/astral-sh/ruff/issues/9457