search

vercel / turbo

Incremental bundler and build system optimized for JavaScript and TypeScript, written in Rust – including Turbopack and Turborepo.
https://turbo.build
MIT License
25.54k stars 1.74k forks source link

Turborepo canary panics with 'index out of bounds: the len is 65535 but the index is 65535' #7843

Closed weyert closed 2 months ago

weyert commented 3 months ago

Verify canary release

Link to code that reproduces this issue

None

What package manager are you using / does the bug impact?

pnpm

What operating system are you using?

Mac

Which canary version will you have in your reproduction?

1.13.1-canary.2

Describe the Bug

  1   │ name = "turbo"
   2   │ operating_system = "Mac OS 14.4.0 [64-bit]"
   3   │ crate_version = "1.13.1-canary.2"
   4   │ explanation = """
   5   │ file '/Users/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/ratatui-0.26.1/src/buffer/buffer.rs' at line 108
   6   │ """
   7   │ cause = "index out of bounds: the len is 65535 but the index is 65535"
   8   │ method = "Panic"
   9   │ backtrace = """
  10   │
  11   │    0: 0x1005b74e4 - __mh_execute_header
  12   │    1: 0x100f78ab0 - __mh_execute_header
  13   │    2: 0x100f7dca8 - __mh_execute_header
  14   │    3: 0x100daf650 - __mh_execute_header
  15   │    4: 0x10080dfa4 - __mh_execute_header
  16   │    5: 0x100804bb4 - __mh_execute_header
  17   │    6: 0x1007db824 - __mh_execute_header
  18   │    7: 0x195be2f94 - __pthread_joiner_wake"""

Expected Behavior

Not to crash

To Reproduce

Not sure how to reproduce it without sharing work code

Additional context

No response

chris-olszewski commented 3 months ago

@weyert Is this happening consistently?

weyert commented 3 months ago

Hmm, that's a good question. I have downgraded to the .0 to continue with my work. I am not sure if it could because I had to instance of turbo run [command] running with both running the same steps.

It did look a bit odd with multiple 'window's of log items instead of one big one

chris-olszewski commented 3 months ago

Hmm, interesting. The panic message does look like some integer underflow so I'm doing a quick comb through of recent changes. If you have the time could you see if it happens with canary.1 or just canary.2.

Could you share what OS/terminal you were using?

weyert commented 3 months ago

I was using iTerm with Darwin Weyerts-Mac-mini.local 23.4.0 Darwin Kernel Version 23.4.0: Wed Feb 21 21:45:49 PST 2024; root:xnu-10063.101.15~2/RELEASE_ARM64_T6020 arm64

Yesterday or the day before I was getting the error below when using canary.1:

name = "turbo"
operating_system = "Mac OS 14.4.0 [64-bit]"
crate_version = "1.13.1-canary.1"
explanation = """
file '/Users/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/ratatui-0.26.1/src/terminal/terminal.rs' at line 442
"""
cause = "chunk size must be non-zero"
method = "Panic"
backtrace = """

   0: 0x10160b460 - mh_execute_header
   1: 0x101069f24 - mh_execute_header
   2: 0x101060b34 - mh_execute_header
   3: 0x1010377a4 - mh_execute_header
   4: 0x195be2f94 - __pthread_joiner_wake"""

If there is any logging or some flag you want me to use to help, let me know. I can sent you the summarize log from canary.0

chris-olszewski commented 3 months ago

Ah yes, that error from canary.1 was fixed with https://github.com/vercel/turbo/pull/7837.

The most helpful thing would be for you to use a debug build build which will have a better stack trace and has underflow checks. I wasn't able to get a build with all debug symbols that fit within the upload size limit, but turbo.tar.gz should be better than the release build.

weyert commented 3 months ago

Thanks, I will running it now. I have sent my email address to Anthony on Discord so you should be able to get it through him :)

I did notice in the latest canary and the above debug build everything looks odd compared to canary.0. Maybe it's related? This version has these weird windows on top the Task list

CleanShot 2024-03-26 at 18 13 28

weyert commented 3 months ago

Okay, I ran my command to run all my backend services through turbo which runs like ~30 build tasks before my dev tasks runs. After it all done and I start switching to the service I am currently interested in. When I try to scroll up with my mouse at this time. The following error happened:

name = "turbo"
operating_system = "Mac OS 14.4.0 [64-bit]"
crate_version = "1.13.1-canary.1"
explanation = """
file 'crates/turborepo-vt100/src/grid.rs' at line 141
"""
cause = "attempt to subtract with overflow"
method = "Panic"
backtrace = """

   0: 0x104b56d98 - __mh_execute_header
   1: 0x104b555ac - __mh_execute_header
   2: 0x104b56b58 - __mh_execute_header
   3: 0x104bdfa74 - __mh_execute_header
   4: 0x104bdfae8 - __mh_execute_header
   5: 0x103940bb4 - __mh_execute_header
   6: 0x103940dd0 - __mh_execute_header
   7: 0x103940efc - __mh_execute_header
   8: 0x10392ec2c - __mh_execute_header
   9: 0x1039332d0 - __mh_execute_header
  10: 0x1038b7b50 - __mh_execute_header
  11: 0x1038752e8 - __mh_execute_header
  12: 0x1038afc94 - __mh_execute_header
  13: 0x1038b3c2c - __mh_execute_header
  14: 0x10387d11c - __mh_execute_header
  15: 0x10387c5e4 - __mh_execute_header
  16: 0x1038889b4 - __mh_execute_header
  17: 0x10387c3a8 - __mh_execute_header
  18: 0x103880c1c - __mh_execute_header
  19: 0x1026e3a90 - __mh_execute_header
  20: 0x102919610 - __mh_execute_header
  21: 0x102a88f8c - __mh_execute_header
  22: 0x102ad1710 - __mh_execute_header
  23: 0x102b3dfc0 - __mh_execute_header
  24: 0x1029c3608 - __mh_execute_header
  25: 0x102bd3bd8 - __mh_execute_header
  26: 0x102bfe020 - __mh_execute_header
  27: 0x102ba5508 - __mh_execute_header
  28: 0x102c35938 - __mh_execute_header
  29: 0x102b29bc4 - __mh_execute_header
  30: 0x102b7e138 - __mh_execute_header
  31: 0x102b7e3c8 - __mh_execute_header
  32: 0x102c91338 - __mh_execute_header
  33: 0x1044d4f60 - __mh_execute_header
  34: 0x10449b79c - __mh_execute_header
  35: 0x10451be14 - __mh_execute_header
  36: 0x10451e1ec - __mh_execute_header
  37: 0x10451dfa0 - __mh_execute_header
  38: 0x1044a7770 - __mh_execute_header
  39: 0x1044de924 - __mh_execute_header
  40: 0x104498d3c - __mh_execute_header
  41: 0x1044d72b4 - __mh_execute_header
  42: 0x1044d9740 - __mh_execute_header
  43: 0x1044d6640 - __mh_execute_header
  44: 0x1044de7a8 - __mh_execute_header
  45: 0x1044c1df8 - __mh_execute_header
  46: 0x104b5c498 - __mh_execute_header
  47: 0x195be2f94 - __pthread_joiner_wake"""
weyert commented 3 months ago

Tried the latest canary build and still panics:

name = "turbo"
operating_system = "Mac OS 14.4.0 [64-bit]"
crate_version = "1.13.1-canary.3"
explanation = """
file 'crates/turborepo-lib/src/task_graph/visitor.rs' at line 323
"""
cause = "render thread panicked: JoinError::Panic(Id(97), ...)"
method = "Panic"
backtrace = """

   0: 0x104e2ad94 - __mh_execute_header
   1: 0x104b92a80 - __mh_execute_header
   2: 0x104b75b80 - __mh_execute_header
   3: 0x104d59218 - __mh_execute_header
   4: 0x104f1ac30 - __mh_execute_header
   5: 0x104c7c334 - __mh_execute_header
   6: 0x104d0e214 - __mh_execute_header
   7: 0x104d0a774 - __mh_execute_header
   8: 0x104a406e8 - __mh_execute_header
   9: 0x104a3fba0 - __mh_execute_header
  10: 0x104a3f6bc - __mh_execute_header
  11: 0x104a40540 - __mh_execute_header"""

Feel free to mail me a debug build and I will try it out tomorrow :)

chris-olszewski commented 3 months ago

Syncing to the public thread, but I believe I've identified the underlying cause.

The library we use to render the UI to terminal clamps the area to 2^16 - 1 (https://github.com/ratatui-org/ratatui/blob/main/src/layout/rect.rs#L75), even when the width height exceeds that. This gets triggered by having a task that produces n lines of output in a terminal with m width where n m > 2^16 - 1.

I'll get a work around where we truncate logs to fit this constraint up soon while I work on fixing the upstream issue.

weyert commented 2 months ago

Any plans to release a canary with the fix?

chris-olszewski commented 2 months ago

Sorry, needed to fix up the release process. 1.13.3-canary.1 is now out with this fix.

chris-olszewski commented 2 months ago

Closing as this has been released for awhile. Please reopen if this is still being encountered.

joshka commented 2 months ago

Syncing to the public thread, but I believe I've identified the underlying cause.

The library we use to render the UI to terminal clamps the area to 2^16 - 1 (https://github.com/ratatui-org/ratatui/blob/main/src/layout/rect.rs#L75), even when the width height exceeds that. This gets triggered by having a task that produces n lines of output in a terminal with m width where n m > 2^16 - 1.

I'll get a work around where we truncate logs to fit this constraint up soon while I work on fixing the upstream issue.

(Ratatui maintainer here). We inherited this u16 clamp on total area from tui-rs, and I'm not sure anyone clearly knows why this is how it is like this. If this is something that you can help us understand why it causes your app issues, then we can definitely look into it more (as it seems like a weird thing to do to clamp as u16).

chris-olszewski commented 2 months ago

@joshka We were misusing/abusing ratatui by trying render a widget that was far larger than the actual terminal by using insert_before. We're switching to rendering to the alternative screen and then using standard writes to stdout for persisting this output.

I'm not sure if it makes sense for any action to be taken by ratatui regarding this. A panic/error upon Rect construction or area() when the area exceeds u16 would have made this slightly easier to debug instead of hitting an out of bounds panic down the line.