Deadlock in output redirection when printing large amount of text to stderr

[karliss]

Environment information

• Operating System: Linux
• Cutter version: dd7e01be3a4d21d4e788ec534f6ffd125372bdb4
• Obtained from: Built from source
• File format: ELF

Describe the bug

When trying to load large binary (Blender 70MB) UI freezes.

To Reproduce

Steps to reproduce the behavior:

1. Select Linux version of Blender as executable
2. Disable analysis
3. Click open
4. Without ELF symbol loading fix initial part will take 15min, this issue isn't about that.
5. After a while timer and UI will freeze

Expected behavior File opening doesn't get stuck

Screenshots

Additional context

Stacktraces

GUI thread

1   __lll_lock_wait_private                                                                                                                                                                                                0x7ffff4bd199a 
2   fwrite                                                                                                                                                                                                                 0x7ffff4bc21ce 
3   r_bin_file_compute_hashes                                                                                                                                                                bfile.c                  873  0x7ffff6278817 
4   cmd_info                                                                                                                                                                                 cmd_info.c               612  0x7ffff71b5ed0 
5   r_cmd_call                                                                                                                                                                               cmd_api.c                248  0x7ffff71fc91e 
6   r_core_cmd_subst_i                                                                                                                                                                       cmd.c                    3624 0x7ffff71f7196 
7   r_core_cmd_subst                                                                                                                                                                         cmd.c                    2585 0x7ffff71f350c 
8   run_cmd_depth                                                                                                                                                                            cmd.c                    5558 0x7ffff71f9cd4 
9   r_core_cmd                                                                                                                                                                               cmd.c                    5640 0x7ffff71fa08a 
10  r_core_cmd_str                                                                                                                                                                           cmd.c                    5880 0x7ffff71faa75 
11  CutterCore::cmdj                                                                                                                                                                         Cutter.cpp               405  0x55555579f7ad 
12  Dashboard::updateContents                                                                                                                                                                Dashboard.cpp            78   0x55555585cf7f 
13  Dashboard::qt_static_metacall                                                                                                                                                            moc_Dashboard.cpp        80   0x55555578677e 
14  ??                                                                                                                                                                                                                     0x7ffff53364a0 
15  CutterCore::refreshAll                                                                                                                                                                   moc_Cutter.cpp           463  0x55555577e6bb 
16  CutterCore::triggerRefreshAll                                                                                                                                                            Cutter.cpp               910  0x5555557a3280 
17  MainWindow::refreshAll                                                                                                                                                                   MainWindow.cpp           779  0x55555581c884 
18  MainWindow::finalizeOpen                                                                                                                                                                 MainWindow.cpp           542  0x55555581a453 
19  InitialOptionsDialog::<lambda()>::operator()(void) const                                                                                                                                 InitialOptionsDialog.cpp 287  0x5555557f8905 
20  QtPrivate::FunctorCall<QtPrivate::IndexesList<>, QtPrivate::List<>, void, InitialOptionsDialog::setupAndStartAnalysis()::<lambda()>>::call(InitialOptionsDialog::<lambda()> &, void * *) qobjectdefs_impl.h       146  0x5555557fa3eb 
... <More>                                                                                                                                                                                                                                

Thread for string update task

1  write                                            0x7ffff4c3c59f 
2  _IO_file_write * *GLIBC_2.2.5                    0x7ffff4bcd24d 
3  new_do_write                                     0x7ffff4bcc5a6 
4  __GI__IO_file_xsputn                             0x7ffff4bcd97e 
5  __vfprintf_internal                              0x7ffff4bb6ba2 
6  fprintf                                          0x7ffff4ba31fa 
7  CutterCore::parseJson         Cutter.cpp    448  0x55555579fb3f 
8  CutterCore::parseJson         Cutter.h      106  0x5555557c0e11 
9  CutterCore::cmdjTask          Cutter.cpp    427  0x55555579f963 
10 CutterCore::getAllStrings     Cutter.cpp    2625 0x5555557b148e 
11 StringsTask::runTask          StringsTask.h 21   0x55555578f0e5 
12 AsyncTask::run                AsyncTask.cpp 52   0x55555590a8fc 
13 ??                                               0x7ffff51281f2 
14 ??                                               0x7ffff5124fc6 
15 start_thread                                     0x7ffff4b1746f 
16 clone                                            0x7ffff4c4b3d3 

Here is my theory for what happens.

• string update task starts writing error message. The JSON parser error is "QJsonParseError::DocumentTooLarge (13)" so trying to print the whole JSON as error message probably takes some time.
• Gui thread tries to prints it's own message but is blocked because string update task is still writing
• Since the GUI thread is blocked, buffer for redirected output isn't emptied
• String update task can't finish printing error message because buffer is full and task processing it is blocked
• deadlock

I haven't looked at how the stdout stderr redirection code actually works so my theory might be wrong.

[karliss]

Two error messages causing deadlock means that in addition to deadlock there are two errors or warning. In this case

• Warning: r_bin_file_hash: file exceeds bin.hashlimit , not sure if this is a problem
• Qt JSON parser failing with "QJsonParseError::DocumentTooLarge (13)". How big the JSON actaully is? Is it really really too big, or it's a configurable soft limit.
• In ideal case cutter wouldn't ask r2 for queries so big JSON parser fails but only the data it currently needs. Use some kind of iterator API from r2. Most likely requires nontrivial changes on r2 and Cutter side.

[karliss]

After disabling output redirection UI didn't froze.

[msmshazan]

Similar issue when loading a 100mb nds rom

[ITAYC0HEN]

had the same with printing big amount of info using qDebug()

[XVilka]

Much of the related code in Rizin and Cutter has changed since. Even while opening huge files it's not freezes (at least for me). If you still experience this kind of issue - please open a new one.