ClexPass::rm-toks-1 has encountered a bug: pass got stuck state: 50000

[zephyrus00jp]

Hi, I am new to cvise. I filed a GCC compile-time issue and wanted to provide a reduced case. Manual reduction was impossible for me. So I turned to cvise which. I had learned, was used for people to reduce test cases. GCC bugzilla: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=109041

Although the bug seen in gcc-11 is fixed in gcc-12 and the bugzilla is closed there, the cvise bug I am reporting seems to be real.

Since I am new to cvise, I may be doing something wacky, but since cvise reported an error and prompted me to file a bug. Here it is.

--- begin quote of PASS_BUG_INFO.TXT Package: cvise 2.7.0 Git version: d9e4a505 LLVM version: 15.0.6 System: uname_result(system='Linux', node='ip030', release='6.0.0-2-amd64', version='#1 SMP PREEMPT_DYNAMIC Debian 6.0.5-1 (2022-10-28)', machine='x86_64')

---

ClexPass::rm-toks-1 has encountered a bug: pass got stuck state: 50000

Please consider tarring up cvise_bug_1 and creating an issue at https://github.com/marxin/cvise/issues and we will try to fix the bug.

---

--- end quote ---

Strange thing, though, is that sqlie3-preprocessed-2.i included in the tar seems to contain only the function prototype declaration/typedefs/variable declaration from my original preprocessed file and does not include the function(s) where compile time misguided warning was printed. But that might be the intention of the cvise bug report.

Just for your reference, I am attaching the ORIGINAL preprocessed file in compressed format, too.

Thank you for making this wonderful tool available. It seems to perform wonderfully by seasoned compiler maintainers.

cvise_bug_1.tar.gz

sqlite3-preprocessed-2.i.orig.gz

[zephyrus00jp]

To my surprise, while I was attempting to report the first bug report, cvise created another report. So far, the message printed to the invoking tty console under linux is as follows.

cvise t-test.sh sqlite3-preprocessed-2.i 
00:00:11 INFO ===< 3936773 >===
00:00:11 INFO running 7 interestingness tests in parallel
00:00:11 INFO INITIAL PASSES
00:00:11 INFO ===< IncludesPass >===
00:00:11 INFO ===< UnIfDefPass >===
00:00:12 INFO ===< CommentsPass >===
00:00:12 INFO ===< IfPass >===
00:00:12 INFO ===< LineMarkersPass >===
00:00:12 INFO ===< BlankPass >===
00:00:12 INFO ===< ClangBinarySearchPass::replace-function-def-with-decl >===
00:00:17 INFO using C++ standard: c++2b with 2981 transformation opportunities
00:00:33 INFO (84.1%, 704353 bytes, 19849 lines)
00:00:33 INFO ===< ClangBinarySearchPass::remove-unused-function >===
00:00:33 INFO using C++ standard: c++2b with 3841 transformation opportunities
00:37:05 INFO ===< LinesPass::0 >===
04:21:06 INFO ===< LinesPass::1 >===
04:21:07 WARNING Skipping pass as sanity check fails for topformflat output
04:21:07 INFO ===< LinesPass::2 >===
04:21:09 WARNING Skipping pass as sanity check fails for topformflat output
04:21:09 INFO ===< LinesPass::3 >===
04:21:10 WARNING Skipping pass as sanity check fails for topformflat output
04:21:10 INFO ===< LinesPass::4 >===
04:21:12 WARNING Skipping pass as sanity check fails for topformflat output
04:21:12 INFO ===< LinesPass::6 >===
04:21:13 WARNING Skipping pass as sanity check fails for topformflat output
04:21:13 INFO ===< LinesPass::8 >===
04:21:15 WARNING Skipping pass as sanity check fails for topformflat output
04:21:15 INFO ===< LinesPass::10 >===
04:21:16 WARNING Skipping pass as sanity check fails for topformflat output
04:21:16 INFO ===< ClangBinarySearchPass::replace-function-def-with-decl >===
04:21:17 INFO using C++ standard: c++2b with 0 transformation opportunities
04:21:17 INFO ===< ClangBinarySearchPass::remove-unused-function >===
04:21:17 INFO cache hit for /tmp/sqlite3-preprocessed-2.i
04:21:17 INFO ===< ClangPass::remove-unused-function >===
04:44:33 INFO ===< BalancedPass::curly >===
04:48:04 INFO ===< BalancedPass::curly2 >===
04:51:34 INFO ===< BalancedPass::curly3 >===
04:51:58 INFO ===< BalancedPass::parens-to-zero >===
06:03:40 INFO ===< ClangPass::callexpr-to-value >===
06:03:40 INFO ===< ClangPass::replace-callexpr >===
06:03:41 INFO ===< ClangPass::simplify-callexpr >===
06:03:41 INFO ===< ClangBinarySearchPass::remove-unused-enum-member >===
06:03:41 INFO using C++ standard: c++2b with 371 transformation opportunities
06:07:57 INFO ===< ClangPass::remove-enum-member-value >===
06:08:15 INFO ===< ClangBinarySearchPass::remove-unused-var >===
06:08:16 INFO using C++ standard: c++2b with 150 transformation opportunities
06:10:05 INFO ===< SpecialPass::a >===
06:10:05 INFO ===< SpecialPass::b >===
06:10:05 INFO ===< SpecialPass::c >===
06:10:05 INFO MAIN PASSES
06:10:05 INFO ===< IncludeIncludesPass >===
06:10:05 INFO ===< TernaryPass::b >===
06:10:10 INFO ===< TernaryPass::c >===
06:10:15 INFO ===< BalancedPass::curly >===
06:10:15 INFO cache hit for /tmp/sqlite3-preprocessed-2.i
06:10:15 INFO ===< BalancedPass::curly2 >===
06:10:15 INFO cache hit for /tmp/sqlite3-preprocessed-2.i
06:10:15 INFO ===< BalancedPass::curly3 >===
06:10:15 INFO cache hit for /tmp/sqlite3-preprocessed-2.i
06:10:15 INFO ===< BalancedPass::parens >===
07:22:06 INFO ===< BalancedPass::angles >===
07:22:17 INFO ===< BalancedPass::square >===
07:23:25 INFO ===< BalancedPass::curly-inside >===
07:26:55 INFO ===< BalancedPass::parens-inside >===
08:38:13 INFO ===< BalancedPass::angles-inside >===
08:38:24 INFO ===< BalancedPass::square-inside >===
08:39:30 INFO ===< BalancedPass::curly-only >===
08:42:53 INFO ===< BalancedPass::angles-only >===
08:43:04 INFO ===< BalancedPass::square-only >===
08:44:10 INFO ===< ClangPass::remove-namespace >===
08:44:10 INFO ===< ClangPass::aggregate-to-scalar >===
08:44:13 INFO ===< ClangPass::param-to-global >===
09:22:09 INFO ===< ClangPass::param-to-local >===
10:01:05 INFO ===< ClangPass::remove-nested-function >===
10:01:06 INFO ===< ClangPass::union-to-struct >===
10:01:21 INFO ===< ClangPass::return-void >===
10:20:26 INFO ===< ClangPass::simple-inliner >===
10:20:27 INFO ===< ClangPass::reduce-pointer-level >===
10:27:31 INFO ===< ClangPass::lift-assignment-expr >===
10:27:31 INFO ===< ClangPass::copy-propagation >===
10:27:31 INFO ===< ClangPass::callexpr-to-value >===
10:27:31 INFO cache hit for /tmp/sqlite3-preprocessed-2.i
10:27:31 INFO ===< ClangPass::replace-callexpr >===
10:27:31 INFO cache hit for /tmp/sqlite3-preprocessed-2.i
10:27:31 INFO ===< ClangPass::simplify-callexpr >===
10:27:31 INFO cache hit for /tmp/sqlite3-preprocessed-2.i
10:27:31 INFO ===< ClangPass::remove-unused-function >===
10:27:31 INFO cache hit for /tmp/sqlite3-preprocessed-2.i
10:27:31 INFO ===< ClangBinarySearchPass::remove-unused-enum-member >===
10:27:31 INFO cache hit for /tmp/sqlite3-preprocessed-2.i
10:27:31 INFO ===< ClangPass::remove-enum-member-value >===
10:27:31 INFO cache hit for /tmp/sqlite3-preprocessed-2.i
10:27:31 INFO ===< ClangBinarySearchPass::remove-unused-var >===
10:27:31 INFO cache hit for /tmp/sqlite3-preprocessed-2.i
10:27:31 INFO ===< ClangPass::simplify-if >===
10:27:32 INFO ===< ClangPass::reduce-array-dim >===
10:27:32 INFO ===< ClangPass::reduce-array-size >===
10:27:55 INFO ===< ClangPass::move-definition-to-declaration >===
10:29:01 INFO ===< ClangPass::simplify-comma-expr >===
10:29:01 INFO ===< ClangPass::simplify-dependent-typedef >===
10:29:02 INFO ===< ClangPass::replace-simple-typedef >===
10:30:18 INFO ===< ClangPass::replace-dependent-typedef >===
10:30:19 INFO ===< ClangPass::replace-one-level-typedef-type >===
10:30:44 INFO ===< ClangPass::remove-unused-field >===
10:47:46 INFO ===< ClangPass::instantiate-template-type-param-to-int >===
10:47:46 INFO ===< ClangPass::instantiate-template-param >===
10:47:46 INFO ===< ClangPass::template-arg-to-int >===
10:47:47 INFO ===< ClangPass::template-non-type-arg-to-int >===
10:47:47 INFO ===< ClangPass::reduce-class-template-param >===
10:47:47 INFO ===< ClangPass::remove-trivial-base-template >===
10:47:47 INFO ===< ClangPass::class-template-to-class >===
10:47:48 INFO ===< ClangPass::merge-base-class >===
10:47:48 INFO ===< ClangPass::remove-base-class >===
10:47:48 INFO ===< ClangPass::replace-derived-class >===
10:47:48 INFO ===< ClangPass::remove-unresolved-base >===
10:47:49 INFO ===< ClangPass::remove-ctor-initializer >===
10:47:49 INFO ===< ClangPass::replace-class-with-base-template-spec >===
10:47:49 INFO ===< ClangPass::simplify-nested-class >===
10:47:49 INFO ===< ClangPass::remove-unused-outer-class >===
10:47:50 INFO ===< ClangPass::empty-struct-to-int >===
10:48:02 INFO ===< ClangPass::remove-pointer >===
10:48:10 INFO ===< ClangPass::reduce-pointer-pairs >===
10:48:10 INFO ===< ClangPass::remove-array >===
10:48:12 INFO ===< ClangPass::remove-addr-taken >===
10:48:13 INFO ===< ClangPass::simplify-struct >===
10:48:15 INFO ===< ClangPass::replace-undefined-function >===
11:07:06 INFO ===< ClangPass::replace-array-index-var >===
11:07:06 INFO ===< ClangPass::replace-array-access-with-index >===
11:07:09 INFO ===< ClangPass::replace-dependent-name >===
11:07:09 INFO ===< ClangPass::simplify-recursive-template-instantiation >===
11:07:09 INFO ===< ClangPass::vector-to-array >===
11:07:09 INFO ===< ClangPass::remove-try-catch >===
11:07:10 INFO ===< ClangPass::class-to-struct >===
11:07:10 INFO ===< ClangPass::member-to-global >===
11:24:52 INFO ===< LinesPass::0 >===
11:24:52 INFO cache hit for /tmp/sqlite3-preprocessed-2.i
11:24:52 INFO ===< LinesPass::1 >===
11:24:52 INFO cache hit for /tmp/sqlite3-preprocessed-2.i
11:24:52 INFO ===< LinesPass::2 >===
11:24:52 INFO cache hit for /tmp/sqlite3-preprocessed-2.i
11:24:52 INFO ===< LinesPass::3 >===
11:24:52 INFO cache hit for /tmp/sqlite3-preprocessed-2.i
11:24:52 INFO ===< LinesPass::4 >===
11:24:52 INFO cache hit for /tmp/sqlite3-preprocessed-2.i
11:24:52 INFO ===< LinesPass::6 >===
11:24:52 INFO cache hit for /tmp/sqlite3-preprocessed-2.i
11:24:52 INFO ===< LinesPass::8 >===
11:24:52 INFO cache hit for /tmp/sqlite3-preprocessed-2.i
11:24:52 INFO ===< LinesPass::10 >===
11:24:52 INFO cache hit for /tmp/sqlite3-preprocessed-2.i
11:24:52 INFO ===< UnIfDefPass >===
11:24:52 INFO ===< CommentsPass >===
11:24:52 INFO ===< SpecialPass::b >===
11:24:52 INFO cache hit for /tmp/sqlite3-preprocessed-2.i
11:24:52 INFO ===< SpecialPass::c >===
11:24:52 INFO cache hit for /tmp/sqlite3-preprocessed-2.i
11:24:52 INFO ===< IndentPass::regular >===
11:24:55 INFO ===< BalancedPass::parens-to-zero >===
11:24:55 INFO cache hit for /tmp/sqlite3-preprocessed-2.i
11:24:55 INFO ===< ClexPass::rm-toks-1 >===
16:15:46 WARNING ClexPass::rm-toks-1 has encountered a non fatal bug: pass got stuck
16:16:26 INFO ===< ClexPass::rm-toks-2 >===
21:04:27 WARNING ClexPass::rm-toks-2 has encountered a non fatal bug: pass got stuck
21:05:06 INFO ===< ClexPass::rm-toks-3 >===

cvise_bug_2.tar.gz

The attached tar.gz is fro the second error. WARNING ClexPass::rm-toks-2 has encountered a non fatal bug: pass got stuck I am afraid that I am simply feeding too large a source code?

EDIT: quoted log from the console had unintended previous history and duplicates. Fixed.

[marxin]

I am afraid that I am simply feeding too large a source code?

Thank you for the report. Yes, that's the simple answer for your problem. Note the later should not end up processing source file close to 1MiB, that's not expected.

[marxin]

Strange thing, though, is that sqlie3-preprocessed-2.i included in the tar seems to contain only the function prototype declaration/typedefs/variable declaration from my original preprocessed file and does not include the function(s) where compile time misguided warning was printed.

Well, are you sure the warning writing 4 bytes into a isn't emitted from a different function? That's pretty common case and that makes the reduction of warnings quite difficult.

[zephyrus00jp]

Strange thing, though, is that sqlie3-preprocessed-2.i included in the tar seems to contain only the function prototype declaration/typedefs/variable declaration from my original preprocessed file and does not include the function(s) where compile time misguided warning was printed.

Well, are you sure the warning writing 4 bytes into a isn't emitted from a different function? That's pretty common case and that makes the reduction of warnings quite difficult.

Sorry for my tardy reply. I got tied up with my day job. It seems that gcc developers have already fixed the issue in the development source line, so this issue is moot.

However, for a different issue, I could reduce a whopping 160K lines of code after four days to produce a smallish reduced case and GCC developers figured out what patch caused the regression already. https://github.com/marxin/cvise/issues/109#issuecomment-1504770612 So my skill is improving a bit.

Thank you for sharing a very useful tool with the developer community at large.

[marxin]

However, for a different issue, I could reduce a whopping 160K lines of code after four days to produce a smallish reduced case and GCC developers figured out what patch caused the regression already. [#109 (comment)]

Good! Note it's better to announce test-case size in MiB (of a pre-processed source file), it's telling more than 160k LOC, which is very likely to be a huge test-case.

Thank you for sharing a very useful tool with the developer community at large.

You're welcome and I'm grateful for any feedback from users of the tool!