search

universal-ctags / ctags

A maintained ctags implementation
https://ctags.io
GNU General Public License v2.0
6.52k stars 621 forks source link

cxx: incorrect huge typeref values #1120

Open techee opened 8 years ago

techee commented 8 years ago

It looks the compiler can get confused and here

https://github.com/universal-ctags/ctags/blob/master/parsers/cxx/cxx_tag.c#L323

it can generate tokens with huge strings. For instance, for

https://github.com/torvalds/linux/blob/master/lib/locking-selftest.c

I get typeref in one tag like

reset_locks lib/locking-selftest.c  /^static void reset_locks(void)$/;" f   typeref:typename:GENERATE_TESTCASE (ABBA_spin)GENERATE_TESTCASE (ABBA_wlock)GENERATE_TESTCASE (ABBA_rlock)GENERATE_TESTCASE (ABBA_mutex)GENERATE_TESTCASE (ABBA_wsem)GENERATE_TESTCASE (ABBA_rsem)GENERATE_TESTCASE (ABBCCA_spin)GENERATE_TESTCASE (ABBCCA_wlock)GENERATE_TESTCASE (ABBCCA_rlock)GENERATE_TESTCASE (ABBCCA_mutex)GENERATE_TESTCASE (ABBCCA_wsem)GENERATE_TESTCASE (ABBCCA_rsem)GENERATE_TESTCASE (ABCABC_spin)GENERATE_TESTCASE (ABCABC_wlock)GENERATE_TESTCASE (ABCABC_rlock)GENERATE_TESTCASE (ABCABC_mutex)GENERATE_TESTCASE (ABCABC_wsem)GENERATE_TESTCASE (ABCABC_rsem)GENERATE_TESTCASE (ABBCCDDA_spin)GENERATE_TESTCASE (ABBCCDDA_wlock)GENERATE_TESTCASE (ABBCCDDA_rlock)GENERATE_TESTCASE (ABBCCDDA_mutex)GENERATE_TESTCASE (ABBCCDDA_wsem)GENERATE_TESTCASE (ABBCCDDA_rsem)GENERATE_TESTCASE (ABCDBDDA_spin)GENERATE_TESTCASE (ABCDBDDA_wlock)GENERATE_TESTCASE (ABCDBDDA_rlock)GENERATE_TESTCASE (ABCDBDDA_mutex)GENERATE_TESTCASE (ABCDBDDA_wsem)GENERATE_TESTCASE (ABCDBDDA_rsem)GENERATE_TESTCASE (ABCDBCDA_spin)GENERATE_TESTCASE (ABCDBCDA_wlock)GENERATE_TESTCASE (ABCDBCDA_rlock)GENERATE_TESTCASE (ABCDBCDA_mutex)GENERATE_TESTCASE (ABCDBCDA_wsem)GENERATE_TESTCASE (ABCDBCDA_rsem)GENERATE_TESTCASE (double_unlock_spin)GENERATE_TESTCASE (double_unlock_wlock)GENERATE_TESTCASE (double_unlock_rlock)GENERATE_TESTCASE (double_unlock_mutex)GENERATE_TESTCASE (double_unlock_wsem)GENERATE_TESTCASE (double_unlock_rsem)GENERATE_TESTCASE (bad_unlock_order_spin)GENERATE_TESTCASE (bad_unlock_order_wlock)GENERATE_TESTCASE (bad_unlock_order_rlock)GENERATE_TESTCASE (bad_unlock_order_mutex)GENERATE_TESTCASE (bad_unlock_order_wsem)GENERATE_TESTCASE (bad_unlock_order_rsem)GENERATE_TESTCASE (init_held_spin)GENERATE_TESTCASE (init_held_wlock)GENERATE_TESTCASE (init_held_rlock)GENERATE_TESTCASE (init_held_mutex)GENERATE_TESTCASE (init_held_wsem)GENERATE_TESTCASE (init_held_rsem)GENERATE_PERMUTATIONS_2_EVENTS (irqsafe1_hard_spin)GENERATE_PERMUTATIONS_2_EVENTS (irqsafe1_hard_rlock)GENERATE_PERMUTATIONS_2_EVENTS (irqsafe1_hard_wlock)GENERATE_PERMUTATIONS_2_EVENTS (irqsafe1_soft_spin)GENERATE_PERMUTATIONS_2_EVENTS (irqsafe1_soft_rlock)GENERATE_PERMUTATIONS_2_EVENTS (irqsafe1_soft_wlock)GENERATE_PERMUTATIONS_2_EVENTS (irqsafe2A_spin)GENERATE_PERMUTATIONS_2_EVENTS (irqsafe2A_wlock)GENERATE_PERMUTATIONS_2_EVENTS (irqsafe2A_rlock)GENERATE_PERMUTATIONS_2_EVENTS (irqsafe2B_hard_spin)GENERATE_PERMUTATIONS_2_EVENTS (irqsafe2B_hard_rlock)GENERATE_PERMUTATIONS_2_EVENTS (irqsafe2B_hard_wlock)GENERATE_PERMUTATIONS_2_EVENTS (irqsafe2B_soft_spin)GENERATE_PERMUTATIONS_2_EVENTS (irqsafe2B_soft_rlock)GENERATE_PERMUTATIONS_2_EVENTS (irqsafe2B_soft_wlock)GENERATE_PERMUTATIONS_3_EVENTS (irqsafe3_hard_spin)GENERATE_PERMUTATIONS_3_EVENTS (irqsafe3_hard_rlock)GENERATE_PERMUTATIONS_3_EVENTS (irqsafe3_hard_wlock)GENERATE_PERMUTATIONS_3_EVENTS (irqsafe3_soft_spin)GENERATE_PERMUTATIONS_3_EVENTS (irqsafe3_soft_rlock)GENERATE_PERMUTATIONS_3_EVENTS (irqsafe3_soft_wlock)GENERATE_PERMUTATIONS_3_EVENTS (irqsafe4_hard_spin)GENERATE_PERMUTATIONS_3_EVENTS (irqsafe4_hard_rlock)GENERATE_PERMUTATIONS_3_EVENTS (irqsafe4_hard_wlock)GENERATE_PERMUTATIONS_3_EVENTS (irqsafe4_soft_spin)GENERATE_PERMUTATIONS_3_EVENTS (irqsafe4_soft_rlock)GENERATE_PERMUTATIONS_3_EVENTS (irqsafe4_soft_wlock)GENERATE_PERMUTATIONS_3_EVENTS (irq_inversion_hard_spin)GENERATE_PERMUTATIONS_3_EVENTS (irq_inversion_hard_rlock)GENERATE_PERMUTATIONS_3_EVENTS (irq_inversion_hard_wlock)GENERATE_PERMUTATIONS_3_EVENTS (irq_inversion_soft_spin)GENERATE_PERMUTATIONS_3_EVENTS (irq_inversion_soft_rlock)GENERATE_PERMUTATIONS_3_EVENTS (irq_inversion_soft_wlock)GENERATE_PERMUTATIONS_3_EVENTS (irq_read_recursion_hard)GENERATE_PERMUTATIONS_3_EVENTS (irq_read_recursion_soft)void file:

Other good files to try are these from the boost library:

boost/phoenix/core/preprocessed/actor_50.hpp
boost/phoenix/core/preprocessed/actor_30.hpp
boost/phoenix/core/preprocessed/actor_20.hpp
boost/phoenix/core/preprocessed/actor_40.hpp
boost/phoenix/scope/preprocessed/lambda_50.hpp
boost/phoenix/scope/preprocessed/lambda_50.hpp
boost/phoenix/scope/preprocessed/lambda_50.hpp

Originally reported in #1111

pragmaware commented 8 years ago

Heh... the problem here is that the file uses the preprocessor to generate statements and we don't have a real preprocessor. All I can do, probably, is refuse to emit types that contain too many tokens.

Another thing we could attempt in the future would be to apply some preprocessing: capture macro names and either expand them properly (hard) or just replace them with a space (simple). It would probably fix this case. Obviously other cases with macros defined in different files would be still hopeless.

techee commented 8 years ago

Heh... the problem here is that the file uses the preprocessor to generate statements and we don't have a real preprocessor. All I can do, probably, is refuse to emit types that contain too many tokens.

The problem is that macros are skipped completely so here the cxx compiler thinks that GENERATE_PERMUTATIONS_3_EVENTS is immediately before reset_locks which isn't the case. So IMO what the compiler should do is that the token chain that is used for typename should be cut off at the nearest macro definition and not to contain anything before it.

Another thing we could attempt in the future would be to apply some preprocessing: capture macro names and either expand them properly (hard) or just replace them with a space (simple). It would probably fix this case. Obviously other cases with macros defined in different files would be still hopeless.

Trying to do macro expansion doesn't probably make sense indeed as they can be stored in another file.

masatake commented 8 years ago

See https://github.com/universal-ctags/ctags/issues/80. We can give tags file as an input for ctags.

masatake commented 8 years ago

Is -I option helpful?

pragmaware commented 8 years ago

1123 fixes this particular instance of bug. However a general fix for macros interference would be nice... it's just that I have no idea on how to do it.

masatake commented 8 years ago

Do you agree with this milestone setting?