RecursionError when Self-Referential Macros

[MatthewShao]

I met a RecursionError when running this project on pcre2

git clone https://github.com/PCRE2Project/pcre2 pcre
pcpp ./pcre/src/pcre2_context.c -I ./pcre/ -I /usr/include   

An fatal error occurs:

*** EXPAND MACROS in PCRE2_SPTR expanding_from= []
*** EXPAND MACROS in PCRE2_SUFFIX(PCRE2_SPTR) expanding_from= ['PCRE2_SPTR']
*** EXPAND MACROS in PCRE2_SPTR expanding_from= []
*** EXPAND MACROS in PCRE2_SUFFIX(PCRE2_SPTR) expanding_from= ['PCRE2_SPTR']
*** EXPAND MACROS in PCRE2_SPTR expanding_from= []
*** EXPAND MACROS in PCRE2_SUFFIX(PCRE2_SPTR) expanding_from= ['PCRE2_SPTR']
*** EXPAND MACROS in PCRE2_SPTR expanding_from= []
*** EXPAND MACROS in PCRE2_SUFFIX(PCRE2_SPTR) expanding_from= ['PCRE2_SPTR']
*** EXPAND MACROS in PCRE2_SPTR expanding_from= []
*** EXPAND MACROS in PCRE2_SUFFIX(PCRE2_SPTR) expanding_from= ['PCRE2_SPTR']
*** EXPAND MACROS in PCRE2_SPTR expanding_from= []
Traceback (most recent call last):
  File "/root/miniconda3/envs/arch/lib/python3.10/site-packages/pcpp/pcmd.py", line 140, in __init__
    self.write(self.args.output)
  File "/root/miniconda3/envs/arch/lib/python3.10/site-packages/pcpp/preprocessor.py", line 1322, in write
    tok = self.token()
  File "/root/miniconda3/envs/arch/lib/python3.10/site-packages/pcpp/preprocessor.py", line 1303, in token
    tok = next(self.parser)
  File "/root/miniconda3/envs/arch/lib/python3.10/site-packages/pcpp/preprocessor.py", line 895, in parsegen
    for tok in self.include(args, x):
  File "/root/miniconda3/envs/arch/lib/python3.10/site-packages/pcpp/preprocessor.py", line 1167, in include
    for tok in self.parsegen(data,filename,fulliname):
  File "/root/miniconda3/envs/arch/lib/python3.10/site-packages/pcpp/preprocessor.py", line 895, in parsegen
    for tok in self.include(args, x):
  File "/root/miniconda3/envs/arch/lib/python3.10/site-packages/pcpp/preprocessor.py", line 1167, in include
    for tok in self.parsegen(data,filename,fulliname):
  File "/root/miniconda3/envs/arch/lib/python3.10/site-packages/pcpp/preprocessor.py", line 903, in parsegen
    for tok in self.expand_macros(chunk):
  File "/root/miniconda3/envs/arch/lib/python3.10/site-packages/pcpp/preprocessor.py", line 580, in expand_macros
    ex = self.expand_macros(rep, expanding_from + [t.value])
  File "/root/miniconda3/envs/arch/lib/python3.10/site-packages/pcpp/preprocessor.py", line 580, in expand_macros
    ex = self.expand_macros(rep, expanding_from + [t.value])
RecursionError: maximum recursion depth exceeded while calling a Python object
None

INTERNAL PREPROCESSOR ERROR AT AROUND ../../../usr/include/pcre2.h:917, FATALLY EXITING NOW

I tracked the code and it was caused by : https://github.com/PCRE2Project/pcre2/blob/master/src/pcre2.h.in#L806, the self referential macros was used.

I made a quick fix in expand_macros to prevent infante recursion:

    def expand_macros(self,tokens,expanding_from=[]):
        """Given a list of tokens, this function performs macro expansion."""
        # Each token needs to track from which macros it has been expanded from to prevent recursion

        for tok in tokens:
            if not hasattr(tok, 'expanded_from'):
                tok.expanded_from = []
            if len(expanding_from) == 1 and tok.value == expanding_from[0]:
                return tokens
        i = 0

Not sure whether this is the correct, hope this issue can be addressed on master.

[ned14]

Condensed repro:

typedef uint8_t  PCRE2_UCHAR8;
typedef uint16_t PCRE2_UCHAR16;
typedef uint32_t PCRE2_UCHAR32;

typedef const PCRE2_UCHAR8  *PCRE2_SPTR8;
typedef const PCRE2_UCHAR16 *PCRE2_SPTR16;
typedef const PCRE2_UCHAR32 *PCRE2_SPTR32;

#define PCRE2_SIZE            size_t

#define PCRE2_COMPILE_FUNCTIONS \
pcre2_code *pcre2_compile(PCRE2_SPTR, PCRE2_SIZE, uint32_t, int *, PCRE2_SIZE *, \
    pcre2_compile_context *);

#define PCRE2_JOIN(a,b) a ## b
#define PCRE2_GLUE(a,b) PCRE2_JOIN(a,b)
#define PCRE2_SUFFIX(a) PCRE2_GLUE(a,PCRE2_LOCAL_WIDTH)

#define PCRE2_SPTR                  PCRE2_SUFFIX(PCRE2_SPTR)
#define pcre2_code                  PCRE2_SUFFIX(pcre2_code_)
#define pcre2_compile_context          PCRE2_SUFFIX(pcre2_compile_context_)
#define pcre2_compile                         PCRE2_SUFFIX(pcre2_compile_)

#define PCRE2_TYPES_STRUCTURES_AND_FUNCTIONS \
PCRE2_COMPILE_FUNCTIONS

#define PCRE2_LOCAL_WIDTH 8
PCRE2_TYPES_STRUCTURES_AND_FUNCTIONS
#undef PCRE2_LOCAL_WIDTH

#define PCRE2_LOCAL_WIDTH 16
PCRE2_TYPES_STRUCTURES_AND_FUNCTIONS
#undef PCRE2_LOCAL_WIDTH

#define PCRE2_LOCAL_WIDTH 32
PCRE2_TYPES_STRUCTURES_AND_FUNCTIONS
#undef PCRE2_LOCAL_WIDTH

Thanks for the bug report!