AttributeError: 'LexToken' object has no attribute 'source'

[ZedThree]

The following raises AttributeError: 'LexToken' object has no attribute 'source':

import pcpp
preprocessor = pcpp.Preprocessor()
preprocessor.define("BAR FOO")
for tok in preprocessor.parsegen("#define FOO 1\n#if FOO == BAR\n#endif\n", None):
    print(tok)

with the following (abridged) backtrace:

~/Codes/pcpp/pcpp/preprocessor.py in expand_macros(self, tokens, expanding_from)
    762                         # A simple macro
    763                         rep = [copy.copy(_x) for _x in m.value]
--> 764                         ex = self.expand_macros(rep, expanding_from + [t.value])
    765                         #print "\nExpanding macro", m, "\ninto", ex, "\nreplacing", tokens[i:i+1]
    766                         for e in ex:

~/Codes/pcpp/pcpp/preprocessor.py in expand_macros(self, tokens, expanding_from)
    765                         #print "\nExpanding macro", m, "\ninto", ex, "\nreplacing", tokens[i:i+1]
    766                         for e in ex:
--> 767                             e.source = t.source
    768                             e.lineno = t.lineno
    769                             if not hasattr(e, 'expanded_from'):

I believe the problem is this line:

rep = [copy.copy(_x) for _x in m.value]

which makes a list of LexTokens which don't have a source attribute. rep gets passed into expand_macros which is expecting a list of tokens which do have a source.

I have a fix, although I'm not 100% sure it's the correct thing to do:

                         rep = [copy.copy(_x) for _x in m.value]
+                        for r in rep:
+                            r.source = m.source
                         ex = self.expand_macros(rep, expanding_from + [t.value])

I'm happy to submit a PR if that's an acceptable solution.

[ned14]

Fixed, and thanks for reporting this bug!