search

ned14 / pcpp

A C99 preprocessor written in pure Python
Other
215 stars 39 forks source link

tokenize() is not compatible with expand_macros() #1

Closed ShaheedHaque closed 6 years ago

ShaheedHaque commented 7 years ago

I'm not sure if this is regarded as a bug, but the output form tokenize() is not consumable by expand_macros() because the ".source" attribute is missing. I can workaround it like this:

    preproc = pcpp.preprocessor.Preprocessor
    tokens = preproc.tokenize("text")
    for t in tokens:
        t.source = "some random value"
    tokens = preproc.expand_macros(tokens)

If the idea is to keep tokens lightweight, perhaps tokenize() could take an optional "source" parameter which, only if not None would be added to the token?

ned14 commented 6 years ago

Shouldn't you be tokenising with group_lines() not tokenise()? The latter is only used at a micro level for macro definition. It's meant to be so lightweight as to be nearly useless.

ShaheedHaque commented 6 years ago

Thanks for the kind reply. I'm afraid that I no longer have a usable context in which to try things and provide a useful reply.

ned14 commented 6 years ago

Sorry for such a late reply. To be honest, I forgot about this issue. My apologies.