Python source code continuation lines are appended

[Phillip-M-Feldman]

I'm using pcpp for processing Python source code. (I realize that this is a somewhat unconventional application). It almost works, except for the following minor issue: pcpp combines a line of source code with its continuation lines to produce a single (potentially very long) output line. It would be great if there were an option to disable this behavior.

[ned14]

Can you supply a repro of the behaviour you describe?

[Phillip-M-Feldman]

Input and output files are attached. (I've changed the file extensions from .py to .txt to get around Github's file type restrictions).

demo.txt demo.txt

[ned14]

So you don't want this:

from numpy import arctan2, arange, array, cos, cumsum, dot, empty, exp, \
  float32, float64, hstack, Inf, int16, int32, int64, linspace, log, mean, \
  NaN, meshgrid, mod, ndarray, nonzero, ones, pi, random, shape, sin, size, \
  sort, sqrt, tan, unique, where, zeros

To turn into this:

from numpy import arctan2, arange, array, cos, cumsum, dot, empty, exp,   float32, float64, hstack, Inf, int16, int32, int64, linspace, log, mean,   NaN, meshgrid, mod, ndarray, nonzero, ones, pi, random, shape, sin, size,   sort, sqrt, tan, unique, where, zeros

pcpp is being standards conforming here. clang's preprocessor does the same as pcpp. GCC's preprocessor would generate bad Python by not concatenating the lines:

from numpy import arctan2, arange, array, cos, cumsum, dot, empty, exp,
  float32, float64, hstack, Inf, int16, int32, int64, linspace, log, mean,
  NaN, meshgrid, mod, ndarray, nonzero, ones, pi, random, shape, sin, size,
  sort, sqrt, tan, unique, where, zeros

Strictly speaking, it wouldn't be hard to change the backslash to line concatenation behaviour. A small modification of group_lines() in Preprocessor would do it.

But I'm also very hesitant indeed to add support to pcpp for bespoke use cases outside its current well defined boundaries. Moreover, the concatenated line is valid Python as far as I can tell. I'm particularly concerned about supporting bespoke use cases into the future, when I have no need for them myself.

What I would say to you is that pcpp was designed to be subclassed and its functions overriden. If you want this behaviour, it's very easy for you to override group_lines() in Preprocessor and get exactly what you're looking for. Is there any good reason that you cannot do this?

[Phillip-M-Feldman]

The suggestion re. overridding group_lines() sounds like exactly what I need; I will see if I can figure out how to do this. Thanks!

[ned14]

I've decided to not action on this issue report, as pcpp matches clang. But thanks for raising the issue.