Line length calculation is dependent on line endings

[GoogleCodeExporter]

What steps will reproduce the problem?
1. Run pyang --ietf on the attached testLF.yang:

% pyang --ietf testLF.yang 
testLF.yang:1: warning: IETF rule (RFC formatting): line length 71 exceeds 70 
characters
testLF.yang:2: warning: IETF rule (RFC formatting): line length 72 exceeds 70 
characters
testLF.yang:3: warning: IETF rule (RFC formatting): line length 73 exceeds 70 
characters
...

(note that testLF.yang has "Unix" LF line endings)

2. Run pyang --ietf on the attached testCRLF.yang:

% pyang --ietf testCRLF.yang 
testCRLF.yang:1: warning: IETF rule (RFC formatting): line length 72 exceeds 70 
characters
testCRLF.yang:2: warning: IETF rule (RFC formatting): line length 73 exceeds 70 
characters
testCRLF.yang:3: warning: IETF rule (RFC formatting): line length 74 exceeds 70 
characters
...

(note that testCRLF.yang has "DOS" CR LF line endings)

What is the expected output? What do you see instead?

By my calculation, these lines are of length 70, 71 and 72, so I would expect 
to see the following for both of them:

testCRLF.yang:2: warning: IETF rule (RFC formatting): line length 71 exceeds 70 
characters
testCRLF.yang:3: warning: IETF rule (RFC formatting): line length 72 exceeds 70 
characters

What version of the product are you using? On what operating system?

Latest SVN version on OS X Yosemite.

Please provide any additional information below.

It looks to me as though this behaviour stems from text.splitlines(True) in 
yang_parser.py below:

class YangTokenizer(object):
    def __init__(self, text, pos, errors,
                 max_line_len=None, keep_comments=False):
        self.lines = collections.deque(text.splitlines(True))

This is retaining line endings, which will be either LF (adding 1 to the line 
length) or CR LF (adding 2 to the line length).

So the fix might be as simple as changing the above to text.splitlines(False)?

Original issue reported on code.google.com by william....@gmail.com on 20 Jan 2015 at 10:12

Attachments:

• test.zip