Fixes handling of unicode characters

pr0ton commented 12 years ago

Simple fix to directly return a unicode string instead of converting into a str and then converting into a unicode object.

rennat commented 11 years ago

There are some test failures on my testing environment. The ones that are preventing this merge are the BeautifulSoupBugs.

$ python --version
Python 2.7.2

$ pip freeze
BeautifulSoup==3.2.0
Jinja2==2.6
Pygments==1.4
Sphinx==1.0.7
cssutils==0.9.8a3
docutils==0.8
wsgiref==0.1.2

$ python tests.py 
.......F.FF........F.FF.......
======================================================================
FAIL: test_08_fromURL (__main__.Basic)
Test 'fromURL' constructor
----------------------------------------------------------------------
Traceback (most recent call last):
  File "tests.py", line 71, in test_08_fromURL
    self.assertEqual(p.style_string, "p {color: #999}")
AssertionError: u'' != 'p {color: #999}'

======================================================================
FAIL: test_double_comment (__main__.BeautifulSoupBugs)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "tests.py", line 209, in test_double_comment
    self.assertNotIn("<!--<!--", output)
AssertionError: '<!--<!--' unexpectedly found in u'<!--<!-- comment -->-->'

======================================================================
FAIL: test_double_doctype (__main__.BeautifulSoupBugs)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "tests.py", line 204, in test_double_doctype
    self.assertNotIn("<!<!", output)
AssertionError: '<!<!' unexpectedly found in u'<!<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"\n"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">>'

======================================================================
FAIL: test_adjacent_selector (__main__.ComplexSelectors)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "tests.py", line 246, in test_adjacent_selector
    self.assertEqual(output, expected)
AssertionError: u'<h1>Hello World!</h1><h2>How are you?</h2>' != u'<h1>Hello World!</h1><h2 style="color: red">How are you?</h2>'
- <h1>Hello World!</h1><h2>How are you?</h2>
+ <h1>Hello World!</h1><h2 style="color: red">How are you?</h2>
?                         +++++++++++++++++++

======================================================================
FAIL: test_child_selector (__main__.ComplexSelectors)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "tests.py", line 239, in test_child_selector
    self.assertEqual(output, expected)
AssertionError: u'<h1><span>Hello World!</span></h1>' != u'<h1><span style="color: red">Hello World!</span></h1>'
- <h1><span>Hello World!</span></h1>
+ <h1><span style="color: red">Hello World!</span></h1>
?          +++++++++++++++++++

======================================================================
FAIL: test_combination_selector (__main__.ComplexSelectors)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "tests.py", line 225, in test_combination_selector
    self.assertEqual(output, expected)
AssertionError: u'<h1 id="a" class="b">Hello World!</h1>' != u'<h1 id="a" class="b" style="color: red">Hello World!</h1>'
- <h1 id="a" class="b">Hello World!</h1>
+ <h1 id="a" class="b" style="color: red">Hello World!</h1>
?                     +++++++++++++++++++

----------------------------------------------------------------------
Ran 30 tests in 0.758s

FAILED (failures=6)

cyounkins commented 11 years ago

rennat, with the latest master of your branch, I'm seeing 4 failed tests. The same tests fail with pr0ton's patch.

Python 2.7.3
(virtual_env)Craigs-MacBook-Pro:pynliner cyounkins$ pip freeze
BeautifulSoup==3.2.1
cssutils==0.9.10b1
distribute==0.6.28
wsgiref==0.1.2
(virtual_env)Craigs-MacBook-Pro:pynliner cyounkins$ python tests.py 

# .......F...........F.FF......

FAIL: test_08_fromURL (**main**.Basic)

## Test 'fromURL' constructor

Traceback (most recent call last):
  File "tests.py", line 71, in test_08_fromURL
    self.assertEqual(p.style_string, "p {color: #999}")
AssertionError: u'' != 'p {color: #999}'

# 

## FAIL: test_adjacent_selector (**main**.ComplexSelectors)

Traceback (most recent call last):
  File "tests.py", line 246, in test_adjacent_selector
    self.assertEqual(output, expected)
AssertionError: u'<h1>Hello World!</h1><h2>How are you?</h2>' != u'<h1>Hello World!</h1><h2 style="color: red">How are you?</h2>'
- <h1>Hello World!</h1><h2>How are you?</h2>
- <h1>Hello World!</h1><h2 style="color: red">How are you?</h2>
  ?                         +++++++++++++++++++

# 

## FAIL: test_child_selector (**main**.ComplexSelectors)

Traceback (most recent call last):
  File "tests.py", line 239, in test_child_selector
    self.assertEqual(output, expected)
AssertionError: u'<h1><span>Hello World!</span></h1>' != u'<h1><span style="color: red">Hello World!</span></h1>'
- <h1><span>Hello World!</span></h1>
- <h1><span style="color: red">Hello World!</span></h1>
  ?          +++++++++++++++++++

# 

## FAIL: test_combination_selector (**main**.ComplexSelectors)

Traceback (most recent call last):
  File "tests.py", line 225, in test_combination_selector
    self.assertEqual(output, expected)
AssertionError: u'<h1 id="a" class="b">Hello World!</h1>' != u'<h1 id="a" class="b" style="color: red">Hello World!</h1>'
- <h1 id="a" class="b">Hello World!</h1>
- <h1 id="a" class="b" style="color: red">Hello World!</h1>
  ?                     +++++++++++++++++++

---

Ran 29 tests in 0.195s

FAILED (failures=4)

rennat commented 11 years ago

While not merged in directly, the changes proposed in this pull request are covered in a previous pull request that has been merged.

rennat / pynliner

Fixes handling of unicode characters #19