search

pytest-dev / pytest

The pytest framework makes it easy to write small tests, yet scales to support complex functional testing
https://pytest.org
MIT License
12.13k stars 2.69k forks source link

Unicode error when launching pytest #368

Closed pytestbot closed 9 years ago

pytestbot commented 11 years ago

Originally reported by: BitBucket: glushonkov, GitHub: glushonkov

First I launched pytest on my unicode python project, it failed on junitxml.py file from pytest. Then I found fix for this issue:

https://bitbucket.org/apkawa/pytest/commits/1a284b99104e2d525b46a13a0fffff081158377a

Now it fails with the following message:

File "C:\Python27\lib\site-packages\py_xmlgen.py", line 133, in __object self.write(escape(unicode(obj))) UnicodeDecodeError: 'ascii' codec can't decode byte 0xd0 in position 1305: ordinal not in range(128)

Launching like this: py.test --pep8 --junitxml=reports\pep8.log

It seems pytest goes through all the py files of the project and then show this exception for unknown reason.

pytestbot commented 11 years ago

Original comment by holger krekel (BitBucket: hpk42, GitHub: hpk42):

Please provide an example file or zip archive that triggers this behaviour.

pytestbot commented 11 years ago

Original comment by BitBucket: glushonkov, GitHub: glushonkov:

I think i'm not putting all the project source here, so here is one of the files with utf-8 symbols

https://cloud.mail.ru/public/ee50b3f4d836/api.py

pytestbot commented 11 years ago

Original comment by holger krekel (BitBucket: hpk42, GitHub: hpk42):

This is not enough information, sorry. Maybe let's forget the source code if you don't want to disclose it. Please use latest "pytest" (2.4.2) and pytest-pep8 (1.0.5) and then paste the exact pytest invocation and output from your test program. I will then try to replicate the issue.

pytestbot commented 11 years ago

Original comment by BitBucket: glushonkov, GitHub: glushonkov:

#!python

C:\Jenkins\jobs\python-template-2\workspace>py.test --pep8 --junitxml=reports\pep8.log 
============================= test session starts ==============================
platform win32 -- Python 2.7.5 -- pytest-2.4.1
plugins: cache, pep8
collected 122 items / 1 errors

manage.py s
academies/__init__.py s
academies/admin.py F
academies/api.py F
academies/forms.py F
academies/models.py F
academies/urls.py F
academies/migrations/0001_initial.py F
academies/migrations/0002_auto__add_useracademy.py F
academies/migrations/__init__.py s
accounts/__init__.py s
accounts/admin.py F
accounts/api.py F
accounts/backends.py s
accounts/fields.py F
accounts/forms.py F
accounts/integration.py F
accounts/middleware.py F
accounts/models.py F
accounts/urls.py F
accounts/management/__init__.py s
accounts/management/commands/__init__.py s
accounts/management/commands/fixphoto.py F
accounts/management/commands/otkritki_add_points.py F
accounts/management/commands/otkritki_send_add_points_msg.py F
accounts/migrations/0001_initial.py F
accounts/migrations/0002_auto__add_invite__add_unique_invite_user_friend_id_social_type__add_qu.py F
accounts/migrations/0003_auto__add_field_user_winner.py F
accounts/migrations/0004_auto__del_field_user_winner__add_field_user_win_stage.py F
accounts/migrations/0005_auto__add_field_pet_is_home.py F
accounts/migrations/0006_auto__add_field_pet_avatar.py F
accounts/migrations/0007_auto__add_field_user_is_subscribe_purina_.py F
accounts/migrations/0008_auto__add_field_pet_avatar__add_field_user_is_from_odplus.py F
accounts/migrations/0009_auto__add_field_user_odlpus_stage.py F
accounts/migrations/0010_auto__add_field_question_is_moderate.py F
accounts/migrations/0011_auto__add_field_question_email__add_field_question_pet.py F
accounts/migrations/0012_auto__del_field_question_email.py F
accounts/migrations/0013_auto__add_field_user_odplus_manual_points.py F
accounts/migrations/0014_auto__add_field_user_odplus_notification_sent.py F
accounts/migrations/0015_auto__add_field_question_was_fail.py F
accounts/migrations/__init__.py s
activities/__init__.py s
activities/admin.py F
activities/api.py F
activities/models.py F
activities/urls.py F
activities/migrations/0001_initial.py F
activities/migrations/0002_auto__add_useractivity__add_activity.py F
activities/migrations/__init__.py s
app/__init__.py s
app/config.py F
app/dashboard.py F
app/middleware.py F
app/monitor.py F
app/settings.py F
app/settings_dev.py .
app/settings_test.py s
app/urls.py F
app/wsgi.py s
app/social/__init__.py .
app/social/dev.py F
app/social/local.py F
app/social/prod.py F
contents/__init__.py s
contents/admin.py F
contents/api.py F
contents/models.py F
contents/urls.py F
contents/migrations/0001_initial.py F
contents/migrations/0002_auto__add_academy__add_academyarticle__add_content__add_contenttype.py F
contents/migrations/__init__.py s
contests/__init__.py s
contests/admin.py F
contests/api.py F
contests/forms.py .
contests/models.py F
contests/urls.py F
contests/management/__init__.py s
contests/management/commands/__init__.py s
contests/management/commands/collage_rename_thumbnail.py F
contests/management/commands/rating.py F
contests/migrations/0001_initial.py F
contests/migrations/0002_auto__add_collage.py F
contests/migrations/0003_auto__add_votinghistory.py F
contests/migrations/0004_auto__add_winnerdate.py F
contests/migrations/0005_auto__del_winnerdate__add_stagewinner.py F
contests/migrations/0006_auto__add_userstagewinner.py F
contests/migrations/__init__.py s
notifications/__init__.py s
notifications/admin.py F
notifications/email.py F
notifications/models.py F
notifications/odn.py F
notifications/vk.py F
notifications/migrations/0001_initial.py F
notifications/migrations/0002_auto__add_notification__add_notificationlog.py F
notifications/migrations/__init__.py s
photos/__init__.py s
photos/admin.py F
photos/api.py F
photos/forms.py F
photos/models.py F
photos/urls.py F
photos/management/__init__.py .
photos/management/commands/__init__.py .
photos/management/commands/check_photo.py F
photos/management/commands/photos_set_status_inprogress.py F
photos/migrations/0001_initial.py F
photos/migrations/0002_auto__add_photo.py F
photos/migrations/0003_auto__add_field_photo_moderate_status.py F
photos/migrations/__init__.py s
utilities/__init__.py s
utilities/apiresponse/__init__.py F
utilities/capable/__init__.py s
utilities/capable/admin.py F
utilities/capable/models.py F
utilities/filebrousefieldupload/__init__.py F
utilities/templatetags/__init__.py s
utilities/templatetags/common/__init__.py s
utilities/templatetags/common/templatetags/__init__.py s
utilities/templatetags/common/templatetags/common.py F
utilities/uploadrename/__init__.py FTraceback (most recent call last):
  File "C:\Python27\Scripts\py.test-script.py", line 9, in <module>
    load_entry_point('pytest==2.4.1', 'console_scripts', 'py.test')()
  File "C:\Python27\lib\site-packages\_pytest\config.py", line 19, in main
    exitstatus = config.hook.pytest_cmdline_main(config=config)
  File "C:\Python27\lib\site-packages\_pytest\core.py", line 364, in __call__
    return self._docall(methods, kwargs)
  File "C:\Python27\lib\site-packages\_pytest\core.py", line 375, in _docall
    res = mc.execute()
  File "C:\Python27\lib\site-packages\_pytest\core.py", line 293, in execute
    res = method(**kwargs)
  File "C:\Python27\lib\site-packages\_pytest\main.py", line 113, in pytest_cmdline_main
    return wrap_session(config, _main)
  File "C:\Python27\lib\site-packages\_pytest\main.py", line 106, in wrap_session
    exitstatus=session.exitstatus)
  File "C:\Python27\lib\site-packages\_pytest\core.py", line 364, in __call__
    return self._docall(methods, kwargs)
  File "C:\Python27\lib\site-packages\_pytest\core.py", line 375, in _docall
    res = mc.execute()
  File "C:\Python27\lib\site-packages\_pytest\core.py", line 293, in execute
    res = method(**kwargs)
  File "C:\Python27\lib\site-packages\_pytest\terminal.py", line 340, in pytest_sessionfinish
    __multicall__.execute()
  File "C:\Python27\lib\site-packages\_pytest\core.py", line 293, in execute
    res = method(**kwargs)
  File "C:\Python27\lib\site-packages\_pytest\junitxml.py", line 253, in pytest_sessionfinish
    ).unicode(indent=0))
  File "C:\Python27\lib\site-packages\py\_xmlgen.py", line 54, in unicode
    SimpleUnicodeVisitor(l.append, indent).visit(self)
  File "C:\Python27\lib\site-packages\py\_xmlgen.py", line 127, in visit
    visitmethod(node)
  File "C:\Python27\lib\site-packages\py\_xmlgen.py", line 159, in Tag
    self.visit(x)
  File "C:\Python27\lib\site-packages\py\_xmlgen.py", line 127, in visit
    visitmethod(node)
  File "C:\Python27\lib\site-packages\py\_xmlgen.py", line 142, in list
    self.visit(elem)
  File "C:\Python27\lib\site-packages\py\_xmlgen.py", line 127, in visit
    visitmethod(node)
  File "C:\Python27\lib\site-packages\py\_xmlgen.py", line 159, in Tag
    self.visit(x)
  File "C:\Python27\lib\site-packages\py\_xmlgen.py", line 127, in visit
    visitmethod(node)
  File "C:\Python27\lib\site-packages\py\_xmlgen.py", line 159, in Tag
    self.visit(x)
  File "C:\Python27\lib\site-packages\py\_xmlgen.py", line 127, in visit
    visitmethod(node)
  File "C:\Python27\lib\site-packages\py\_xmlgen.py", line 133, in __object
    self.write(escape(unicode(obj)))
UnicodeDecodeError: 'ascii' codec can't decode byte 0xd0 in position 1305: ordinal not in range(128)
pytestbot commented 11 years ago

Original comment by holger krekel (BitBucket: hpk42, GitHub: hpk42):

Does pytest-2.4.2 give the same issue?
And could you re-edit your post here to format it as code? (see the web interface for how it's broken currently)

pytestbot commented 11 years ago

Original comment by holger krekel (BitBucket: hpk42, GitHub: hpk42):

and please supply output of "py.test --version" (this will show plugin version infos as well).

pytestbot commented 11 years ago

Original comment by BitBucket: glushonkov, GitHub: glushonkov:

#!python

C:\Jenkins\jobs\python-template-2\workspace>py.test --version
This is py.test version 2.4.1, imported from C:\Python27\lib\site-packages\pytest.pyc
setuptools registered plugins:
  pytest-cache-1.0 at C:\Python27\lib\site-packages\pytest_cache.pyc
  pytest-pep8-1.0.4 at C:\Python27\lib\site-packages\pytest_pep8.pyc
pytestbot commented 11 years ago

Original comment by BitBucket: glushonkov, GitHub: glushonkov:

I updated to 2.4.2 the error is the same:

#!python

INTERNALERROR>   File "C:\Python27\lib\site-packages\_pytest\junitxml.py", line 134, in append_failure
INTERNALERROR>     fail.append(str(report.longrepr))
INTERNALERROR> UnicodeEncodeError: 'ascii' codec can't encode characters in position 1305-1309: ordinal not in range(128)
pytestbot commented 11 years ago

Original comment by holger krekel (BitBucket: hpk42, GitHub: hpk42):

fix unicode handling with junitxml, fixes issue368.

pytestbot commented 11 years ago

Original comment by holger krekel (BitBucket: hpk42, GitHub: hpk42):

i hope it is fixed -- could you try installing with something like:

#!python

pip install -U -i http://devpi.net/hpk/dev/+simple/ pytest

Should give you pytest-2.4.3.dev2 at least.

pytestbot commented 11 years ago

Original comment by holger krekel (BitBucket: hpk42, GitHub: hpk42):

actually also pass in --pre so pip install -U --pre -i ....

pytestbot commented 11 years ago

Original comment by BitBucket: glushonkov, GitHub: glushonkov:

Thanks, it works now

pytestbot commented 11 years ago

Original comment by BitBucket: glushonkov, GitHub: glushonkov:

Now my Jenkins' Violations plugin can not find the pep8 report pytest generates (it couldn't do it before this fix too, but the error was not the same as now), is this issue can be solved or it's violation's plugin bug? It uses very strange path to file to find it:

#!python

generated xml file: C:\Jenkins\jobs\python-template-2\workspace\reports\pep8.log 
=============== 89 failed, 33 skipped, 1 error in 1.48 seconds ================

C:\Jenkins\jobs\python-template-2\workspace>exit 1 
Build step 'Выполнить команду Windows' marked build as failure
ERROR: Publisher hudson.plugins.violations.ViolationsPublisher aborted due to exception
java.io.FileNotFoundException: C:\Jenkins\jobs\python-template-2\builds\2013-10-22_13-30-44\violations\file\<\failure><\testcase><testcase classname="" name="contests.migrations.0003_auto__add_votinghistory" time="0.0160000324249"><failure message="test failure">C:\Jenkins\jobs\python-template-2\workspace\contests\migrations\0003_auto__add_votinghistory.py.xml (Syntax error in filename)
    at java.io.FileOutputStream.open(Native Method)
    at java.io.FileOutputStream.<init>(Unknown Source)
    at java.io.FileOutputStream.<init>(Unknown Source)
    at hudson.FilePath.write(FilePath.java:1642)
    at hudson.plugins.violations.generate.ExecuteFilePath.execute(ExecuteFilePath.java:40)
    at hudson.plugins.violations.generate.GenerateXML.execute(GenerateXML.java:47)
    at hudson.plugins.violations.ViolationsCollector.invoke(ViolationsCollector.java:122)
    at hudson.plugins.violations.ViolationsCollector.invoke(ViolationsCollector.java:25)
    at hudson.FilePath.act(FilePath.java:912)
    at hudson.FilePath.act(FilePath.java:885)
    at hudson.plugins.violations.ViolationsPublisher.perform(ViolationsPublisher.java:74)
    at hudson.tasks.BuildStepMonitor$3.perform(BuildStepMonitor.java:45)
    at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:781)
    at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:753)
    at hudson.model.Build$BuildExecution.post2(Build.java:183)
    at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:706)
    at hudson.model.Run.execute(Run.java:1690)
    at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
    at hudson.model.ResourceController.execute(ResourceController.java:88)
    at hudson.model.Executor.run(Executor.java:230)
Finished: FAILURE
pytestbot commented 11 years ago

Original comment by holger krekel (BitBucket: hpk42, GitHub: hpk42):

I don't know about integration of Jenkins violations and pep8 reports, sorry. If you find out something, let me know. thanks, holger