UTF-8

[DEKHTIARJonathan]

Hello dear friends,

I am experiencing a very dirty problem here. I can't manage to find the error.

My project is not open source, but you can find it if you are working at CodeClimate : https://codeclimate.com/repos/5772aa87f97096006f001cb4/feed

I don't know where this UTF-8 error is coming from ... Everything is in UTF-8 as far as I know in my files.

PS: It's a Django project and the workflow is Git Push => Github => Travis-CI => CodeClimate

Thanks a lot for the help

[dblandin]

Sigh :disappointed:. These encoding errors are the worst.

I just found this package which attempts to detect the file encoding for a file. Might look more into that to see if we can leverage that guess to avoid these encoding errors during file read.

[DEKHTIARJonathan]

Very nice finding. If you need me to test anything. Do not hesitate I will be glad to help!

[dblandin]

Very nice finding. If you need me to test anything. Do not hesitate I will be glad to help!

Will do! Thanks!

[DEKHTIARJonathan]

@dblandin any news ?

[dblandin]

Not yet. Code Climate was out on a retreat last week so I have a few things to catch up on before I dig more into this. Will plan to have an update for you by Friday.

Thanks!

If any Python developer happens to stumble upon this and has familiarity with these encoding issues... we accept PRs :)

[DEKHTIARJonathan]

Thanks a lot for your help. I Wait for your answer then

[dblandin]

@DEKHTIARJonathan,

Hey, sorry for the delay here. I'm going to look into this issue today and will get back to you.

[DEKHTIARJonathan]

Awesome ;) If you need me to beta-test some early deploy. Do not hesitate, I have no clue why it's not working on Travis ...

[dblandin]

Hey @DEKHTIARJonathan,

I just released v0.2.0 of the reporter which includes file processing debug output. We might be able to narrow down the issue if we can figure out which file the reporter is crashing on.

Could you trigger a new build on Travis with the following invocation and post the relevant output here?

$ codeclimate-test-reporter --debug

Thanks!

[DEKHTIARJonathan]

I have launched a new build with the version 0.2.0.

Here is the output.

$ codeclimate-test-reporter --debug
codeclimate-test-repoter 0.2.0
Requests 2.11.1
Python 2.7.9 (default, Feb  5 2015, 15:48:42) 
[GCC 4.6.3]
/home/travis/virtualenv/python2.7.9/bin/python
Linux 3.13.0-40-genericProcessing file path=manage.py
Processing file path=/home/travis/virtualenv/python2.7.9/lib/python2.7/site-packages/dj_database_url.py
Processing file path=/home/travis/virtualenv/python2.7.9/lib/python2.7/site-packages/dotenv.py
Processing file path=/home/travis/virtualenv/python2.7.9/lib/python2.7/site-packages/feedparser.py
Processing file path=/home/travis/virtualenv/python2.7.9/lib/python2.7/site-packages/ipaddress.py
Processing file path=/home/travis/virtualenv/python2.7.9/lib/python2.7/site-packages/mock.py
Processing file path=/home/travis/virtualenv/python2.7.9/lib/python2.7/site-packages/six.py
Processing file path=/home/travis/virtualenv/python2.7.9/lib/python2.7/site-packages/validate_email.py
Processing file path=/home/travis/virtualenv/python2.7.9/lib/python2.7/site-packages/Crypto/__init__.py
Processing file path=/home/travis/virtualenv/python2.7.9/lib/python2.7/site-packages/Crypto/pct_warnings.py
Processing file path=/home/travis/virtualenv/python2.7.9/lib/python2.7/site-packages/Crypto/Cipher/AES.py
Processing file path=/home/travis/virtualenv/python2.7.9/lib/python2.7/site-packages/Crypto/Cipher/__init__.py
Processing file path=/home/travis/virtualenv/python2.7.9/lib/python2.7/site-packages/Crypto/Cipher/blockalgo.py
Processing file path=/home/travis/virtualenv/python2.7.9/lib/python2.7/site-packages/Crypto/Hash/SHA.py
Processing file path=/home/travis/virtualenv/python2.7.9/lib/python2.7/site-packages/Crypto/Hash/SHA256.py
Processing file path=/home/travis/virtualenv/python2.7.9/lib/python2.7/site-packages/Crypto/Hash/__init__.py
Processing file path=/home/travis/virtualenv/python2.7.9/lib/python2.7/site-packages/Crypto/Hash/hashalgo.py
Processing file path=/home/travis/virtualenv/python2.7.9/lib/python2.7/site-packages/Crypto/PublicKey/DSA.py
Processing file path=/home/travis/virtualenv/python2.7.9/lib/python2.7/site-packages/Crypto/PublicKey/RSA.py
Processing file path=/home/travis/virtualenv/python2.7.9/lib/python2.7/site-packages/Crypto/PublicKey/_DSA.py
Processing file path=/home/travis/virtualenv/python2.7.9/lib/python2.7/site-packages/Crypto/PublicKey/_RSA.py
Processing file path=/home/travis/virtualenv/python2.7.9/lib/python2.7/site-packages/Crypto/PublicKey/__init__.py
Processing file path=/home/travis/virtualenv/python2.7.9/lib/python2.7/site-packages/Crypto/PublicKey/_slowmath.py
Processing file path=/home/travis/virtualenv/python2.7.9/lib/python2.7/site-packages/Crypto/PublicKey/pubkey.py
Processing file path=/home/travis/virtualenv/python2.7.9/lib/python2.7/site-packages/Crypto/Random/_UserFriendlyRNG.py
Processing file path=/home/travis/virtualenv/python2.7.9/lib/python2.7/site-packages/Crypto/Random/__init__.py
Processing file path=/home/travis/virtualenv/python2.7.9/lib/python2.7/site-packages/Crypto/Random/Fortuna/FortunaAccumulator.py
Processing file path=/home/travis/virtualenv/python2.7.9/lib/python2.7/site-packages/Crypto/Random/Fortuna/FortunaGenerator.py
Processing file path=/home/travis/virtualenv/python2.7.9/lib/python2.7/site-packages/Crypto/Random/Fortuna/SHAd256.py
Processing file path=/home/travis/virtualenv/python2.7.9/lib/python2.7/site-packages/Crypto/Random/Fortuna/__init__.py
Processing file path=/home/travis/virtualenv/python2.7.9/lib/python2.7/site-packages/Crypto/Random/OSRNG/__init__.py
Processing file path=/home/travis/virtualenv/python2.7.9/lib/python2.7/site-packages/Crypto/Random/OSRNG/posix.py
Processing file path=/home/travis/virtualenv/python2.7.9/lib/python2.7/site-packages/Crypto/Random/OSRNG/rng_base.py
Processing file path=/home/travis/virtualenv/python2.7.9/lib/python2.7/site-packages/Crypto/Util/Counter.py
Processing file path=/home/travis/virtualenv/python2.7.9/lib/python2.7/site-packages/Crypto/Util/__init__.py
Processing file path=/home/travis/virtualenv/python2.7.9/lib/python2.7/site-packages/Crypto/Util/_number_new.py
Processing file path=/home/travis/virtualenv/python2.7.9/lib/python2.7/site-packages/Crypto/Util/asn1.py
Processing file path=/home/travis/virtualenv/python2.7.9/lib/python2.7/site-packages/Crypto/Util/number.py
Processing file path=/home/travis/virtualenv/python2.7.9/lib/python2.7/site-packages/Crypto/Util/py3compat.py
Processing file path=/home/travis/virtualenv/python2.7.9/lib/python2.7/site-packages/DNS/Base.py
Processing file path=/home/travis/virtualenv/python2.7.9/lib/python2.7/site-packages/DNS/Class.py
Processing file path=/home/travis/virtualenv/python2.7.9/lib/python2.7/site-packages/DNS/Lib.py
Traceback (most recent call last):
  File "/home/travis/virtualenv/python2.7.9/bin/codeclimate-test-reporter", line 9, in <module>
    load_entry_point('codeclimate-test-reporter==0.2.0', 'console_scripts', 'codeclimate-test-reporter')()
  File "/home/travis/virtualenv/python2.7.9/lib/python2.7/site-packages/codeclimate_test_reporter/__main__.py", line 11, in run
    sys.exit(runner.run())
  File "/home/travis/virtualenv/python2.7.9/lib/python2.7/site-packages/codeclimate_test_reporter/components/runner.py", line 49, in run
    exit_status = reporter.run()
  File "/home/travis/virtualenv/python2.7.9/lib/python2.7/site-packages/codeclimate_test_reporter/components/reporter.py", line 36, in run
    payload = formatter.payload()
  File "/home/travis/virtualenv/python2.7.9/lib/python2.7/site-packages/codeclimate_test_reporter/components/formatter.py", line 20, in payload
    source_files = self.__source_files()
  File "/home/travis/virtualenv/python2.7.9/lib/python2.7/site-packages/codeclimate_test_reporter/components/formatter.py", line 58, in __source_files
    payload = file_coverage.payload()
  File "/home/travis/virtualenv/python2.7.9/lib/python2.7/site-packages/codeclimate_test_reporter/components/file_coverage.py", line 18, in payload
    "blob_id": self.__blob(),
  File "/home/travis/virtualenv/python2.7.9/lib/python2.7/site-packages/codeclimate_test_reporter/components/file_coverage.py", line 44, in __blob
    contents = self.__file_body()
  File "/home/travis/virtualenv/python2.7.9/lib/python2.7/site-packages/codeclimate_test_reporter/components/file_coverage.py", line 51, in __file_body
    self.file_body = open(self.__filename(), "r", encoding="utf-8-sig").read()
  File "/home/travis/virtualenv/python2.7.9/lib/python2.7/codecs.py", line 296, in decode
    (result, consumed) = self._buffer_decode(data, self.errors, final)
  File "/home/travis/virtualenv/python2.7.9/lib/python2.7/encodings/utf_8_sig.py", line 66, in _buffer_decode
    return codecs.utf_8_decode(input, errors, final)
UnicodeDecodeError: 'utf8' codec can't decode byte 0xf6 in position 23423: invalid start byte

If needed here is the requirements I use:

awscli==1.11.8
boto==2.43.0
botocore==1.4.65
click==6.6
codeclimate-test-reporter==0.2.0
colorama==0.3.7
coverage==4.2
coveralls==1.1
dateutils==0.6.6
dj-database-url==0.4.1
Django==1.10.2
django-dotenv==1.4.1
django-downloadview==1.9
django-encrypted-fields==1.1.2
django-filter==0.15.3
django-getenv==1.3.1
django-material==0.9.1
django-storages==1.5.1
djangorestframework==3.5.1
docopt==0.6.2
docutils==0.12
factory-boy==2.7.0
fake-factory==0.7.2
feedgen==0.4.0
feedparser==5.2.1
future==0.15.2
futures==3.0.5
gunicorn==19.6.0
ipaddress==1.0.17
jmespath==0.9.0
lxml==3.6.4
Markdown==2.6.7
oauthlib==2.0.0
Pillow==3.4.1
psycopg2==2.6.2
pyasn1==0.1.9
pycrypto==2.6.1
pydns==2.3.6
python-dateutil==2.5.3
python-keyczar==0.716
python-twitter==3.1
pytz==2016.7
requests==2.11.1
requests-oauthlib==0.7.0
rsa==3.4.2
s3transfer==0.1.9
six==1.10.0
twython==3.4.0
validate-email==1.3
whitenoise==3.2.2

Thanks a lot for your help,

Best Regards

[dblandin]

Ahh, interesting. So it looks like your .coverage file contains coverage data for your third-party dependencies.

Since those are typically files that you don't care for when calculating test coverage, I think we should try omitting those files from the report.

If you're using py.test to generate the coverage report, you can adjust your invocation to include the --cov-config flag:

py.test --cov-config .coveragerc --cov=myproj myproj/tests/

with .coveragerc containing the following:

[run]
omit = /home/travis/virtualenv/*/lib/*/site-packages/*

source: https://pypi.python.org/pypi/pytest-cov

You could use the more specific /home/travis/virtualenv/python2.7.9/lib/python2.7/site-packages/* value but the glob format above should cover changes to different python versions moving forward.

[dblandin]

Another option as a change to the reporter itself might be to capture git ls-files output and skip any files not included in that list. This would skip any installed packages but creates a stronger dependency on a working git directory during reporting which isn't ideal. Some CI environments don't provide the git repository along with the source code.

[DEKHTIARJonathan]

You absolutely rock !

.coveragerc :

[run]
omit = /home/travis/virtualenv/*/lib/*/site-packages/*

Perfectly working ! Thanks a lot @dblandin.

Maybe it should be documented somewhere ;)

[dblandin]

@DEKHTIARJonathan Awesome! Glad that worked for you. I'll make a note to add that documentation.

Thanks!

[BuonOmo]

I juste found this issue now, mine (#37) is definetily a dup of this one..