search

alan-turing-institute / CleverCSV

CleverCSV is a Python package for handling messy CSV files. It provides a drop-in replacement for the builtin CSV module with improved dialect detection, and comes with a handy command line application for working with CSV files.
https://clevercsv.readthedocs.io
MIT License
1.24k stars 70 forks source link

0.7.5: pytest is failing in `tests/test_unit/test_encoding.py::EncodingTestCase::test_encoding_cchardet` unit #85

Closed kloczek closed 1 year ago

kloczek commented 1 year ago

I'm packaging your module as an rpm package so I'm using the typical PEP517 based build, install and test cycle used on building packages from non-root account.

Here is pytest output:

```console + PYTHONPATH=/home/tkloczko/rpmbuild/BUILDROOT/python-clevercsv-0.7.5-2.fc35.x86_64/usr/lib64/python3.8/site-packages:/home/tkloczko/rpmbuild/BUILDROOT/python-clevercsv-0.7.5-2.fc35.x86_64/usr/lib/python3.8/site-packages + /usr/bin/pytest -ra -m 'not network' ==================================================================================== test session starts ==================================================================================== platform linux -- Python 3.8.16, pytest-7.2.1, pluggy-1.0.0 rootdir: /home/tkloczko/rpmbuild/BUILD/CleverCSV-0.7.5 collected 156 items tests/test_unit/test_consistency.py .. [ 1%] tests/test_unit/test_console.py .................... [ 14%] tests/test_unit/test_cparser.py ................................................. [ 45%] tests/test_unit/test_detect.py .... [ 48%] tests/test_unit/test_detect_pattern.py .................... [ 60%] tests/test_unit/test_detect_type.py .......... [ 67%] tests/test_unit/test_dict.py .................... [ 80%] tests/test_unit/test_encoding.py F. [ 81%] tests/test_unit/test_fuzzing.py . [ 82%] tests/test_unit/test_normal_forms.py ..... [ 85%] tests/test_unit/test_potential_dialects.py .... [ 87%] tests/test_unit/test_reader.py ......... [ 93%] tests/test_unit/test_wrappers.py ..... [ 96%] tests/test_unit/test_write.py ..... [100%] ========================================================================================= FAILURES ========================================================================================== __________________________________________________________________________ EncodingTestCase.test_encoding_cchardet __________________________________________________________________________ self = @unittest.skipIf( platform.system() == "Windows", reason="No faust-cchardet wheels for Windows (yet)", ) def test_encoding_cchardet(self): for case in self.cases: table = case["table"] encoding = case["encoding"] with self.subTest(encoding=encoding): out_encoding = case["cchardet_encoding"] tmpfname = self._build_file(table, encoding) detected = get_encoding(tmpfname, try_cchardet=True) > self.assertEqual(out_encoding, detected) E AssertionError: 'WINDOWS-1252' != 'ISO-8859-1' E - WINDOWS-1252 E + ISO-8859-1 tests/test_unit/test_encoding.py:83: AssertionError ===================================================================================== warnings summary ====================================================================================== ../../../../../usr/lib/python3.8/site-packages/wilderness/tester.py:23 /usr/lib/python3.8/site-packages/wilderness/tester.py:23: PytestCollectionWarning: cannot collect test class 'Tester' because it has a __init__ constructor (from: tests/test_unit/test_console.py) class Tester: -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html ================================================================================== short test summary info ================================================================================== FAILED tests/test_unit/test_encoding.py::EncodingTestCase::test_encoding_cchardet - AssertionError: 'WINDOWS-1252' != 'ISO-8859-1' ========================================================================= 1 failed, 155 passed, 1 warning in 0.86s ========================================================================== ```

Here is list of installed modules in build env

```console Package Version ----------------------------- ----------------- alabaster 0.7.12 appdirs 1.4.4 asn1crypto 1.5.1 attrs 22.2.0 Babel 2.11.0 bcrypt 3.2.2 build 0.9.0 cffi 1.15.1 chardet 5.1.0 charset-normalizer 3.0.1 contourpy 1.0.6 cryptography 38.0.4 cssselect 1.1.0 cycler 0.11.0 Cython 0.29.33 distro 1.8.0 docutils 0.19 exceptiongroup 1.0.0 extras 1.0.0 fixtures 4.0.0 fonttools 4.38.0 gpg 1.18.0-unknown idna 3.4 imagesize 1.4.1 importlib-metadata 5.1.0 iniconfig 1.1.1 Jinja2 3.1.2 kiwisolver 1.4.4 libcomps 0.1.19 lxml 4.9.1 MarkupSafe 2.1.1 matplotlib 3.6.3 numpy 1.24.1 olefile 0.46 packaging 21.3 pandas 1.5.2 pbr 5.9.0 pep517 0.13.0 Pillow 9.4.0 pip 22.3.1 pluggy 1.0.0 ply 3.11 pyasn1 0.4.8 pyasn1-modules 0.2.8 pycparser 2.21 Pygments 2.14.0 PyGObject 3.42.2 pyparsing 3.0.9 pytest 7.2.1 python-dateutil 2.8.2 pytz 2022.4 PyYAML 6.0 regex 2022.10.31 requests 2.28.1 rpm 4.17.0 scour 0.38.2 setuptools 65.6.3 six 1.16.0 snowballstemmer 2.2.0 Sphinx 5.3.0 sphinxcontrib-applehelp 1.0.2.dev20221204 sphinxcontrib-devhelp 1.0.2.dev20221204 sphinxcontrib-htmlhelp 2.0.0 sphinxcontrib-jsmath 1.0.1.dev20221204 sphinxcontrib-qthelp 1.0.3.dev20221204 sphinxcontrib-serializinghtml 1.1.5 termcolor 1.1.0 testtools 2.5.0 tomli 2.0.1 tpm2-pkcs11-tools 1.33.7 tpm2-pytss 1.1.0 urllib3 1.26.12 wheel 0.38.4 wilderness 0.1.9 zipp 3.11.0 ```
GjjvdBurg commented 1 year ago

Thanks for reporting this issue @kloczek! From the details it looks like you're missing the faust-cchardet dependency. That dependency is optional for the base version of CleverCSV but not for the "full" version. I see that the unit test in question assumes that this is available though, so I'll leave this issue open to resolve that.

GjjvdBurg commented 1 year ago

I think this should be fixed in v0.7.6, but let me know if that's not the case

kloczek commented 1 year ago

Just tested new verzaion and indeed issue has been solved. Thank you very much 👍 What about that pytest warning?

+ PYTHONPATH=/home/tkloczko/rpmbuild/BUILDROOT/python-clevercsv-0.7.6-2.fc35.x86_64/usr/lib64/python3.8/site-packages:/home/tkloczko/rpmbuild/BUILDROOT/python-clevercsv-0.7.6-2.fc35.x86_64/usr/lib/python3.8/site-packages
+ /usr/bin/pytest -ra -m 'not network'
==================================================================================== test session starts ====================================================================================
platform linux -- Python 3.8.16, pytest-7.3.0, pluggy-1.0.0
rootdir: /home/tkloczko/rpmbuild/BUILD/CleverCSV-0.7.6
plugins: cov-4.0.0
collected 156 items

tests/test_unit/test_consistency.py ..                                                                                                                                                [  1%]
tests/test_unit/test_console.py ....................                                                                                                                                  [ 14%]
tests/test_unit/test_cparser.py .................................................                                                                                                     [ 45%]
tests/test_unit/test_detect.py ....                                                                                                                                                   [ 48%]
tests/test_unit/test_detect_pattern.py ....................                                                                                                                           [ 60%]
tests/test_unit/test_detect_type.py ..........                                                                                                                                        [ 67%]
tests/test_unit/test_dict.py ....................                                                                                                                                     [ 80%]
tests/test_unit/test_encoding.py s.                                                                                                                                                   [ 81%]
tests/test_unit/test_fuzzing.py .                                                                                                                                                     [ 82%]
tests/test_unit/test_normal_forms.py .....                                                                                                                                            [ 85%]
tests/test_unit/test_potential_dialects.py ....                                                                                                                                       [ 87%]
tests/test_unit/test_reader.py .........                                                                                                                                              [ 93%]
tests/test_unit/test_wrappers.py .....                                                                                                                                                [ 96%]
tests/test_unit/test_write.py .....                                                                                                                                                   [100%]

===================================================================================== warnings summary ======================================================================================
../../../../../usr/lib/python3.8/site-packages/wilderness/tester.py:23
  /usr/lib/python3.8/site-packages/wilderness/tester.py:23: PytestCollectionWarning: cannot collect test class 'Tester' because it has a __init__ constructor (from: tests/test_unit/test_console.py)
    class Tester:

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================================================================== short test summary info ==================================================================================
SKIPPED [1] tests/test_unit/test_encoding.py:69: Failed to import cchardet, skipping this test
========================================================================= 155 passed, 1 skipped, 1 warning in 0.78s =========================================================================
GjjvdBurg commented 1 year ago

Good to hear! That pytest warning was never a real issue, but should now be fixed when using Wilderness v0.1.10.