search

dridk / PyVCF3

A Variant Call Format reader for Python.
http://pyvcf.readthedocs.org/en/latest/index.html
Other
52 stars 9 forks source link

Test failing with python 3.11 #6

Closed kbipinkumar closed 10 months ago

kbipinkumar commented 1 year ago

when building pyVCF3 against python 3.11 the test are failing

running pytest
/usr/lib/python3.11/site-packages/setuptools/command/test.py:194: _DeprecatedInstaller: setuptools.installer and fetch_build_eggs are deprecated.
!!

        ********************************************************************************
        Requirements should be satisfied by a PEP 517 installer.
        If you are using pip, you can try `pip install --use-pep517`.
        ********************************************************************************

!!
  ir_d = dist.fetch_build_eggs(dist.install_requires)
/usr/lib/python3.11/site-packages/setuptools/command/test.py:195: _DeprecatedInstaller: setuptools.installer and fetch_build_eggs are deprecated.
!!

        ********************************************************************************
        Requirements should be satisfied by a PEP 517 installer.
        If you are using pip, you can try `pip install --use-pep517`.
        ********************************************************************************

!!
  tr_d = dist.fetch_build_eggs(dist.tests_require or [])
/usr/lib/python3.11/site-packages/setuptools/command/test.py:196: _DeprecatedInstaller: setuptools.installer and fetch_build_eggs are deprecated.
!!

        ********************************************************************************
        Requirements should be satisfied by a PEP 517 installer.
        If you are using pip, you can try `pip install --use-pep517`.
        ********************************************************************************

!!
  er_d = dist.fetch_build_eggs(
running egg_info
writing PyVCF3.egg-info/PKG-INFO
writing dependency_links to PyVCF3.egg-info/dependency_links.txt
writing entry points to PyVCF3.egg-info/entry_points.txt
writing requirements to PyVCF3.egg-info/requires.txt
writing top-level names to PyVCF3.egg-info/top_level.txt
reading manifest file 'PyVCF3.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
adding license file 'LICENSE'
writing manifest file 'PyVCF3.egg-info/SOURCES.txt'
running build_ext
skipping 'vcf/cparse.c' Cython extension (up-to-date)
building 'vcf.cparse' extension
gcc -DNDEBUG -g -fwrapv -O3 -Wall -march=x86-64 -mtune=generic -O3 -pipe -fno-plt -fexceptions -Wp,-D_FORTIFY_SOURCE=2 -Wformat -Werror=format-security -fstack-clash-protection -fcf-protection -g -ffile-prefix-map=/build/python/src=/usr/src/debug/python -flto=auto -ffat-lto-objects -march=x86-64 -mtune=generic -O3 -pipe -fno-plt -fexceptions -Wp,-D_FORTIFY_SOURCE=2 -Wformat -Werror=format-security -fstack-clash-protection -fcf-protection -g -ffile-prefix-map=/build/python/src=/usr/src/debug/python -flto=auto -march=x86-64 -mtune=generic -O3 -pipe -fno-plt -fexceptions -Wp,-D_FORTIFY_SOURCE=2 -Wformat -Werror=format-security -fstack-clash-protection -fcf-protection -g -ffile-prefix-map=/build/python/src=/usr/src/debug/python -flto=auto -march=x86-64 -mtune=generic -O2 -pipe -fno-plt -fexceptions -Wp,-D_FORTIFY_SOURCE=2 -Wformat -Werror=format-security -fstack-clash-protection -fcf-protection -fPIC -I/usr/include/python3.11 -c vcf/cparse.c -o build/temp.linux-x86_64-cpython-311/vcf/cparse.o
gcc -shared -Wl,-O1,--sort-common,--as-needed,-z,relro,-z,now -flto=auto -Wl,-O1,--sort-common,--as-needed,-z,relro,-z,now -flto=auto -Wl,-O1,--sort-common,--as-needed,-z,relro,-z,now -march=x86-64 -mtune=generic -O2 -pipe -fno-plt -fexceptions -Wp,-D_FORTIFY_SOURCE=2 -Wformat -Werror=format-security -fstack-clash-protection -fcf-protection build/temp.linux-x86_64-cpython-311/vcf/cparse.o -L/usr/lib -o /build/python-pyvcf3/src/PyVCF3-1.0.3/vcf/cparse.cpython-311-x86_64-linux-gnu.so
============================= test session starts ==============================
platform linux -- Python 3.11.3, pytest-7.3.1, pluggy-1.0.0
rootdir: /build/python-pyvcf3/src/PyVCF3-1.0.3
collected 104 items

vcf/test/test_vcf.py ........................F...F.F..FFFF.............. [ 49%]
.....................sssssss............ssFss.F....F.                    [100%]

=================================== FAILURES ===================================
___________________________ Test1kgSites.test_writer ___________________________

self = <vcf.test.test_vcf.Test1kgSites testMethod=test_writer>

    def test_writer(self):
        """FORMAT should not be written if not present in the template and no
        extra tab character should be printed if there are no FORMAT fields."""
        reader = vcf.Reader(fh('1kg.sites.vcf', 'r'))
        out = StringIO()
>       writer = vcf.Writer(out, reader, lineterminator='\n')

vcf/test/test_vcf.py:298: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <vcf.parser.Writer object at 0x7ff899922510>
stream = <_io.StringIO object at 0x7ff89a033760>
template = <vcf.parser.Reader object at 0x7ff899c73e50>, lineterminator = '\n'

    def __init__(self, stream, template, lineterminator="\n"):
>       self.writer = csv.writer(
            stream,
            delimiter="\t",
            lineterminator=lineterminator,
            quotechar="",
            quoting=csv.QUOTE_NONE,
        )
E       TypeError: "quotechar" must be a 1-character string

vcf/parser.py:776: TypeError
__________________________ TestInfoOrder.test_writer ___________________________

self = <vcf.test.test_vcf.TestInfoOrder testMethod=test_writer>

    def test_writer(self):
        """
        Order of INFO fields should be compatible with the order of their
        definition in the header and undefined fields should be last and in
        alphabetical order.
        """
        reader = vcf.Reader(fh('1kg.sites.vcf', 'r'))
        out = StringIO()
>       writer = vcf.Writer(out, reader, lineterminator='\n')

vcf/test/test_vcf.py:354: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <vcf.parser.Writer object at 0x7ff8996a2190>
stream = <_io.StringIO object at 0x7ff899a00700>
template = <vcf.parser.Reader object at 0x7ff8996a31d0>, lineterminator = '\n'

    def __init__(self, stream, template, lineterminator="\n"):
>       self.writer = csv.writer(
            stream,
            delimiter="\t",
            lineterminator=lineterminator,
            quotechar="",
            quoting=csv.QUOTE_NONE,
        )
E       TypeError: "quotechar" must be a 1-character string

vcf/parser.py:776: TypeError
_______________________ TestInfoTypeCharacter.test_write _______________________

self = <vcf.test.test_vcf.TestInfoTypeCharacter testMethod=test_write>

    def test_write(self):
        reader = vcf.Reader(fh('info-type-character.vcf'))
        out = StringIO()
>       writer = vcf.Writer(out, reader)

vcf/test/test_vcf.py:383: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <vcf.parser.Writer object at 0x7ff8999d2c10>
stream = <_io.StringIO object at 0x7ff899a01990>
template = <vcf.parser.Reader object at 0x7ff8999d2d50>, lineterminator = '\n'

    def __init__(self, stream, template, lineterminator="\n"):
>       self.writer = csv.writer(
            stream,
            delimiter="\t",
            lineterminator=lineterminator,
            quotechar="",
            quoting=csv.QUOTE_NONE,
        )
E       TypeError: "quotechar" must be a 1-character string

vcf/parser.py:776: TypeError
_________________________ TestParseMetaLine.test_write _________________________

self = <vcf.test.test_vcf.TestParseMetaLine testMethod=test_write>

    def test_write(self):
        reader = vcf.Reader(fh('parse-meta-line.vcf'))
        out = StringIO()
>       writer = vcf.Writer(out, reader)

vcf/test/test_vcf.py:427: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <vcf.parser.Writer object at 0x7ff899f4d790>
stream = <_io.StringIO object at 0x7ff899a01900>
template = <vcf.parser.Reader object at 0x7ff899f4cc50>, lineterminator = '\n'

    def __init__(self, stream, template, lineterminator="\n"):
>       self.writer = csv.writer(
            stream,
            delimiter="\t",
            lineterminator=lineterminator,
            quotechar="",
            quoting=csv.QUOTE_NONE,
        )
E       TypeError: "quotechar" must be a 1-character string

vcf/parser.py:776: TypeError
________________________ TestGatkOutputWriter.testWrite ________________________

self = <vcf.test.test_vcf.TestGatkOutputWriter testMethod=testWrite>

    def testWrite(self):

        reader = vcf.Reader(fh('gatk.vcf'))
        out = StringIO()
>       writer = vcf.Writer(out, reader)

vcf/test/test_vcf.py:452: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <vcf.parser.Writer object at 0x7ff8996a1890>
stream = <_io.StringIO object at 0x7ff89a033910>
template = <vcf.parser.Reader object at 0x7ff8996a1010>, lineterminator = '\n'

    def __init__(self, stream, template, lineterminator="\n"):
>       self.writer = csv.writer(
            stream,
            delimiter="\t",
            lineterminator=lineterminator,
            quotechar="",
            quoting=csv.QUOTE_NONE,
        )
E       TypeError: "quotechar" must be a 1-character string

vcf/parser.py:776: TypeError
______________________ TestBcfToolsOutputWriter.testWrite ______________________

self = <vcf.test.test_vcf.TestBcfToolsOutputWriter testMethod=testWrite>

    def testWrite(self):

        reader = vcf.Reader(fh('bcftools.vcf'))
        out = StringIO()
>       writer = vcf.Writer(out, reader)

vcf/test/test_vcf.py:486: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <vcf.parser.Writer object at 0x7ff899d78390>
stream = <_io.StringIO object at 0x7ff899a00e50>
template = <vcf.parser.Reader object at 0x7ff899d79850>, lineterminator = '\n'

    def __init__(self, stream, template, lineterminator="\n"):
>       self.writer = csv.writer(
            stream,
            delimiter="\t",
            lineterminator=lineterminator,
            quotechar="",
            quoting=csv.QUOTE_NONE,
        )
E       TypeError: "quotechar" must be a 1-character string

vcf/parser.py:776: TypeError
______________________ TestWriterDictionaryMeta.testWrite ______________________

self = <vcf.test.test_vcf.TestWriterDictionaryMeta testMethod=testWrite>

    def testWrite(self):

        reader = vcf.Reader(fh('example-4.1-bnd.vcf'))
        out = StringIO()
>       writer = vcf.Writer(out, reader)

vcf/test/test_vcf.py:515: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <vcf.parser.Writer object at 0x7ff89a1fd2d0>
stream = <_io.StringIO object at 0x7ff899a01510>
template = <vcf.parser.Reader object at 0x7ff89a1fc290>, lineterminator = '\n'

    def __init__(self, stream, template, lineterminator="\n"):
>       self.writer = csv.writer(
            stream,
            delimiter="\t",
            lineterminator=lineterminator,
            quotechar="",
            quoting=csv.QUOTE_NONE,
        )
E       TypeError: "quotechar" must be a 1-character string

vcf/parser.py:776: TypeError
___________________ TestSampleFilter.testSampleFilterModule ____________________

self = <vcf.test.test_vcf.TestSampleFilter testMethod=testSampleFilterModule>

    @unittest.skipUnless(IS_NOT_PYPY, "test broken for PyPy")
    def testSampleFilterModule(self):
        # init filter with filename, get list of samples
        filt = vcf.SampleFilter('vcf/test/example-4.1.vcf')
        self.assertEqual(filt.samples, ['NA00001', 'NA00002', 'NA00003'])
        # set filter, check which samples will be kept
        filtered = filt.set_filters(filters="0", invert=True)
        self.assertEqual(filtered, ['NA00001'])
        # write filtered file to StringIO
        buf = StringIO()
>       filt.write(buf)

vcf/test/test_vcf.py:1482: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
vcf/sample_filter.py:109: in write
    writer = Writer(_out, self.parser)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <vcf.parser.Writer object at 0x7ff89b863a90>
stream = <_io.StringIO object at 0x7ff89a033910>
template = <vcf.parser.Reader object at 0x7ff8996bbd50>, lineterminator = '\n'

    def __init__(self, stream, template, lineterminator="\n"):
>       self.writer = csv.writer(
            stream,
            delimiter="\t",
            lineterminator=lineterminator,
            quotechar="",
            quoting=csv.QUOTE_NONE,
        )
E       TypeError: "quotechar" must be a 1-character string

vcf/parser.py:776: TypeError
----------------------------- Captured stdout call -----------------------------
Keeping these samples: ['NA00001']

Writing to '<_io.StringIO object at 0x7ff89a033910>'

------------------------------ Captured log call -------------------------------
INFO     root:sample_filter.py:96 Keeping these samples: ['NA00001']

INFO     root:sample_filter.py:108 Writing to '<_io.StringIO object at 0x7ff89a033910>'
________________________ TestRegression.test_null_mono _________________________

self = <vcf.test.test_vcf.TestRegression testMethod=test_null_mono>

    def test_null_mono(self):
        # null qualities were written as blank, causing subsequent parse to fail
        print(os.path.abspath(os.path.join(os.path.dirname(__file__),  'null_genotype_mono.vcf') ))
        p = vcf.Reader(fh('null_genotype_mono.vcf'))
        assert p.samples
        out = StringIO()
>       writer = vcf.Writer(out, p)

vcf/test/test_vcf.py:1559: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <vcf.parser.Writer object at 0x7ff899dea350>
stream = <_io.StringIO object at 0x7ff89a030820>
template = <vcf.parser.Reader object at 0x7ff899dea390>, lineterminator = '\n'

    def __init__(self, stream, template, lineterminator="\n"):
>       self.writer = csv.writer(
            stream,
            delimiter="\t",
            lineterminator=lineterminator,
            quotechar="",
            quoting=csv.QUOTE_NONE,
        )
E       TypeError: "quotechar" must be a 1-character string

vcf/parser.py:776: TypeError
----------------------------- Captured stdout call -----------------------------
/build/python-pyvcf3/src/PyVCF3-1.0.3/vcf/test/null_genotype_mono.vcf
__________________ TestUncalledGenotypes.test_write_uncalled ___________________

self = <vcf.test.test_vcf.TestUncalledGenotypes testMethod=test_write_uncalled>

    def test_write_uncalled(self):
        """Test that uncalled genotypes are written just as
        they were read in the input file."""

        reader = vcf.Reader(fh('uncalled_genotypes.vcf'))

        # Write all reader records to a stream.
        out = StringIO()
>       writer = vcf.Writer(out, reader, lineterminator='\n')

vcf/test/test_vcf.py:1697: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <vcf.parser.Writer object at 0x7ff899deb7d0>
stream = <_io.StringIO object at 0x7ff899a01cf0>
template = <vcf.parser.Reader object at 0x7ff899deba50>, lineterminator = '\n'

    def __init__(self, stream, template, lineterminator="\n"):
>       self.writer = csv.writer(
            stream,
            delimiter="\t",
            lineterminator=lineterminator,
            quotechar="",
            quoting=csv.QUOTE_NONE,
        )
E       TypeError: "quotechar" must be a 1-character string

vcf/parser.py:776: TypeError
=============================== warnings summary ===============================
vcf/parser.py:380
  /build/python-pyvcf3/src/PyVCF3-1.0.3/vcf/parser.py:380: DeprecationWarning: invalid escape sequence '\['
    self._alt_pattern = re.compile("[\[\]]")

vcf/parser.py:654
  /build/python-pyvcf3/src/PyVCF3-1.0.3/vcf/parser.py:654: DeprecationWarning: invalid escape sequence '\['
    remoteOrientation = re.search("\[", str) is not None

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
=========================== short test summary info ============================
FAILED vcf/test/test_vcf.py::Test1kgSites::test_writer - TypeError: "quotecha...
FAILED vcf/test/test_vcf.py::TestInfoOrder::test_writer - TypeError: "quotech...
FAILED vcf/test/test_vcf.py::TestInfoTypeCharacter::test_write - TypeError: "...
FAILED vcf/test/test_vcf.py::TestParseMetaLine::test_write - TypeError: "quot...
FAILED vcf/test/test_vcf.py::TestGatkOutputWriter::testWrite - TypeError: "qu...
FAILED vcf/test/test_vcf.py::TestBcfToolsOutputWriter::testWrite - TypeError:...
FAILED vcf/test/test_vcf.py::TestWriterDictionaryMeta::testWrite - TypeError:...
FAILED vcf/test/test_vcf.py::TestSampleFilter::testSampleFilterModule - TypeE...
FAILED vcf/test/test_vcf.py::TestRegression::test_null_mono - TypeError: "quo...
FAILED vcf/test/test_vcf.py::TestUncalledGenotypes::test_write_uncalled - Typ...
============ 10 failed, 83 passed, 11 skipped, 2 warnings in 3.55s =============
dmnfarrell commented 1 year ago

As far as I can see this package is now broken on Python>=3.11 which is default on Ubuntu 23.04. This means the library won't work going forward. When I try to write a vcf I get this error, (one of the test failures above).

  File "app.py", line 694, in overwrite_vcf
    vcf_writer = vcf.Writer(open(out, 'w'), vcf_reader)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".venv/lib/python3.11/site-packages/vcf/parser.py", line 776, in __init__
    self.writer = csv.writer(
                  ^^^^^^^^^^^
TypeError: "quotechar" must be a 1-character string

I imagine these errors can be fixed quite easily?

davmlaw commented 11 months ago

@dridk I think you can close this now pull request is merged

kbipinkumar commented 11 months ago

Can confirm we are able to build packages using latest git commit with tests completing without any errors. Thanks @davmlaw

hope to see a official release with fixes on pypi soon.

davmlaw commented 2 months ago

@dridk - could you please make an official release to PyPi?