Open kloczek opened 2 years ago
Here is pytest output. At the enad are some pytest warnings as well.
Version 4.1.1 is 1.5 years old. Maybe try grabbing from master?
Just tested new 4.2.0. I see some inmprovements as less units now are failing.
As you see there are some warnings about use rfc3986
module.
Just tested 4.2.1. Only will quote summary
Do you have aby idea why those units may be failing? And/or is it anything else what I can try to diagnose those fails? 🤔
gentle ping .. 🤔
Summary from testing latest 4.3.0
@kloczek there is a probably a problem in how you run the tests, more on that below. Here is how to see the tests pass with plain pytest (meaning without tox unlike vcrpy's CI), starting with an empty venv:
cd "$(mktemp -d)"
wget https://github.com/kevin1024/vcrpy/archive/refs/tags/v4.3.0.tar.gz
tar xf v4.3.0.tar.gz
cd vcrpy-4.3.0/
python3.10 -m venv venv
source venv/bin/activate
pip install -e .
pip install pytest aiohttp boto3 httpx requests pytest-httpbin tornado httplib2 pytest-aiohttp 'urllib3<2' Werkzeug==2.0.3 # NOTE LAST TWO!
REQUESTS_CA_BUNDLE=`python -m pytest_httpbin.certs` pytest -ra # NOTE REQUESTS_CA_BUNDLE!
Note the two things marked with NOTE
in that^^ block.
@kloczek please run the tests this very way and report back any failures. We can then work from there. Thanks :pray:
First of all python -m foo
should be never used because python when starts module that way add current directory to sys.path
, This is why pytest
module provides wrapper script.
No use 'python -m pytest` was discussed hundreds times on pytest forum and maintained many times told that running pytest that way in nothing more than asking for troubles.
Second: I'm not interested to test any module in source tree because after packaging exact module will be NEVER used that way. This is why it was developed "testing as installed" methodology.
Third: I'm not interested to test any module against exact version of other modules taken from pypi.
I'm trying to form set of packaged modules which will be working together in that set. This is why I've added list of modules with versions installed in build env during the testing vcrpy
.
Currently that set has +1.13k of packaged modules which are mostly latest version of all modules in that set.
Latest version ot the werkzeug
is not 2.0.0 but 2.3.4 (I'll be soon testing everything against 2.3.4).
2.0.0 has been released almost year ago and since than many things has been fixed in that module.
I'm not going to roll back to 2.0.0 in my distribution because this will break many other modules.
Sooner or later you will be probably this or another way forced to use latest version of the werkzeug
so maybe now is the best moment to have look on that.
FYI: I'm not complaining or trying to you to force to upgrade werkzeug
to latest version.
I'm only reporting how testing with exact set of modules behaves.
Just please try to understand that my goals are embedded in much broader landscape.
Just checked and looks like werkzeug
is not on the install time vcrpy
dependencies list so if werkzeug
is causing that issue looks like it is only test suite issue.
Hi @kloczek,
I don't see how your criticism of python -m
applies to our case. All that python -m pytest_httpbin.certs
as used above does is produce a filename to a certificate to be put into the environment, e.g. see:
# python -m pytest_httpbin.certs
/tmp/tmp.MLDZlkbUla/vcrpy-4.3.0/venv/lib/python3.10/site-packages/pytest_httpbin/certs/cacert.pem
The call to pytest itself did not use python -m
, see above. I don't see how that would be a problem to any test environment, including distro packging. I've been packaging for Linux distros myself, I understand what situation packaging and Linux distros are in regarding Python dependencies. Please let me know what I am missing or misunderstanding here.
The pinning of Werkzeug is unfortunate and caused by httpbin (https://github.com/postmanlabs/httpbin/issues/673) which is pulled in by pytest-httpbin (https://github.com/kevin1024/pytest-httpbin/issues/72). Given that httpbin's last release is of 2018, mid-term fix upstream here is probably to replace all use of httpbin (and httpbin.org) in the current test suite by something else. I will edit the title of this ticket now to better reflect what needs to be done. A short-term fix downstream for Fedora could be fixing httpbin packaging to support Werkzeug >=2.1.0 using pull request https://github.com/postmanlabs/httpbin/pull/674/files which is what Gentoo did.
CC @kevin1024 @jairhenrique
Hmm. I’m a maintainer on httpbin and might be able to move things along over there.
don't see how your criticism of python -m applies to our case.
I'm not criticising anything or trying to connect this case to use 'python -m foo`. I'm only informing you that pytest generally should be used not that way how you are using it. Only this and nothing more 😋
Hmm. I’m a maintainer on httpbin and might be able to move things along over there.
BTW using certs: IMO it would be good to abandon use custom copies of CA and use system CA (usually now it is /etc/pki/tls/certs/ca-bundle.crt) and/or use certs over certifi
modules (which could be easily centrally patched to use system system /etc/pki/tls/certs/ca-bundle.crt).
Just checked my rpm spec filers library which one modules currently are using certify
[tkloczko@pers-jacek SPECS]$ grep "python3dist(certifi)" python*
python-aiohttp-cors.spec:BuildRequires: python3dist(certifi)
python-branca.spec:BuildRequires: python3dist(certifi)
python-elastic-transport.spec:BuildRequires: python3dist(certifi)
python-geventhttpclient.spec:BuildRequires: python3dist(certifi)
python-google-auth.spec:BuildRequires: python3dist(certifi)
python-httpcore.spec:BuildRequires: python3dist(certifi)
python-httpx.spec:BuildRequires: python3dist(certifi)
python-jsonschema.spec:BuildRequires: python3dist(certifi)
python-matplotlib.spec:BuildRequires: python3dist(certifi)
python-notebook.spec:BuildRequires: python3dist(certifi)
python-pdm.spec:BuildRequires: python3dist(certifi)
python-pip.spec:Provides: bundled(python3dist(certifi)) = 2022.12.7
python-pyppeteer.spec:BuildRequires: python3dist(certifi)
python-pyproj.spec:BuildRequires: python3dist(certifi)
python-requests-file.spec:BuildRequires: python3dist(certifi)
python-selenium.spec:BuildRequires: python3dist(certifi)
python-seleniumbase.spec:BuildRequires: python3dist(certifi) >= 2021.10.8
python-sphinx-json-schema-spec.spec:BuildRequires: python3dist(certifi)
python-sphinxext-rediraffe.spec:BuildRequires: python3dist(certifi)
python-sphobjinv.spec:BuildRequires: python3dist(certifi)
Hmm. I’m a maintainer on httpbin and might be able to move things along over there.
@kevin1024 that would rock the house, yes please! :+1: :+1:
I'm not criticising anything or trying to connect this case to use 'python -m foo`. I'm only informing you that pytest generally should be used not that way how you are using it. Only this and nothing more :yum:
@kloczek except there was zero python -m pytest
in here above before you brought it up. Zero, I checked.
Just FTR. Here is summary pytest output on testing 5.0.0:
Full log is in attachment python-vcrpy-pytest.txt
Here is list of installed modules in build env
Am I passing wrong path to ca bundle? (REQUESTS_CA_BUNDLE=/etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
) 🤔
After change to REQUESTS_CA_BUNDLE=/etc/pki/tls/certs/ca-bundle.crt
pytest fails the same way.
@kloczek that question has been answered by https://github.com/kevin1024/vcrpy/issues/645#issuecomment-1561933604 already, I believe.
@kloczek that question has been answered by #645 (comment) already, I believe.
Only detail which I see in that reply is use Werkzeug==2.0.3
(please correct me if I'm.
I'm using 2.3.6.
Is this cause of the fails?
If yes: is it any plan to update to be able use latest version of the Werkzeug
?
@kloczek you were asking for REQUESTS_CA_BUNDLE
. The answer is:
REQUESTS_CA_BUNDLE=`python -m pytest_httpbin.certs`
@kloczek regarding Werkzeug please see the discussion above and https://github.com/gentoo/gentoo/blob/5ca1af4f70b22a042102c08858746e174376725a/dev-python/vcrpy/vcrpy-5.0.0.ebuild#L43-L49C3 .
@kloczek you were asking for
REQUESTS_CA_BUNDLE
. The answer is:REQUESTS_CA_BUNDLE=`python -m pytest_httpbin.certs`
[tkloczko@pers-jacek SPECS]$ python3 -m pytest_httpbin.certs
/usr/lib/python3.8/site-packages/pytest_httpbin/certs/cacert.pem
Hmm I must patch that module to use cert file from /ets/pki. After pass correct file
Hmm I must patch that module to use cert file from /ets/pki.
@kloczek why do you feel there is need for that? I would expected that the httpbin package would be the place to patch then if these certificates would be a problem (I doubt they are), not VCR.py — am I missing something?
After pass correct file
Part of these errors is cut off and some of the errors seems related to use of the wrong certificates. The instructions at https://github.com/kevin1024/vcrpy/issues/645#issuecomment-1561933604 could hardly be more detailed about how to get the tests to pass. The instructions still work for 5.0.0 for me, I just tried:
cd "$(mktemp -d)"
wget https://github.com/kevin1024/vcrpy/archive/refs/tags/v5.0.0.tar.gz
tar xf v5.0.0.tar.gz
cd vcrpy-5.0.0/
python3.10 -m venv venv
source venv/bin/activate
pip install -e .
pip install pytest aiohttp boto3 httpx requests pytest-httpbin tornado httplib2 pytest-aiohttp 'urllib3<2' Werkzeug==2.0.3 # NOTE LAST TWO!
REQUESTS_CA_BUNDLE=`python -m pytest_httpbin.certs` pytest -ra # NOTE REQUESTS_CA_BUNDLE!
For anything that works in this context but not in your packaging context, please track down the difference and cause on your side and create new issues with clear details as I would do for the packages I maintain in Gentoo, so that we can move forward, keep this ticket on topic about werkzeug. Thank you.
@kloczek why do you feel there is need for that? I would expected that the httpbin package would be the place to patch then if these certificates would be a problem (I doubt they are), not VCR.py — am I missing something?
No one knows why long current version of exact piece of software with own copy of ca-bundle will be used.
If exact software is using system wide PKI keys/certs it iw possible to maintain that for ALL software.
I'm not going to submit httpbin
module patch because exact location could be OS distribution dependent it is matter of those who are maintaining OS distros to sync that location across all packages in distribution.
I can only say "thank you" that accidentally I found something which is inconsistent in some resources according to some distro goals. 🙇♂️
To be honest I've been thinking from some time to open python RFE to provide for example sys.cert
which could solve all that kind dilemmas in single point in python ecosystem with CA path (I think that will try to do that today).
For anything that works in this context but not in your packaging context, please track down the difference and cause on your side and create new issues with clear details as I would do for the packages I maintain in Gentoo, so that we can move forward, keep this ticket on topic about werkzeug. Thank you.
I'm aware that I'm usually using way freshen versions of some modules. All what I'm doing it is reporting that on use those version it is possible to see some error/fails.
In other words If someone is thinking that I'm not asking to fix those issue instantly. God .. NO!!! I'm only flagging some issues in exact env .. only this and nothing more.
Because this ticket still is opened I'm assuming that at least some of those issues are real but maybe time is not the best to address them now.
As long as in python packages on which I'm working there is obligatory enabled test suite execution + some set of external python test units with high probability I can assume all/most of those errors/fails are related to vcrpy
test suites and not tested vcrpy
code.
[tkloczko@pers-jacek SPECS]$ ls python-*spec | wc -l; grep ^%pytest python-*spec | wc -l
1163
1146
From that point of view this conversation allows me to assign importance of this ticket in relation to whole distribution as non-critical as well because I'm not able to point any other known me issues with vcrpy
module when it is used with other modules build/install/test procedures.
And one more time: Thank you very much for provide me some really crucial details which allowed me to improve not only vcrpy
module package build/install/testing procedure but as well expose some other issues in other modules 👍
Because I'm working +2.5 years on all those python modules I've learned al lot about that area however because I'm maliny specialist about packaging stuff I'm still opened on any suggestions how can I improve way of reporting python modules issues found during my work.
Sorry for a long comment but I really want to be only well understood ..
I have pushed a new version of the httpbin package that has compatibility with updated versions of werkzeug. We needed to fork the httpbin package due to lack of response from the repo owner.
Does that help with this issue?
@kevin1024 interesting, thanks! I guess it's a start. Can you point me to the Git repository backing the new 0.10.0 release? I think we should at least update the PyPI link "Homepage" at https://pypi.org/project/httpbin/ to point to the new repository.
Good call! It’s here: https://github.com/psf/httpbin
@kevin1024 thanks!
I think there is a small mixup in reated commit https://github.com/psf/httpbin/commit/e7d491f907a9cd39031904371f112c7cc1853791: The message says "Update github URL to point to the fork" but the new URL does not point to the Git repository but the PyPI page (where the user is already located). Is there a chance that you could update it once more for https://github.com/psf/httpbin ?
I just moved to python 3.9 and pytest 8.1.1 and now I have different set of failing units
I'm trying to package your module as an rpm package. So I'm using the typical PEP517 based build, install and test cycle used on building packages from non-root account.
python3 -sBm build -w --no-isolation
build
with--no-isolation
I'm using during all processes only locally installed modules