search

distributed-system-analysis / pbench

A benchmarking and performance analysis framework
http://distributed-system-analysis.github.io/pbench/
GNU General Public License v3.0
188 stars 108 forks source link

Intermittent unit test failure: it'd be great to figure out why and fix it... #3381

Closed portante closed 1 year ago

portante commented 1 year ago

I re-triggered. We've seen this unit test failure before; it'd be great to figure out why and fix it...

Traceback (most recent call last):
  File "/var/tmp/jenkins/tox/py39/lib/python3.9/site-packages/pbench/test/unit/server/auth/test_auth.py", line 599, in test_verify_auth_oidc_user_update
    assert user.id == "12345"
AttributeError: 'NoneType' object has no attribute 'id'
------------------------------ Captured log call -------------------------------
ERROR    TEST:auth.py:150 Unexpected exception occurred while verifying the auth token 'eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJpYXQiOjE2NTk0NzY3MDYsImV4cCI6Ijk5OTk5OTk5OTk5Iiwic3ViIjoiMTIzNDUiLCJhdWQiOiJ1cyIsInByZWZlcnJlZF91c2VybmFtZSI6ImR1bW15In0.VKXM9nepZ8UiFJZuRV51VglxRaTfDct5rmiX03seDYGQsRpEn-zrDcQnm0Txfdc_IZYjr1XByZIaUiV9C26UxMxF6Tc6TcQz_GDa2-6UW1cCqlTQlr2-xjbFTRSqYcOYigTUi8SWRaZeGlgJhAtwAUkP47IPCNsuvUIqTL__BZvgZICbOchQHlk0vv3DfqzW-nz5U8R3YBz6qmKmU_AV4d-cKZevP36KWn1KA-g62dMxf0tlq3QsU4KCKY8vBMYUZEQo20vXrfDkzMZnpi6aHJYp7Uhyxe2G9e37KrEWUl-leYrV-pqbY12CtTFKDKtwlSGlqCmyUp6Cs8ulOn1gdA': 'NoneType' object has no attribute 'query'
Traceback (most recent call last):
  File "/var/tmp/jenkins/tox/py39/lib/python3.9/site-packages/pbench/server/auth/auth.py", line 146, in verify_auth
    user = verify_auth_oidc(auth_token)
  File "/var/tmp/jenkins/tox/py39/lib/python3.9/site-packages/pbench/server/auth/auth.py", line 244, in verify_auth_oidc
    user = User.query(id=user_id)
  File "/var/tmp/jenkins/tox/py39/lib/python3.9/site-packages/pbench/server/database/models/users.py", line 132, in query
    dbsq = Database.db_session.query(User)
AttributeError: 'NoneType' object has no attribute 'query'

Originally posted by @dbutenhof in https://github.com/distributed-system-analysis/pbench/issues/3380#issuecomment-1507233423

webbnh commented 1 year ago

The cause seems to be that the Database.db_session somehow becomes None. The problem doesn't happen every time, so presumably it's due to some sort of interference from another test running in parallel. 🤷

dbutenhof commented 1 year ago

My suspicion is that it's implicitly depending on someone else's client or db_session fixture, but doesn't specify that dependency itself. In the CI parallel runs, it sometimes gets scheduled into a session where nobody else has done the setup, and fails.

E.g., note that the client fixture calls create_app which opens the database, and then app.app_context(), while these tests just call app.app_context().

dbutenhof commented 1 year ago

I decided to take a look at this, because it's causing repeated pain. I'm pretty sure I've verified my suspicion:

pytest -k test_verify_auth_oidc_user_update lib/pbench/test/unit/server/auth/test_auth.py

Fails reliably by isolating just that failing test.

Adding the db_session fixture explicitly makes it work. I'm going to figure out which other test cases require DB setup, and I'll push a PR.