search

pytest-dev / pytest-xdist

pytest plugin for distributed testing and loop-on-failures testing modes.
https://pytest-xdist.readthedocs.io
MIT License
1.46k stars 232 forks source link

stuck replacing workers - maybe is related with pytest-timeout plugin? #1094

Open andreabisello opened 3 months ago

andreabisello commented 3 months ago

Take a look at that execution

C:\ec4a9867-4d56\qa-lsd-webservices>pytest --tc-file=configurations/utenze/eu-it1.ini --tc-file=configurations/puntamenti/legal.ini --dist=loadgroup -n auto --alluredir=allure-results/legal --force-flaky parametro -> configurations/utenze/eu-it1.ini parametro -> configurations/puntamenti/legal.ini ============================================================================================= test session starts ============================================================================================== platform win32 -- Python 3.11.6, pytest-7.4.4, pluggy-1.3.0 Suite di test dei webservices rootdir: C:\ec4a9867-4d56\qa-lsd-webservices configfile: pytest.ini plugins: allure-pytest-2.13.2, anyio-4.3.0, flaky-3.7.0, base-url-2.0.0, html-4.0.2, metadata-3.0.0, playwright-0.5.0, testconfig-0.2.0, timeout-2.2.0, xdist-3.6.1 timeout: 900.0s timeout method: thread timeout func_only: False 6 workers [1420 items] skipped scheduling tests via LoadGroupScheduling

test/test_api/test_metodi_conservazione.py::test_getAllSoggettiParam_e_count_disponibili_solo_per_amministratori_Sdc@serialize test/test_browser/test_admin/test_document_type/test_document_type.py::TestDocumentType::test_create_document_type_with_document_type_script[chromium] test/test_api/test_metodi_registrazione_admin_cliente.py::test_registraCliente_FatturazioneElettronicaB2B[chromium-contratto_senza_barcode.pdf] test/test_api/test_lta_rest_api.py::test_open_swagger_page[chromium] test/test_browser/test_admin/test_document_type/test_document_type.py::TestDocumentType::test_create_document_type_with_index_completion_script[chromium] test_upload_documento_esaw.py::TestEsawArchivePlugin::test_send_and_sign_a_document[chromium] [gw0] [ 0%] PASSED test/test_api/test_metodi_conservazione.py::test_getAllSoggettiParam_e_count_disponibili_solo_per_amministratori_Sdc@serialize test/test_api/test_metodi_conservazione.py::test_getAllAziende_getAzienda_UpdateAzienda@serialize [gw0] [ 0%] PASSED test/test_api/test_metodi_conservazione.py::test_getAllAziende_getAzienda_UpdateAzienda@serialize test/test_api/test_metodi_conservazione.py::test_visibilita_e_funzionalita_getUserByUserName@serialize [gw0] [ 0%] PASSED test/test_api/test_metodi_conservazione.py::test_visibilita_e_funzionalita_getUserByUserName@serialize test/test_api/test_metodi_conservazione.py::test_getAllPdvParamCount_confermami_lacount_dei_pdv@serialize

[a lot of log here]

[gw5] [ 40%] SKIPPED test/test_api/test_lta_rest_api.py::test_pre_pdv_combinazioni[True---False-False] test/test_api/test_lta_rest_api.py::test_pre_pdv_combinazioni[False-5ddfd582163b9b2e789d7563-5ddfd582163b9b2e789d7563-False-False] +++++++++++++++++++++++++++++++++++ Timeout ++++++++++++++++++++++++++++++++++++


PDF created successfully.

~~~~~~~~~~~~~~~~~~~~~~~~~~ Stack of asyncio_0 (16712) ~~~~~~~~~~~~~~~~~~~~~~~~~~
  File "C:\Python311\Lib\threading.py", line 1002, in _bootstrap
    self._bootstrap_inner()
  File "C:\Python311\Lib\threading.py", line 1045, in _bootstrap_inner
    self.run()
  File "C:\Python311\Lib\threading.py", line 982, in run
    self._target(*self._args, **self._kwargs)
  File "C:\Python311\Lib\concurrent\futures\thread.py", line 81, in _worker
    work_item = work_queue.get(block=True)

~~~~~~~~~~~~~~~~~~~~~~~~~~ Stack of <unknown> (14536) ~~~~~~~~~~~~~~~~~~~~~~~~~~
  File "C:\Python311\Lib\site-packages\execnet\gateway_base.py", line 411, in _perform_spawn
    reply.run()
  File "C:\Python311\Lib\site-packages\execnet\gateway_base.py", line 341, in run
    self._result = func(*args, **kwargs)
  File "C:\Python311\Lib\site-packages\execnet\gateway_base.py", line 1160, in _thread_receiver
    msg = Message.from_io(io)
  File "C:\Python311\Lib\site-packages\execnet\gateway_base.py", line 567, in from_io
    header = io.read(9)  # type 1, channel 4, payload 4
  File "C:\Python311\Lib\site-packages\execnet\gateway_base.py", line 534, in read
    data = self._read(numbytes - len(buf))

~~~~~~~~~~~~~~~~~~~~~~~~~ Stack of MainThread (26152) ~~~~~~~~~~~~~~~~~~~~~~~~~~
  File "C:\Python311\Lib\site-packages\playwright\sync_api\_context_manager.py", line 56, in greenlet_main
    self._loop.run_until_complete(self._connection.run_as_sync())
  File "C:\Python311\Lib\asyncio\base_events.py", line 640, in run_until_complete
    self.run_forever()
  File "C:\Python311\Lib\asyncio\windows_events.py", line 321, in run_forever
    super().run_forever()
  File "C:\Python311\Lib\asyncio\base_events.py", line 607, in run_forever
    self._run_once()
  File "C:\Python311\Lib\asyncio\base_events.py", line 1884, in _run_once
    event_list = self._selector.select(timeout)
  File "C:\Python311\Lib\asyncio\windows_events.py", line 444, in select
    self._poll(timeout)
  File "C:\Python311\Lib\asyncio\windows_events.py", line 817, in _poll
    status = _overlapped.GetQueuedCompletionStatus(self._iocp, ms)

+++++++++++++++++++++++++++++++++++ Timeout ++++++++++++++++++++++++++++++++++++

[gw3] [ 40%] PASSED test/test_api/test_lta_rest_api.py::test_pre_pdv_combinazioni[True---False-True]
test/test_api/test_lta_rest_api.py::test_pre_pdv_combinazioni[False-5ddfd582163b9b2e789d7563-5ddfd582163b9b2e789d7563-True-False]
[gw0] [ 40%] PASSED test/test_api/test_lta_rest_api.py::test_pre_pdv_combinazioni[True--5ddfd582163b9b2e789d7563-False-True]
test/test_api/test_lta_rest_api.py::test_pre_pdv_combinazioni[True---True-False]
[gw1] node down: Not properly terminated
[gw1] [ 40%] FAILED test_upload_documento_esaw.py::TestEsawArchivePlugin::test_send_and_sign_a_document[chromium]

replacing crashed worker gw1
collecting: 6/7 workers[gw2] [ 40%] PASSED test/test_api/test_lta_rest_api.py::test_pre_pdv_combinazioni[False-5ddfd582163b9b2e789d7563-5ddfd582163b9b2e789d7563-True-True]

[a lot of log here]

[gw6] [100%] PASSED test/test_api/test_lta_rest_api_sensible_document.py::test_cerca_per_metadata_equal_to
+++++++++++++++++++++++++++++++++++ Timeout ++++++++++++++++++++++++++++++++++++

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Captured stdout ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
PDF created successfully.

~~~~~~~~~~~~~~~~~~~~~~~~~~ Stack of asyncio_0 (14848) ~~~~~~~~~~~~~~~~~~~~~~~~~~
  File "C:\Python311\Lib\threading.py", line 1002, in _bootstrap
    self._bootstrap_inner()
  File "C:\Python311\Lib\threading.py", line 1045, in _bootstrap_inner
    self.run()
  File "C:\Python311\Lib\threading.py", line 982, in run
    self._target(*self._args, **self._kwargs)
  File "C:\Python311\Lib\concurrent\futures\thread.py", line 81, in _worker
    work_item = work_queue.get(block=True)

~~~~~~~~~~~~~~~~~~~~~~~~~~ Stack of <unknown> (6052) ~~~~~~~~~~~~~~~~~~~~~~~~~~~
  File "C:\Python311\Lib\site-packages\execnet\gateway_base.py", line 411, in _perform_spawn
    reply.run()
  File "C:\Python311\Lib\site-packages\execnet\gateway_base.py", line 341, in run
    self._result = func(*args, **kwargs)
  File "C:\Python311\Lib\site-packages\execnet\gateway_base.py", line 1160, in _thread_receiver
    msg = Message.from_io(io)
  File "C:\Python311\Lib\site-packages\execnet\gateway_base.py", line 567, in from_io
    header = io.read(9)  # type 1, channel 4, payload 4
  File "C:\Python311\Lib\site-packages\execnet\gateway_base.py", line 534, in read
    data = self._read(numbytes - len(buf))

~~~~~~~~~~~~~~~~~~~~~~~~~~ Stack of MainThread (4196) ~~~~~~~~~~~~~~~~~~~~~~~~~~
  File "C:\Python311\Lib\site-packages\playwright\sync_api\_context_manager.py", line 56, in greenlet_main
    self._loop.run_until_complete(self._connection.run_as_sync())
  File "C:\Python311\Lib\asyncio\base_events.py", line 640, in run_until_complete
    self.run_forever()
  File "C:\Python311\Lib\asyncio\windows_events.py", line 321, in run_forever
    super().run_forever()
  File "C:\Python311\Lib\asyncio\base_events.py", line 607, in run_forever
    self._run_once()
  File "C:\Python311\Lib\asyncio\base_events.py", line 1884, in _run_once
    event_list = self._selector.select(timeout)
  File "C:\Python311\Lib\asyncio\windows_events.py", line 444, in select
    self._poll(timeout)
  File "C:\Python311\Lib\asyncio\windows_events.py", line 817, in _poll
    status = _overlapped.GetQueuedCompletionStatus(self._iocp, ms)

+++++++++++++++++++++++++++++++++++ Timeout ++++++++++++++++++++++++++++++++++++

[gw0] node down: Not properly terminated
[gw0] [100%] FAILED test_upload_documento_esaw.py::TestEsawArchivePlugin::test_send_and_sign_a_document[chromium]

replacing crashed worker gw0
8 workers [1420 items]

at some points, after a timeout (https://pypi.org/project/pytest-timeout/) a worker were replaced, but for pytest now workers are 8, not 6 as the beginning, and nothing happening for 30 minutes

image

what other informations could i share to make this situation actionable? thanks for help