search

ORNL / flowcept

Runtime data integration system that empowers any data processing system to capture and query workflow provenance using data observability.
MIT License
1 stars 3 forks source link

Intermittent error when running MacOS CI tests, which only happen in the CI #113

Open renan-souza opened 6 months ago

renan-souza commented 6 months ago

Error seems to happen when creating the index, but I'm not sure if it's something related to index creation. It seems to be an error with the mongo connection. See full log below or see the failed job.

    self._tasks_collection.create_index(TaskObject.task_id_field())

Attempting to run it again makes the error go away: https://github.com/ORNL/flowcept/actions/runs/8022376166

This has low priority because it only happens in the CI and with the MacOS job only.

==================================== ERRORS ==================================== ERROR collecting tests/decorator_tests/flowcept_task_decorator_test.py tests/decorator_tests/flowcept_task_decorator_test.py:6: in import flowcept.instrumentation.decorators flowcept/instrumentation/decorators/init.py:9: in instrumentation_interceptor = BaseInterceptor(plugin_key=None) flowcept/flowceptor/adapters/base_interceptor.py:48: in init self._db_api = DBAPI() flowcept/flowcept_api/db_api.py:27: in init self._dao = DocumentDBDao() flowcept/commons/daos/document_db_dao.py:44: in init self._tasks_collection.create_index(TaskObject.task_id_field()) ../../../hostedtoolcache/Python/3.10.13/x64/lib/python3.10/site-packages/pymongo/collection.py:2055: in create_index return self.create_indexes([index], session, *cmd_options)[0] ../../../hostedtoolcache/Python/3.10.13/x64/lib/python3.10/site-packages/pymongo/_csot.py:105: in csot_wrapper return func(self, args, **kwargs) ../../../hostedtoolcache/Python/3.10.13/x64/lib/python3.10/site-packages/pymongo/collection.py:1911: in create_indexes with self._socket_for_writes(session) as sock_info: ../../../hostedtoolcache/Python/3.10.13/x64/lib/python3.10/site-packages/pymongo/collection.py:248: in _socket_for_writes return self.database.client._socket_for_writes(session) ../../../hostedtoolcache/Python/3.10.13/x64/lib/python3.10/site-packages/pymongo/mongo_client.py:1267: in _socket_for_writes server = self._select_server(writable_server_selector, session) ../../../hostedtoolcache/Python/3.10.13/x64/lib/python3.10/site-packages/pymongo/mongo_client.py:1257: in _select_server server = topology.select_server(server_selector) ../../../hostedtoolcache/Python/3.10.13/x64/lib/python3.10/site-packages/pymongo/topology.py:272: in select_server server = self._select_server(selector, server_selection_timeout, address) ../../../hostedtoolcache/Python/3.10.13/x64/lib/python3.10/site-packages/pymongo/topology.py:261: in _select_server servers = self.select_servers(selector, server_selection_timeout, address) ../../../hostedtoolcache/Python/3.10.13/x64/lib/python3.10/site-packages/pymongo/topology.py:223: in select_servers server_descriptions = self._select_servers_loop(selector, server_timeout, address) ../../../hostedtoolcache/Python/3.10.13/x64/lib/python3.10/site-packages/pymongo/topology.py:238: in _select_servers_loop raise ServerSelectionTimeoutError( E pymongo.errors.ServerSelectionTimeoutError: localhost:27017: [Errno 61] Connection refused, Timeout: 30s, Topology Description: <TopologyDescription id: 65d8ceb527d0aa6805fd1915, topology_type: Unknown, servers: [<ServerDescription ('localhost', 27017) server_type: Unknown, rtt: None, error=AutoReconnect('localhost:27017: [Errno 61] Connection refused')>]> ERROR collecting tests/decorator_tests/ml_tests/ml_decorator_dasktest.py tests/decorator_tests/ml_tests/ml_decorator_dask_test.py:16: in from tests.decorator_tests.ml_tests.dl_trainer import ModelTrainer tests/decorator_tests/ml_tests/dl_trainer.py:7: in import flowcept.instrumentation.decorators flowcept/instrumentation/decorators/init.py:9: in instrumentation_interceptor = BaseInterceptor(plugin_key=None) flowcept/flowceptor/adapters/base_interceptor.py:48: in init self._db_api = DBAPI() flowcept/flowcept_api/db_api.py:27: in init self._dao = DocumentDBDao() flowcept/commons/daos/document_db_dao.py:44: in init self._tasks_collection.create_index(TaskObject.task_id_field()) ../../../hostedtoolcache/Python/3.10.13/x64/lib/python3.10/site-packages/pymongo/collection.py:2055: in create_index return self.create_indexes([index], session, cmd_options)[0] ../../../hostedtoolcache/Python/3.10.13/x64/lib/python3.10/site-packages/pymongo/_csot.py:105: in csot_wrapper return func(self, args, kwargs) ../../../hostedtoolcache/Python/3.10.13/x64/lib/python3.10/site-packages/pymongo/collection.py:1911: in create_indexes with self._socket_for_writes(session) as sock_info: ../../../hostedtoolcache/Python/3.10.13/x64/lib/python3.10/site-packages/pymongo/collection.py:248: in _socket_for_writes return self.database.client._socket_for_writes(session) ../../../hostedtoolcache/Python/3.10.13/x64/lib/python3.10/site-packages/pymongo/mongo_client.py:1267: in _socket_for_writes server = self._select_server(writable_server_selector, session) ../../../hostedtoolcache/Python/3.10.13/x64/lib/python3.10/site-packages/pymongo/mongo_client.py:1257: in _select_server server = topology.select_server(server_selector) ../../../hostedtoolcache/Python/3.10.13/x64/lib/python3.10/site-packages/pymongo/topology.py:272: in select_server server = self._select_server(selector, server_selection_timeout, address) ../../../hostedtoolcache/Python/3.10.13/x64/lib/python3.10/site-packages/pymongo/topology.py:261: in _select_server servers = self.select_servers(selector, server_selection_timeout, address) ../../../hostedtoolcache/Python/3.10.13/x64/lib/python3.10/site-packages/pymongo/topology.py:223: in select_servers server_descriptions = self._select_servers_loop(selector, server_timeout, address) ../../../hostedtoolcache/Python/3.10.13/x64/lib/python3.10/site-packages/pymongo/topology.py:238: in _select_servers_loop raise ServerSelectionTimeoutError( E pymongo.errors.ServerSelectionTimeoutError: localhost:27017: [Errno 61] Connection refused, Timeout: 30s, Topology Description: <TopologyDescription id: 65d8ced627d0aa6805fd1916, topology_type: Unknown, servers: [<ServerDescription ('localhost', 27017) servertype: Unknown, rtt: None, error=AutoReconnect('localhost:27017: [Errno 61] Connection refused')>]> ____ ERROR collecting tests/decorator_tests/ml_tests/ml_decoratortest.py ____ tests/decorator_tests/ml_tests/ml_decorator_test.py:7: in from tests.decorator_tests.ml_tests.dl_trainer import ( tests/decorator_tests/ml_tests/dl_trainer.py:7: in import flowcept.instrumentation.decorators flowcept/instrumentation/decorators/init.py:9: in instrumentation_interceptor = BaseInterceptor(plugin_key=None) flowcept/flowceptor/adapters/base_interceptor.py:48: in init self._db_api = DBAPI() flowcept/flowcept_api/db_api.py:27: in init self._dao = DocumentDBDao() flowcept/commons/daos/document_db_dao.py:44: in init self._tasks_collection.create_index(TaskObject.task_id_field()) ../../../hostedtoolcache/Python/3.10.13/x64/lib/python3.10/site-packages/pymongo/collection.py:2055: in create_index return self.__create_indexes([index], session, cmd_options)[0] ../../../hostedtoolcache/Python/3.10.13/x64/lib/python3.10/site-packages/pymongo/_csot.py:105: in csot_wrapper return func(self, args, kwargs) ../../../hostedtoolcache/Python/3.10.13/x64/lib/python3.10/site-packages/pymongo/collection.py:1911: in create_indexes with self._socket_for_writes(session) as sock_info: ../../../hostedtoolcache/Python/3.10.13/x64/lib/python3.10/site-packages/pymongo/collection.py:248: in _socket_for_writes return self.database.client._socket_for_writes(session) ../../../hostedtoolcache/Python/3.10.13/x64/lib/python3.10/site-packages/pymongo/mongo_client.py:1267: in _socket_for_writes server = self._select_server(writable_server_selector, session) ../../../hostedtoolcache/Python/3.10.13/x64/lib/python3.10/site-packages/pymongo/mongo_client.py:1257: in _select_server server = topology.select_server(server_selector) ../../../hostedtoolcache/Python/3.10.13/x64/lib/python3.10/site-packages/pymongo/topology.py:272: in select_server server = self._select_server(selector, server_selection_timeout, address) ../../../hostedtoolcache/Python/3.10.13/x64/lib/python3.10/site-packages/pymongo/topology.py:261: in _select_server servers = self.select_servers(selector, server_selection_timeout, address) ../../../hostedtoolcache/Python/3.10.13/x64/lib/python3.10/site-packages/pymongo/topology.py:223: in select_servers server_descriptions = self._select_servers_loop(selector, server_timeout, address) ../../../hostedtoolcache/Python/3.10.13/x64/lib/python3.10/site-packages/pymongo/topology.py:238: in _select_servers_loop raise ServerSelectionTimeoutError( E pymongo.errors.ServerSelectionTimeoutError: localhost:27017: [Errno 61] Connection refused, Timeout: 30s, Topology Description: <TopologyDescription id: 65d8cef527d0aa6805fd1917, topology_type: Unknown, servers: [<ServerDescription ('localhost', 27017) server_type: Unknown, rtt: None, error=AutoReconnect('localhost:27017: [Errno 61] Connection refused')>]> =========================== short test summary info ============================ ERROR tests/decorator_tests/flowcept_task_decorator_test.py - pymongo.errors.... ERROR tests/decorator_tests/ml_tests/ml_decorator_dask_test.py - pymongo.erro... ERROR tests/decorator_tests/ml_tests/ml_decorator_test.py - pymongo.errors.Se...

renan-souza commented 6 months ago

This might be a related error, with Docker now.

https://github.com/ORNL/flowcept/actions/runs/8022831523/job/21917999558