Missing timeouts can lead to hanging clients and ressource exhaustion

dsmf commented 6 months ago

Link

related to https://github.com/eclipse-tractusx/item-relationship-service/issues/495

Description

There are several places in the code where no timeouts are defined potentially leading to hanging clients and resource exhaustion. See FIXME comments in the todos_no_timeouts.patch

Current behaviour

Expected behavior

Any request should timeout after some time to avoid hanging clients and resource exhaustion.

Steps to reproduce the Bug

I discovered this when I had a wiremock test with an incomplete setup resulting in tests that never finished. See 5fdbd0b8eb787fbe9f49996d46e264313eb4fabb (EdcSubmodelClientTest and SubmodelFacadeWiremockTest)

dsmf commented 6 months ago

Timeouts need to be defined
Timeouts should probably be configurable

dsmf commented 6 months ago

detected during implementation of #214 / #395

ds-jhartmann commented 6 months ago

Planning 2

add timeouts for the .get() of CompletableFuture
default to 10 Minutes (it is supposed to be a safety net)
make this configurable
same value for every place

dsmf commented 6 months ago

Pull Request: https://github.com/catenax-ng/tx-item-relationship-service/pull/808

Outcome:

all future.gets have timeout now
configured under irs-edc-client.asyncTimeout
default is PT10M
corrected README concerning local deployment MinIO connection problems and added some troubleshooting help (because I ran into problems here):
added helm chart deps and Chart.lock to gitignore

Note: Hard to simulate on DEV. The timeouts are a safety net to avoid jobs that hang forever. Probably all that can be tested on DEV is whether queries still return the expected results and none keeps hanging.

mkanal commented 5 months ago

{ "content": [ { "id": "4ecefbb4-73dc-48e5-a14d-b60249dfb0a3", "state": "RUNNING", "startedOn": "2024-03-19T11:37:17.551369625Z", "completedOn": null }, { "id": "6ca19e8e-0187-4443-bde8-9f3456875732", "state": "COMPLETED", "startedOn": "2024-03-19T11:37:05.075202498Z", "completedOn": "2024-03-19T11:37:19.257425867Z" }, { "id": "3a247122-e689-477a-87e3-dd39eaa58b03", "state": "COMPLETED", "startedOn": "2024-03-19T11:36:54.851223585Z", "completedOn": "2024-03-19T11:37:13.215424152Z" }, { "id": "37f4a0ec-df67-44b6-9dd4-2b2a73824955", "state": "COMPLETED", "startedOn": "2024-03-19T11:36:45.152558053Z", "completedOn": "2024-03-19T11:37:00.763422497Z" }, { "id": "a264cec3-b9bd-409d-95c5-ea086fc40f16", "state": "COMPLETED", "startedOn": "2024-03-19T11:36:12.053200965Z", "completedOn": "2024-03-19T11:36:30.938208297Z" }, { "id": "d550ba58-f941-41dc-b9d0-45648c8c0bf1", "state": "COMPLETED", "startedOn": "2024-03-19T11:35:59.705523133Z", "completedOn": "2024-03-19T11:36:16.916164668Z" }, { "id": "ea982f0f-3bb8-43db-93cc-128a68197f00", "state": "COMPLETED", "startedOn": "2024-03-19T11:35:51.691409474Z", "completedOn": "2024-03-19T11:36:07.894505466Z" }, { "id": "2bdf9ca8-5a50-446a-858f-2436290ace9d", "state": "COMPLETED", "startedOn": "2024-03-19T11:35:37.290989991Z", "completedOn": "2024-03-19T11:35:54.52777269Z" }, { "id": "4644f4e7-1ee8-4e79-9f5d-011c51273e0d", "state": "COMPLETED", "startedOn": "2024-03-19T11:35:32.521210719Z", "completedOn": "2024-03-19T11:35:50.923834653Z" }, { "id": "ecfc2d44-88d3-4d8f-872a-f6f7f0fb8e0c", "state": "COMPLETED", "startedOn": "2024-03-19T09:02:07.772711676Z", "completedOn": "2024-03-19T09:02:27.083143624Z" }, { "id": "c616dc5f-7982-47d9-bb20-8cd0ffd499a6", "state": "COMPLETED", "startedOn": "2024-03-19T09:01:28.793547556Z", "completedOn": "2024-03-19T09:02:03.200291495Z" }, { "id": "090675c8-daef-4549-ad70-2d1f78e9c1cd", "state": "COMPLETED", "startedOn": "2024-03-19T08:57:52.071902963Z", "completedOn": "2024-03-19T09:01:26.078805527Z" }, { "id": "ba874147-26d0-4c0c-8c80-fa85b9ed5209", "state": "COMPLETED", "startedOn": "2024-03-19T08:57:07.168453375Z", "completedOn": "2024-03-19T08:57:46.614504536Z" }, { "id": "7ff2588a-4877-4170-86c9-dbd92986b6d3", "state": "COMPLETED", "startedOn": "2024-03-19T08:55:48.792066048Z", "completedOn": "2024-03-19T08:57:00.89086984Z" }, { "id": "bc2b9a15-739a-4720-9605-80ea86d04ca4", "state": "COMPLETED", "startedOn": "2024-03-19T08:54:09.216539931Z", "completedOn": "2024-03-19T08:55:42.968767074Z" }, { "id": "1c5718b9-de5b-407f-a06b-bb4ca72ce646", "state": "COMPLETED", "startedOn": "2024-03-19T08:52:40.657911496Z", "completedOn": "2024-03-19T08:54:03.841710574Z" }, { "id": "e4a959ba-8450-4a21-beea-8a0ea56ca1c8", "state": "COMPLETED", "startedOn": "2024-03-19T08:50:43.988938955Z", "completedOn": "2024-03-19T08:52:33.779497136Z" }, { "id": "12d77d9d-1946-4025-9ddd-aab29e4bdbd0", "state": "COMPLETED", "startedOn": "2024-03-19T03:55:46.052974829Z", "completedOn": "2024-03-19T03:57:12.420469361Z" }, { "id": "22a3909c-a56d-4b6a-a0fe-0db468894194", "state": "COMPLETED", "startedOn": "2024-03-19T03:52:11.045948949Z", "completedOn": "2024-03-19T03:55:05.552308125Z" }, { "id": "9056cfb8-7dd8-4ecb-aa1b-3f4d57b6d4e2", "state": "COMPLETED", "startedOn": "2024-03-19T03:48:36.089712351Z", "completedOn": "2024-03-19T03:51:27.360563019Z" } ], "pageNumber": 0, "pageCount": 21, "pageSize": 20, "totalElements": 411 }

mkanal commented 5 months ago

@jzbmw I have tested that using insomnia collection. I am aware that this could not be tested manually in a sufficient way. I will bring the focus back to load and stress test scenarios so I have created a new issue to test this in an automatic way: https://github.com/eclipse-tractusx/item-relationship-service/issues/495

eclipse-tractusx / item-relationship-service