EP Merge: Fix memory leak

[dfitchett]

What was the problem?

While working on #2783 to make the calls to datadog asynchronous, I noticed that the current implementation of EP Merge app uses creates an asyncio event loop but never closes this loop. The investigation/troubleshooting on that ticket, led me to realize we had a memory leak as was evidenced by the dashboard chart linked in the description.

I proceeded to run some tests locally to verify that was in fact the issue.

The 5 minute fixed rate tests demonstrated this effect. The tests sent 2 concurrent requests / second to the EP merge app deployed to a docker container. This allows independent tracking of memory and CPU usage similar to that of a container deployed on LHDI. Note, the traffic, however, is not similar because in production we hardly ever receive concurrent requests due to such a narrowed scope in eligible merges.

Tests

Cases:

1. Currently deployed functionality of not closing the asyncio event loop
2. Closing created event loop using asyncio.run(...)

Setup:

• Deployed via docker along with rabbitmq, postgres, and all dependent micro-services and their mocks (BIP, BGS)
  • Monitor memory usage of container
• Fixed rate:
  • 2 concurrent requests per second to submit merge jobs
  • 5 minutes

Results

Case 1: current deployment

• Starting Memory: 78 MB
• Ending Memory: 1.91 GB
• Requests: 625
• Error Rate: 0%
• Successful Merge Rate: 100%

Case 2: closing loop

• Starting Memory: 78 MB
• Ending Memory: 88 MB
• Requests: 621
• Error Rate: 0%
• Successful Merge Rate: 100%

How does this fix it?[^1]

Uses asyncio.run(...) which closes the event loop upon finishing coroutine.

How to test this PR

• pull code
• Unit Test:
  • from /domain-ee/ee-ep-merge-app run pytest
• Integration Tests:
  • run base platform
  • from /domain-ee/ee-ep-merge-app run pytest ./integration
• End to End tests:
  • run this CI action to verify successful integration test
  • run end to end tests locally:
  • export BIP_CLAIM_URL=mock-bip-claims-api:20300
  • run bgs and bip services and mocks:
    • COMPOSE_PROFILES=“all” ./gradlew dockerComposeUp
    • COMPOSE_PROFILES="bip,bgs" ./gradlew -p mocks :dockerComposeUp
  • ./gradlew :domain-ee:ee-ep-merge-app:endToEndTest

[github-actions[bot]]

Test Results

116 tests  ±0   116 :white_check_mark: ±0   49s :stopwatch: +9s  34 suites ±0     0 :zzz: ±0   34 files   ±0     0 :x: ±0 

Results for commit 1fa6e9dd. ± Comparison against base commit dffaadb2.

:recycle: This comment has been updated with latest results.

[github-actions[bot]]

JaCoCo Test Coverage

Overall Project	68.83%	:x:

There is no coverage information present for the Files changed