Profile memory consumption of harvest runner under load - Githubissues

GSA / data.gov

Main repository for the data.gov service

https://data.gov

Other

649 stars 101 forks source link

Profile memory consumption of harvest runner under load #4875

Open btylerburton opened 2 months ago

btylerburton commented 2 months ago

User Story

In order to identify memory leaks in the harvester, datagovteam wants to conduct a formal analysis using industry standard memory profiling tools.

Acceptance Criteria

[ACs should be clearly demoable/verifiable whenever possible. Try specifying them using BDD.]

[ ] GIVEN I have integrated a memory profiler into the datagov-harvester codebase AND it has been tested locally and show to perform THEN I want to push it to cloud.gov and run a proper load test of a large harvest source, such as DOI.

Background

[Any helpful contextual notes or links to artifacts/evidence, if needed]

Security Considerations (required)

[Any security concerns that might be implicated in the change. "None" is OK, just be explicit here!]

Sketch

[ ] Select a memory profiler library
- [ ] such as https://github.com/bloomberg/memray
Confirm it outputs meaningful data locally
Push to cloud.gov
Test on a small source
Confirm output is meaningful
Test on DOI source

btylerburton commented 2 months ago

Discussed whether breaking the harvest up into discrete processes: extract, transform, validate, sync/load will make this a non-issue. We can leave this ticket as a might-do just in case that doesn't work.