DTIN-3938: Accommodate busy k8s clusters with short-lived pods

[jmakar-s1]

• CRIEnumerator._get_containers do not cache unavailable (404) pods if querying by filesystem (see comments)
• Change max_allowed_checkpoint_age to be a proper / configurable option
• Added note about making _K8sCache a fixed-size LRU cache

[github-actions[bot]]

Test Results

     20 files  ±0       20 suites  ±0   31m 19s :stopwatch: -3s 1 489 tests ±0  1 469 :heavy_check_mark: ±0    20 :zzz: ±0  0 :x: ±0  7 116 runs  ±0  6 889 :heavy_check_mark: ±0  227 :zzz: ±0  0 :x: ±0 

Results for commit 0ef47cf5. ± Comparison against base commit c3d23701.

:recycle: This comment has been updated with latest results.

[alesnovak-s1]

Some of the kubernetes integration test are failing, it seems it's related to short-lived pods. You can ingore the linux packages tests, there's a know issue, which will be handled soon.

[jmakar-s1]

Some of the kubernetes integration test are failing, it seems it's related to short-lived pods. You can ingore the linux packages tests, there's a know issue, which will be handled soon.

Ok, unit test is now fixed, ready for another look

[alesnovak-s1]

Some of the kubernetes integration test are failing, it seems it's related to short-lived pods. You can ingore the linux packages tests, there's a know issue, which will be handled soon.

Ok, unit test is now fixed, ready for another look

Thank you. Can you take a look at the k8s integration tests as well?

i.e. https://github.com/scalyr/scalyr-agent-2/actions/runs/9175077249/job/25228143728

[codecov[bot]]

Codecov Report

Attention: Patch coverage is 66.66667% with 2 lines in your changes are missing coverage. Please review.

Project coverage is 54.77%. Comparing base (b514d71) to head (03cd9c9). Report is 5 commits behind head on master.

:exclamation: Current head 03cd9c9 differs from pull request most recent head 0ef47cf

Please upload reports for the commit 0ef47cf to get more accurate results.

Additional details and impacted files

[](https://app.codecov.io/gh/scalyr/scalyr-agent-2/pull/1263?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=scalyr) ```diff @@ Coverage Diff @@ ## master #1263 +/- ## ========================================== - Coverage 54.83% 54.77% -0.06% ========================================== Files 51 51 Lines 18666 18597 -69 Branches 3294 3286 -8 ========================================== - Hits 10235 10186 -49 + Misses 7232 7225 -7 + Partials 1199 1186 -13 ``` | [Files](https://app.codecov.io/gh/scalyr/scalyr-agent-2/pull/1263?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=scalyr) | Coverage Δ | | |---|---|---| | [scalyr\_agent/configuration.py](https://app.codecov.io/gh/scalyr/scalyr-agent-2/pull/1263?src=pr&el=tree&filepath=scalyr_agent%2Fconfiguration.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=scalyr#diff-c2NhbHlyX2FnZW50L2NvbmZpZ3VyYXRpb24ucHk=) | `77.51% <100.00%> (-0.03%)` | :arrow_down: | | [scalyr\_agent/monitor\_utils/k8s.py](https://app.codecov.io/gh/scalyr/scalyr-agent-2/pull/1263?src=pr&el=tree&filepath=scalyr_agent%2Fmonitor_utils%2Fk8s.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=scalyr#diff-c2NhbHlyX2FnZW50L21vbml0b3JfdXRpbHMvazhzLnB5) | `67.20% <ø> (-0.91%)` | :arrow_down: | | [...calyr\_agent/builtin\_monitors/kubernetes\_monitor.py](https://app.codecov.io/gh/scalyr/scalyr-agent-2/pull/1263?src=pr&el=tree&filepath=scalyr_agent%2Fbuiltin_monitors%2Fkubernetes_monitor.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=scalyr#diff-c2NhbHlyX2FnZW50L2J1aWx0aW5fbW9uaXRvcnMva3ViZXJuZXRlc19tb25pdG9yLnB5) | `56.90% <0.00%> (-0.42%)` | :arrow_down: | ... and [5 files with indirect coverage changes](https://app.codecov.io/gh/scalyr/scalyr-agent-2/pull/1263/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=scalyr)

[jmakar-s1]

Thank you. Can you take a look at the k8s integration tests as well?

i.e. https://github.com/scalyr/scalyr-agent-2/actions/runs/9175077249/job/25228143728

Fixed now & ready for another look