add support for distributed Offline Eval

[alexnikulkov]

Summary: Adding support for distributed Offline Eval. This requires maintaining local buffers in each trainer instance and syncing them across all trainers periodically. The sync happens under one of 2 conditions:

1. When the "critical" weight of data has been consumed (will be set approximately equal to the size of 1-hr partition)
2. At the end of the training epoch (if data has been consumed since last sync)

Also, updating the FREE pipeline to remove the restriction on number of nodes for Offline Eval runs

Differential Revision: D42407669

[facebook-github-bot]

This pull request was exported from Phabricator. Differential Revision: D42407669

[facebook-github-bot]

This pull request was exported from Phabricator. Differential Revision: D42407669

[facebook-github-bot]

This pull request was exported from Phabricator. Differential Revision: D42407669

[codecov-commenter]

Codecov Report

Base: 87.72% // Head: 87.73% // Increases project coverage by +0.01% :tada:

Coverage data is based on head (5aac164) compared to base (517a67f). Patch coverage: 92.45% of modified lines in pull request are covered.

Additional details and impacted files

```diff @@ Coverage Diff @@ ## main #708 +/- ## ========================================== + Coverage 87.72% 87.73% +0.01% ========================================== Files 373 373 Lines 24042 24078 +36 Branches 44 44 ========================================== + Hits 21091 21125 +34 - Misses 2925 2927 +2 Partials 26 26 ``` | [Impacted Files](https://codecov.io/gh/facebookresearch/ReAgent/pull/708?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=facebookresearch) | Coverage Δ | | |---|---|---| | [reagent/training/cb/base\_trainer.py](https://codecov.io/gh/facebookresearch/ReAgent/pull/708?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=facebookresearch#diff-cmVhZ2VudC90cmFpbmluZy9jYi9iYXNlX3RyYWluZXIucHk=) | `83.33% <60.00%> (-1.12%)` | :arrow_down: | | [reagent/evaluation/cb/base\_evaluator.py](https://codecov.io/gh/facebookresearch/ReAgent/pull/708?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=facebookresearch#diff-cmVhZ2VudC9ldmFsdWF0aW9uL2NiL2Jhc2VfZXZhbHVhdG9yLnB5) | `92.98% <93.75%> (-0.50%)` | :arrow_down: | | [reagent/evaluation/cb/policy\_evaluator.py](https://codecov.io/gh/facebookresearch/ReAgent/pull/708?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=facebookresearch#diff-cmVhZ2VudC9ldmFsdWF0aW9uL2NiL3BvbGljeV9ldmFsdWF0b3IucHk=) | `97.22% <95.00%> (+1.38%)` | :arrow_up: | | [reagent/test/evaluation/cb/test\_integration.py](https://codecov.io/gh/facebookresearch/ReAgent/pull/708?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=facebookresearch#diff-cmVhZ2VudC90ZXN0L2V2YWx1YXRpb24vY2IvdGVzdF9pbnRlZ3JhdGlvbi5weQ==) | `100.00% <100.00%> (ø)` | | | [...eagent/test/evaluation/cb/test\_policy\_evaluator.py](https://codecov.io/gh/facebookresearch/ReAgent/pull/708?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=facebookresearch#diff-cmVhZ2VudC90ZXN0L2V2YWx1YXRpb24vY2IvdGVzdF9wb2xpY3lfZXZhbHVhdG9yLnB5) | `97.61% <100.00%> (+0.32%)` | :arrow_up: | | [reagent/gym/tests/test\_gym.py](https://codecov.io/gh/facebookresearch/ReAgent/pull/708?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=facebookresearch#diff-cmVhZ2VudC9neW0vdGVzdHMvdGVzdF9neW0ucHk=) | `95.93% <0.00%> (-0.82%)` | :arrow_down: | | [reagent/core/utils.py](https://codecov.io/gh/facebookresearch/ReAgent/pull/708?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=facebookresearch#diff-cmVhZ2VudC9jb3JlL3V0aWxzLnB5) | `87.23% <0.00%> (+2.12%)` | :arrow_up: | Help us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=facebookresearch). Have a feature suggestion? [Share it here.](https://app.codecov.io/gh/feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=facebookresearch)

:umbrella: View full report at Codecov.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.

[facebook-github-bot]

This pull request has been merged in facebookresearch/ReAgent@89519d714c673f63dc797fc2eaabc60fa63d2dd7.