flyteorg / flyte

Scalable and flexible workflow orchestration platform that seamlessly unifies data, ML and analytics stacks.
https://flyte.org
Apache License 2.0
5.81k stars 660 forks source link

Inject offloading literal env vars #6027

Closed eapolinario closed 6 days ago

eapolinario commented 1 week ago

Tracking issue

Related to https://github.com/flyteorg/flyte/pull/5103

Why are the changes needed?

Flytekit uses the environment variables _F_L_MIN_SIZE_MB and _F_L_MAX_SIZE_MB to define the minimum and maximum sizes (in MB) for offloading literals. This PR injects those variables in case offloading is enabled.

What changes were proposed in this pull request?

Inject 2 new variables to task pods in case offloading of literals is enabled.

How was this patch tested?

Setup process

Screenshots

Check all the applicable boxes

Related PRs

Docs link

codecov[bot] commented 1 week ago

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 37.06%. Comparing base (25cfe16) to head (9b1f108). Report is 29 commits behind head on master.

Additional details and impacted files ```diff @@ Coverage Diff @@ ## master #6027 +/- ## ========================================== + Coverage 33.09% 37.06% +3.97% ========================================== Files 1013 1316 +303 Lines 107489 132115 +24626 ========================================== + Hits 35570 48967 +13397 - Misses 68767 78895 +10128 - Partials 3152 4253 +1101 ``` | [Flag](https://app.codecov.io/gh/flyteorg/flyte/pull/6027/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=flyteorg) | Coverage Δ | | |---|---|---| | [unittests-datacatalog](https://app.codecov.io/gh/flyteorg/flyte/pull/6027/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=flyteorg) | `51.58% <ø> (ø)` | | | [unittests-flyteadmin](https://app.codecov.io/gh/flyteorg/flyte/pull/6027/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=flyteorg) | `54.10% <ø> (?)` | | | [unittests-flytecopilot](https://app.codecov.io/gh/flyteorg/flyte/pull/6027/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=flyteorg) | `22.23% <ø> (ø)` | | | [unittests-flytectl](https://app.codecov.io/gh/flyteorg/flyte/pull/6027/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=flyteorg) | `62.46% <ø> (+0.02%)` | :arrow_up: | | [unittests-flyteidl](https://app.codecov.io/gh/flyteorg/flyte/pull/6027/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=flyteorg) | `7.25% <ø> (+0.32%)` | :arrow_up: | | [unittests-flyteplugins](https://app.codecov.io/gh/flyteorg/flyte/pull/6027/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=flyteorg) | `53.72% <100.00%> (-0.12%)` | :arrow_down: | | [unittests-flytepropeller](https://app.codecov.io/gh/flyteorg/flyte/pull/6027/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=flyteorg) | `42.63% <ø> (-0.28%)` | :arrow_down: | | [unittests-flytestdlib](https://app.codecov.io/gh/flyteorg/flyte/pull/6027/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=flyteorg) | `57.59% <ø> (+2.27%)` | :arrow_up: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=flyteorg#carryforward-flags-in-the-pull-request-comment) to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


🚨 Try these New Features: