project-codeflare / codeflare

Simplifying the definition and execution, scaling and deployment of pipelines on the cloud.
https://codeflare.dev
Apache License 2.0
222 stars 36 forks source link

Refactor configuration utility tool; added support for latest Ray version #40

Closed JainTwinkle closed 2 years ago

JainTwinkle commented 2 years ago

Related PRs Extending #37

What does this PR do?

This PR extends the Ray resiliency config tool. The PR does the following:

  1. The Ray config utility script now supports configurations from Ray v1.6, 1.7, and 1.8.

  2. The tool now saves config. files into their respective version directory. This is more organized as compared to saving files from all Ray versions into a single folder. For example, now the tools save output config. files in the following manner by default. ├── configs │ ├── 1.0.0 │ │ ├── Ray 1.0.0 related config files │ ├── 1.1.0 │ │ └── Ray 1.1.0 related config files

  3. The configuration parsing code is more generalized than before. Removed some hard-coded conditions and added functions to make the code less cluttered.

  4. Added a new field called config_string in the output config file. This field stores the original string from which we parsed the default value of the configuration. The config_string stores string whenever the default value is not a simple value but a conditional statement. This field will help in explaining how the associated environment variable's value will determine the default value. For example: For raylet_start_wait_time_s configuration, the signature/input is following:

    RAY_CONFIG(uint32_t, raylet_start_wait_time_s,
           std::getenv("RAY_preallocate_plasma_memory") != nullptr &&
                   std::getenv("RAY_preallocate_plasma_memory") == std::string("1")
               ? 120
               : 10)

    And, the script dumps following Yaml entry in the .conf file:

    raylet_start_wait_time_s:
    config_string: 'std::getenv("RAY_preallocate_plasma_memory") != nullptr && std::getenv("RAY_preallocate_plasma_memory") == std::string("1") ? 120 : 10'
    default: '10'
    env: RAY_preallocate_plasma_memory
    type: uint32_t
    value_for_this_mode: '10'

    The new field i.e. config_string is informatory and gives an idea about how the associated environment variable will be processed.

  5. The config tool now uses YAML format variable instead of a hardcoded string for system-config map YAML (system_cm.yaml)

JainTwinkle commented 2 years ago

@chcost @raghukiran1224 Could we assign someone for this PR? Thanks! PS: I have fixed my notification settings this time. I'll respond to comments/suggestions on time.