search

simple-framework / simple_grid_yaml_compiler

Generates the extended YAML output for an input site_level_configuration_file
Apache License 2.0
0 stars 9 forks source link

processing optional schema parameters #56

Open maany opened 4 years ago

maany commented 4 years ago

Describe the bug A clear and concise description of what the bug is. What input files were supplied to the compiler? What output/error did you get? Error:

KeyError: 'supplemental_config'

Input

### url of schema
schema:  "https://raw.githubusercontent.com/WLCG-Lightweight-Sites/simple_grid_site_repo/master/simple_base_schema.yaml"

### Variable declaration:
global_variables:
  - &lightweight_component01_ip_address 188.185.112.251
  - &lightweight_component01_fqdn simple-condor-ce.cern.ch
  - &lightweight_component02_ip_address 188.185.84.16
  - &lightweight_component02_fqdn simple-condor-batch.cern.ch
  - &lightweight_component03_ip_address 188.185.70.127
  - &lightweight_component03_fqdn simple-condor-worker.cern.ch
  - &lightweight_component04_ip_address 188.185.85.178
  - &lightweight_component04_fqdn simple-condor-submit.cern.ch
htcondor_ce_runtime_variables:
  - &htcondor_ce_runtime_var_ce_host simple-condor-ce.cern.ch

htcondor_batch__runtime_variables:
  - &htcondor_runtime_var_batch_host simple-condor-batch.cern.ch

site:
  name: 'simple-dev-site'
  email: 'mayank.sharma@cern.ch'
  latitude: 46.3
  longitude: 6.2
  location: CERN, Geneva, Switzerland
  description: 'A demo site demonstrating the SIMPLE framework for site setup'
  website: 'https://home.cern'
  support_website: 'https://groups.google.com/forum/#!forum/wlcg-lightweight-sites'
  support_email: 'mayank.sharma@cern.ch'
  security_email: 'mayank.sharma@cern.ch'
  grid: 'wlcg' #site_other_grid: str(), wlcg, egi,osg
  tier: 3
  bdii_host: bdii.cern.ch
  cvmfs_http_proxy_list:
    - "http://cvmfs.cat.cbpf.br:3128"
    - "http://cvmfs-sec.cat.cbpf.br:3128"
  use_argus: no

preferred_tech_stack:
  level_1_configuration: puppet
  level_2_configuration: sh
  container_orchestration: docker-swarm
  container: docker

site_infrastructure:
  - fqdn: *lightweight_component01_fqdn
    ip_address: *lightweight_component01_ip_address
  - fqdn: *lightweight_component02_fqdn
    ip_address: *lightweight_component02_ip_address
  - fqdn: *lightweight_component03_fqdn
    ip_address: *lightweight_component03_ip_address
  - fqdn: *lightweight_component04_fqdn
    ip_address: *lightweight_component04_ip_address

lightweight_components:
  - type: compute_element
    name: HTCondor-CE
    repository_url: "https://github.com/WLCG-Lightweight-Sites/simple_htcondor_ce"
    repository_revision: "master"
    execution_id: 0
    lifecycle_hooks:
      pre_config: []
      pre_init: []
      post_init: []
    deploy:
      - node: *lightweight_component01_fqdn
        container_count: 1
    config:
      condor_host_execution_id: 1
    preferred_tech_stack:
      level_2_configuration: ""

  - name: HTCondor-Batch
    type: batch_system
    repository_url: "https://github.com/WLCG-Lightweight-Sites/simple_htcondor_batch"
    repository_revision: "master"
    execution_id: 1
    deploy:
      - node: *lightweight_component02_fqdn
        container_count: 1
    config:
    lifecycle_hooks:
      pre_config: []
      pre_init: []
      post_init: []
    preferred_tech_stack:
      level_2_configuration: ""

  - type: worker_node
    name: HTCondor-Execute
    repository_url: "https://github.com/WLCG-Lightweight-Sites/simple_htcondor_worker"
    repository_revision: "master"
    execution_id: 2
    deploy:
      - node: *lightweight_component03_fqdn
        container_count: 1
      - node: *lightweight_component04_fqdn
        container_count: 1
    config:
      condor_host_execution_id: 1
    lifecycle_hooks:
      pre_config: []
      pre_init: []
      post_init: []
    preferred_tech_stack:
      level_2_configuration: ""

supported_virtual_organizations:
  - *default_vo_alice
  - *default_vo_dteam
  - *default_vo_ops

voms_config:
  - voms_fqan: '/alice'
    pool_accounts:
      - *default_pool_accounts_alice
    vo: *default_vo_alice
  - voms_fqan: '/alice/ROLE=lcgadmin'
    pool_accounts:
      - *default_pool_accounts_alice
      - *default_pool_accounts_alicesgm
    vo: *default_vo_alice
    comment: sgm
  - voms_fqan: '/dteam'
    pool_accounts:
      - *default_pool_accounts_dteam
    vo: *default_vo_dteam
  - voms_fqan: '/dteam/ROLE=lcgadmin'
    pool_accounts:
      - *default_pool_accounts_dteamsgm
      - *default_pool_accounts_dteam
    vo: *default_vo_dteam
    comment: sgm
  - voms_fqan: '/ops'
    vo: *default_vo_ops
    pool_accounts:
      - *default_pool_accounts_ops
  - voms_fqan: '/ops/ROLE=lcgadmin'
    pool_accounts:
      - *default_pool_accounts_opssgm
      - *default_pool_accounts_ops
    vo: *default_vo_ops
    comment: sgm

Output

/Users/mayanksharma/cernbox/Projects/WLCG/simple_grid/simple_grid_yaml_compiler/.virtualenv/bin/python /Users/mayanksharma/cernbox/Projects/WLCG/simple_grid/simple_grid_yaml_compiler/simple_grid_yaml_compiler/yaml_compiler.py ./tests/resources/condor-config.yaml -o ./tests/output.yaml -s ./tests/schema.yaml
[{'url': 'https://github.com/WLCG-Lightweight-Sites/simple_htcondor_ce', 'revision': 'master'}, {'url': 'https://github.com/WLCG-Lightweight-Sites/simple_htcondor_batch', 'revision': 'master'}, {'url': 'https://github.com/WLCG-Lightweight-Sites/simple_htcondor_worker', 'revision': 'master'}]
RUNTIME$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$
Param 1: schema
     - required: True
     - use_default: True
     - specified: True
     - default_value: http://github.com/WLCG-lightweight-Sites/simple_grid_site_repo/simple-base-schema.yaml
     - user_defined_value: https://raw.githubusercontent.com/WLCG-Lightweight-Sites/simple_grid_site_repo/master/simple_base_schema.yaml
     - final_value: https://raw.githubusercontent.com/WLCG-Lightweight-Sites/simple_grid_site_repo/master/simple_base_schema.yaml
Param 2: global_variables
     - required: False
     - use_default: False
     - specified: True
     - user_defined_value: ['188.185.112.251', 'simple-condor-ce.cern.ch', '188.185.84.16', 'simple-condor-batch.cern.ch', '188.185.70.127', 'simple-condor-worker.cern.ch', '188.185.85.178', 'simple-condor-submit.cern.ch']
     - final_value: ['188.185.112.251', 'simple-condor-ce.cern.ch', '188.185.84.16', 'simple-condor-batch.cern.ch', '188.185.70.127', 'simple-condor-worker.cern.ch', '188.185.85.178', 'simple-condor-submit.cern.ch']
Param 3: site_infrastructure
     Param 1: fqdn
          - required: True
          - use_default: False
          - specified: True
          - user_defined_value: simple-condor-ce.cern.ch
          - final_value: simple-condor-ce.cern.ch
     Param 2: ip_address
          - required: True
          - use_default: False
          - specified: True
          - user_defined_value: 188.185.112.251
          - final_value: 188.185.112.251
     -----------
     Param 1: fqdn
          - required: True
          - use_default: False
          - specified: True
          - user_defined_value: simple-condor-batch.cern.ch
          - final_value: simple-condor-batch.cern.ch
     Param 2: ip_address
          - required: True
          - use_default: False
          - specified: True
          - user_defined_value: 188.185.84.16
          - final_value: 188.185.84.16
     -----------
     Param 1: fqdn
          - required: True
          - use_default: False
          - specified: True
          - user_defined_value: simple-condor-worker.cern.ch
          - final_value: simple-condor-worker.cern.ch
     Param 2: ip_address
          - required: True
          - use_default: False
          - specified: True
          - user_defined_value: 188.185.70.127
          - final_value: 188.185.70.127
     -----------
     Param 1: fqdn
          - required: True
          - use_default: False
          - specified: True
          - user_defined_value: simple-condor-submit.cern.ch
          - final_value: simple-condor-submit.cern.ch
     Param 2: ip_address
          - required: True
          - use_default: False
          - specified: True
          - user_defined_value: 188.185.85.178
          - final_value: 188.185.85.178
Param 4: lightweight_components
     Param 1: name
          - required: True
          - use_default: False
          - specified: True
          - user_defined_value: HTCondor-CE
          - final_value: HTCondor-CE
     Param 2: type
          - required: True
          - use_default: False
          - specified: True
          - user_defined_value: compute_element
          - final_value: compute_element
     Param 3: repository_url
          - required: True
          - use_default: False
          - specified: True
          - user_defined_value: https://github.com/WLCG-Lightweight-Sites/simple_htcondor_ce
          - final_value: https://github.com/WLCG-Lightweight-Sites/simple_htcondor_ce
     Param 4: repository_revision
          - required: True
          - use_default: False
          - specified: True
          - user_defined_value: master
          - final_value: master
     Param 5: execution_id
          - required: True
          - use_default: False
          - specified: True
          - user_defined_value: 0
          - final_value: 0
     Param 6: lifecycle_hooks
          Param 1: pre_config
               - required: True
               - use_default: False
               - specified: True
               - user_defined_value: []
               - final_value: []
          Param 2: pre_init
               - required: True
               - use_default: False
               - specified: True
               - user_defined_value: []
               - final_value: []
          Param 3: post_init
               - required: True
               - use_default: False
               - specified: True
               - user_defined_value: []
               - final_value: []
     Param 7: deploy
          Param 1: node
               - required: True
               - use_default: False
               - specified: True
               - user_defined_value: simple-condor-ce.cern.ch
               - final_value: simple-condor-ce.cern.ch
          Param 2: container_count
               - required: True
               - use_default: False
               - specified: True
               - user_defined_value: 1
               - final_value: 1
     Param 8: preferred_tech_stack
          Param 1: level_2_configuration
               - required: False
               - use_default: False
               - specified: True
               - user_defined_value: 
               - final_value: 
     Param 9: config
          - required: True
          - use_default: False
          - specified: True
          - user_defined_value: ordereddict([('condor_host_execution_id', 1)])
          - final_value: ordereddict([('condor_host_execution_id', 1)])
     Param 10: supplemental_config
Traceback (most recent call last):
  File "/Users/mayanksharma/cernbox/Projects/WLCG/simple_grid/simple_grid_yaml_compiler/simple_grid_yaml_compiler/yaml_compiler.py", line 185, in <module>
    main()
  File "/Users/mayanksharma/cernbox/Projects/WLCG/simple_grid/simple_grid_yaml_compiler/simple_grid_yaml_compiler/yaml_compiler.py", line 181, in main
    execute_compiler(site_level_configuration_file, output, schema)
  File "/Users/mayanksharma/cernbox/Projects/WLCG/simple_grid/simple_grid_yaml_compiler/simple_grid_yaml_compiler/yaml_compiler.py", line 163, in execute_compiler
    phase_5_output_file = phase_5(phase_4_output, runtime_vars, yaml)
  File "/Users/mayanksharma/cernbox/Projects/WLCG/simple_grid/simple_grid_yaml_compiler/simple_grid_yaml_compiler/yaml_compiler.py", line 63, in phase_5
    meta_info_file = base_repo_file("meta_info")
  File "/Users/mayanksharma/cernbox/Projects/WLCG/simple_grid/simple_grid_yaml_compiler/simple_grid_yaml_compiler/compiler/processor_config_schemas.py", line 199, in process_complete_schema
    return get_final_config(config, expected_params, defaults, meta_info, expandable_types)
  File "/Users/mayanksharma/cernbox/Projects/WLCG/simple_grid/simple_grid_yaml_compiler/simple_grid_yaml_compiler/compiler/processor_config_schemas.py", line 132, in get_final_config
    indent = indent + 1)
  File "/Users/mayanksharma/cernbox/Projects/WLCG/simple_grid/simple_grid_yaml_compiler/simple_grid_yaml_compiler/compiler/processor_config_schemas.py", line 106, in get_final_config
    config = config[param],
  File "/Users/mayanksharma/cernbox/Projects/WLCG/simple_grid/simple_grid_yaml_compiler/.virtualenv/lib/python2.7/site-packages/ruamel/yaml/comments.py", line 757, in __getitem__
    return ordereddict.__getitem__(self, key)
KeyError: 'supplemental_config'

Process finished with exit code 1

https://github.com/WLCG-Lightweight-Sites/simple_grid_yaml_compiler/blob/master/simple_grid_yaml_compiler/compiler/processor_config_schemas.py#L105 If param is optional in this line (supplemental_config, lifecycle_hooks, preferred_tech_stack), then KeyError will be thrown.

To Reproduce Run against master branch of simple_grid_site_repo

BASE_REPO_URL = "https://github.com/WLCG-Lightweight-Sites/simple_grid_site_repo"
BASE_REPO_REVISION = "master"

Expected behavior

Desktop (please complete the following information):

Additional context Add any other context about the problem here.

maany commented 4 years ago

More Info:

        if param_type in expandable_types:
            config[param] = get_final_config(
                    config = config[param],
                    expected_params = expandable_types[param_type],
                    defaults = defaults,
                    meta_info = meta_info,
                    expandable_types = expandable_types,
                    indent = indent + 1)

            continue

        else:
            match = re.match("list\((.*)\)", param_type)

            if match and match.group(1) in expandable_types:
                final_values = []
                base_type    = match.group(1)

                for index, value in enumerate(config[param]):
                    if index:
                        print "     " * (indent + 1) + "-----------"

                    final_value = get_final_config(
                        config = value,
                        expected_params = expandable_types[base_type],
                        defaults = defaults,
                        meta_info = meta_info,
                        expandable_types = expandable_types,
                        indent = indent + 1)

                    final_values.append(final_value)

                config[param] = final_values
                continue
        is_required  = expected_params[param]["required"]
        use_default  = expected_params[param]["use_default"]
        is_specified = True if param in config else False

supp_config is expandable in the base_schema, but if it is not specified in the site_level_config, the value of config is None in the last line of this script. This should be changed to 

is_specified = False is config or param not in config else True