search

simple-framework / simple_grid_yaml_compiler

Generates the extended YAML output for an input site_level_configuration_file
Apache License 2.0
0 stars 9 forks source link

Allow for 0 params in config-schema files of component repos #57

Open maany opened 4 years ago

maany commented 4 years ago

Describe the bug A clear and concise description of what the bug is. The HTCondor-Batch component repository at present uses 0 parameters in the config-schema. Therefore, the following line in the compiler: 

print " No of expected params: " + str(len(expected_params))

fails.

In theory, we should allow for component repositories that need absolutely zero additional configs from the site admins.

What input files were supplied to the compiler? What output/error did you get?

To Reproduce

### url of schema
schema:  "https://raw.githubusercontent.com/WLCG-Lightweight-Sites/simple_grid_site_repo/master/simple_base_schema.yaml"

### Variable declaration:
global_variables:
  - &lightweight_component01_ip_address 188.185.112.251
  - &lightweight_component01_fqdn simple-condor-ce.cern.ch
  - &lightweight_component02_ip_address 188.185.84.16
  - &lightweight_component02_fqdn simple-condor-batch.cern.ch
  - &lightweight_component03_ip_address 188.185.70.127
  - &lightweight_component03_fqdn simple-condor-worker.cern.ch
  - &lightweight_component04_ip_address 188.185.85.178
  - &lightweight_component04_fqdn simple-condor-submit.cern.ch
htcondor_ce_runtime_variables:
  - &htcondor_ce_runtime_var_ce_host simple-condor-ce.cern.ch

htcondor_batch__runtime_variables:
  - &htcondor_runtime_var_batch_host simple-condor-batch.cern.ch

site:
  name: 'simple-dev-site'
  email: 'mayank.sharma@cern.ch'
  latitude: 46.3
  longitude: 6.2
  location: CERN, Geneva, Switzerland
  description: 'A demo site demonstrating the SIMPLE framework for site setup'
  website: 'https://home.cern'
  support_website: 'https://groups.google.com/forum/#!forum/wlcg-lightweight-sites'
  support_email: 'mayank.sharma@cern.ch'
  security_email: 'mayank.sharma@cern.ch'
  grid: 'wlcg' #site_other_grid: str(), wlcg, egi,osg
  tier: 3
  bdii_host: bdii.cern.ch
  cvmfs_http_proxy_list:
    - "http://cvmfs.cat.cbpf.br:3128"
    - "http://cvmfs-sec.cat.cbpf.br:3128"
  use_argus: no

preferred_tech_stack:
  level_1_configuration: puppet
  level_2_configuration: sh
  container_orchestration: docker-swarm
  container: docker

site_infrastructure:
  - fqdn: *lightweight_component01_fqdn
    ip_address: *lightweight_component01_ip_address
  - fqdn: *lightweight_component02_fqdn
    ip_address: *lightweight_component02_ip_address
  - fqdn: *lightweight_component03_fqdn
    ip_address: *lightweight_component03_ip_address
  - fqdn: *lightweight_component04_fqdn
    ip_address: *lightweight_component04_ip_address

lightweight_components:
  - type: compute_element
    name: HTCondor-CE
    repository_url: "https://github.com/WLCG-Lightweight-Sites/simple_htcondor_ce"
    repository_revision: "master"
    execution_id: 0
    lifecycle_hooks:
      pre_config: []
      pre_init: []
      post_init: []
    deploy:
      - node: *lightweight_component01_fqdn
        container_count: 1
    config:
      condor_host_execution_id: 1
    preferred_tech_stack:
      level_2_configuration: ""
    supplemental_config:
      some_additional_parameter: ""

  - name: HTCondor-Batch
    type: batch_system
    repository_url: "https://github.com/WLCG-Lightweight-Sites/simple_htcondor_batch"
    repository_revision: "master"
    execution_id: 1
    deploy:
      - node: *lightweight_component02_fqdn
        container_count: 1
    config:
      condor_host: ""
    lifecycle_hooks:
      pre_config: []
      pre_init: []
      post_init: []
    preferred_tech_stack:
      level_2_configuration: ""
    supplemental_config:
      some_additional_parameter: ""

  - type: worker_node
    name: HTCondor-Execute
    repository_url: "https://github.com/WLCG-Lightweight-Sites/simple_htcondor_worker"
    repository_revision: "master"
    execution_id: 2
    deploy:
      - node: *lightweight_component03_fqdn
        container_count: 1
      - node: *lightweight_component04_fqdn
        container_count: 1
    config:
      condor_host_execution_id: 1
    lifecycle_hooks:
      pre_config: []
      pre_init: []
      post_init: []
    preferred_tech_stack:
      level_2_configuration: ""
    supplemental_config:
      some_additional_parameter: ""

supported_virtual_organizations:
  - *default_vo_alice
  - *default_vo_dteam
  - *default_vo_ops

voms_config:
  - voms_fqan: '/alice'
    pool_accounts:
      - *default_pool_accounts_alice
    vo: *default_vo_alice
  - voms_fqan: '/alice/ROLE=lcgadmin'
    pool_accounts:
      - *default_pool_accounts_alice
      - *default_pool_accounts_alicesgm
    vo: *default_vo_alice
    comment: sgm
  - voms_fqan: '/dteam'
    pool_accounts:
      - *default_pool_accounts_dteam
    vo: *default_vo_dteam
  - voms_fqan: '/dteam/ROLE=lcgadmin'
    pool_accounts:
      - *default_pool_accounts_dteamsgm
      - *default_pool_accounts_dteam
    vo: *default_vo_dteam
    comment: sgm
  - voms_fqan: '/ops'
    vo: *default_vo_ops
    pool_accounts:
      - *default_pool_accounts_ops
  - voms_fqan: '/ops/ROLE=lcgadmin'
    pool_accounts:
      - *default_pool_accounts_opssgm
      - *default_pool_accounts_ops
    vo: *default_vo_ops
    comment: sgm

Logs

/Users/mayanksharma/cernbox/Projects/WLCG/simple_grid/simple_grid_yaml_compiler/.virtualenv/bin/python /Users/mayanksharma/cernbox/Projects/WLCG/simple_grid/simple_grid_yaml_compiler/simple_grid_yaml_compiler/yaml_compiler.py ./tests/resources/condor-config.yaml -o ./tests/output.yaml -s ./tests/schema.yaml
[{'url': 'https://github.com/WLCG-Lightweight-Sites/simple_htcondor_ce', 'revision': 'master'}, {'url': 'https://github.com/WLCG-Lightweight-Sites/simple_htcondor_batch', 'revision': 'master'}, {'url': 'https://github.com/WLCG-Lightweight-Sites/simple_htcondor_worker', 'revision': 'master'}]
RUNTIME$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$
Param 1: schema
     - required: True
     - use_default: True
     - specified: True
     - default_value: http://github.com/WLCG-lightweight-Sites/simple_grid_site_repo/simple-base-schema.yaml
     - user_defined_value: https://raw.githubusercontent.com/WLCG-Lightweight-Sites/simple_grid_site_repo/master/simple_base_schema.yaml
     - final_value: https://raw.githubusercontent.com/WLCG-Lightweight-Sites/simple_grid_site_repo/master/simple_base_schema.yaml
Param 2: global_variables
     - required: False
     - use_default: False
     - specified: True
     - user_defined_value: ['188.185.112.251', 'simple-condor-ce.cern.ch', '188.185.84.16', 'simple-condor-batch.cern.ch', '188.185.70.127', 'simple-condor-worker.cern.ch', '188.185.85.178', 'simple-condor-submit.cern.ch']
     - final_value: ['188.185.112.251', 'simple-condor-ce.cern.ch', '188.185.84.16', 'simple-condor-batch.cern.ch', '188.185.70.127', 'simple-condor-worker.cern.ch', '188.185.85.178', 'simple-condor-submit.cern.ch']
Param 3: site_infrastructure
     Param 1: fqdn
          - required: True
          - use_default: False
          - specified: True
          - user_defined_value: simple-condor-ce.cern.ch
          - final_value: simple-condor-ce.cern.ch
     Param 2: ip_address
          - required: True
          - use_default: False
          - specified: True
          - user_defined_value: 188.185.112.251
          - final_value: 188.185.112.251
     -----------
     Param 1: fqdn
          - required: True
          - use_default: False
          - specified: True
          - user_defined_value: simple-condor-batch.cern.ch
          - final_value: simple-condor-batch.cern.ch
     Param 2: ip_address
          - required: True
          - use_default: False
          - specified: True
          - user_defined_value: 188.185.84.16
          - final_value: 188.185.84.16
     -----------
     Param 1: fqdn
          - required: True
          - use_default: False
          - specified: True
          - user_defined_value: simple-condor-worker.cern.ch
          - final_value: simple-condor-worker.cern.ch
     Param 2: ip_address
          - required: True
          - use_default: False
          - specified: True
          - user_defined_value: 188.185.70.127
          - final_value: 188.185.70.127
     -----------
     Param 1: fqdn
          - required: True
          - use_default: False
          - specified: True
          - user_defined_value: simple-condor-submit.cern.ch
          - final_value: simple-condor-submit.cern.ch
     Param 2: ip_address
          - required: True
          - use_default: False
          - specified: True
          - user_defined_value: 188.185.85.178
          - final_value: 188.185.85.178
Param 4: lightweight_components
     Param 1: name
          - required: True
          - use_default: False
          - specified: True
          - user_defined_value: HTCondor-CE
          - final_value: HTCondor-CE
     Param 2: type
          - required: True
          - use_default: False
          - specified: True
          - user_defined_value: compute_element
          - final_value: compute_element
     Param 3: repository_url
          - required: True
          - use_default: False
          - specified: True
          - user_defined_value: https://github.com/WLCG-Lightweight-Sites/simple_htcondor_ce
          - final_value: https://github.com/WLCG-Lightweight-Sites/simple_htcondor_ce
     Param 4: repository_revision
          - required: True
          - use_default: False
          - specified: True
          - user_defined_value: master
          - final_value: master
     Param 5: execution_id
          - required: True
          - use_default: False
          - specified: True
          - user_defined_value: 0
          - final_value: 0
     Param 6: lifecycle_hooks
          Param 1: pre_config
               - required: True
               - use_default: False
               - specified: True
               - user_defined_value: []
               - final_value: []
          Param 2: pre_init
               - required: True
               - use_default: False
               - specified: True
               - user_defined_value: []
               - final_value: []
          Param 3: post_init
               - required: True
               - use_default: False
               - specified: True
               - user_defined_value: []
               - final_value: []
     Param 7: deploy
          Param 1: node
               - required: True
               - use_default: False
               - specified: True
               - user_defined_value: simple-condor-ce.cern.ch
               - final_value: simple-condor-ce.cern.ch
          Param 2: container_count
               - required: True
               - use_default: False
               - specified: True
               - user_defined_value: 1
               - final_value: 1
     Param 8: preferred_tech_stack
          Param 1: level_2_configuration
               - required: False
               - use_default: False
               - specified: True
               - user_defined_value: 
               - final_value: 
     Param 9: config
          - required: True
          - use_default: False
          - specified: True
          - user_defined_value: ordereddict([('condor_host_execution_id', 1)])
          - final_value: ordereddict([('condor_host_execution_id', 1)])
     Param 10: supplemental_config
          Param 1: some_additional_parameter
               - required: False
               - use_default: False
               - specified: True
               - user_defined_value: 
               - final_value: 
     -----------
     Param 1: name
          - required: True
          - use_default: False
          - specified: True
          - user_defined_value: HTCondor-Batch
          - final_value: HTCondor-Batch
     Param 2: type
          - required: True
          - use_default: False
          - specified: True
          - user_defined_value: batch_system
          - final_value: batch_system
     Param 3: repository_url
          - required: True
          - use_default: False
          - specified: True
          - user_defined_value: https://github.com/WLCG-Lightweight-Sites/simple_htcondor_batch
          - final_value: https://github.com/WLCG-Lightweight-Sites/simple_htcondor_batch
     Param 4: repository_revision
          - required: True
          - use_default: False
          - specified: True
          - user_defined_value: master
          - final_value: master
     Param 5: execution_id
          - required: True
          - use_default: False
          - specified: True
          - user_defined_value: 1
          - final_value: 1
     Param 6: lifecycle_hooks
          Param 1: pre_config
               - required: True
               - use_default: False
               - specified: True
               - user_defined_value: []
               - final_value: []
          Param 2: pre_init
               - required: True
               - use_default: False
               - specified: True
               - user_defined_value: []
               - final_value: []
          Param 3: post_init
               - required: True
               - use_default: False
               - specified: True
               - user_defined_value: []
               - final_value: []
     Param 7: deploy
          Param 1: node
               - required: True
               - use_default: False
               - specified: True
               - user_defined_value: simple-condor-batch.cern.ch
               - final_value: simple-condor-batch.cern.ch
          Param 2: container_count
               - required: True
               - use_default: False
               - specified: True
               - user_defined_value: 1
               - final_value: 1
     Param 8: preferred_tech_stack
          Param 1: level_2_configuration
               - required: False
               - use_default: False
               - specified: True
               - user_defined_value: 
               - final_value: 
     Param 9: config
          - required: True
          - use_default: False
          - specified: True
          - user_defined_value: ordereddict([('condor_host', '')])
          - final_value: ordereddict([('condor_host', '')])
     Param 10: supplemental_config
          Param 1: some_additional_parameter
               - required: False
               - use_default: False
               - specified: True
               - user_defined_value: 
               - final_value: 
     -----------
     Param 1: name
          - required: True
          - use_default: False
          - specified: True
          - user_defined_value: HTCondor-Execute
          - final_value: HTCondor-Execute
     Param 2: type
          - required: True
          - use_default: False
          - specified: True
          - user_defined_value: worker_node
          - final_value: worker_node
     Param 3: repository_url
          - required: True
          - use_default: False
          - specified: True
          - user_defined_value: https://github.com/WLCG-Lightweight-Sites/simple_htcondor_worker
          - final_value: https://github.com/WLCG-Lightweight-Sites/simple_htcondor_worker
     Param 4: repository_revision
          - required: True
          - use_default: False
          - specified: True
          - user_defined_value: master
          - final_value: master
     Param 5: execution_id
          - required: True
          - use_default: False
          - specified: True
          - user_defined_value: 2
          - final_value: 2
     Param 6: lifecycle_hooks
          Param 1: pre_config
               - required: True
               - use_default: False
               - specified: True
               - user_defined_value: []
               - final_value: []
          Param 2: pre_init
               - required: True
               - use_default: False
               - specified: True
               - user_defined_value: []
               - final_value: []
          Param 3: post_init
               - required: True
               - use_default: False
               - specified: True
               - user_defined_value: []
               - final_value: []
     Param 7: deploy
          Param 1: node
               - required: True
               - use_default: False
               - specified: True
               - user_defined_value: simple-condor-worker.cern.ch
               - final_value: simple-condor-worker.cern.ch
          Param 2: container_count
               - required: True
               - use_default: False
               - specified: True
               - user_defined_value: 1
               - final_value: 1
          -----------
          Param 1: node
               - required: True
               - use_default: False
               - specified: True
               - user_defined_value: simple-condor-submit.cern.ch
               - final_value: simple-condor-submit.cern.ch
          Param 2: container_count
               - required: True
               - use_default: False
               - specified: True
               - user_defined_value: 1
               - final_value: 1
     Param 8: preferred_tech_stack
          Param 1: level_2_configuration
               - required: False
               - use_default: False
               - specified: True
               - user_defined_value: 
               - final_value: 
     Param 9: config
          - required: True
          - use_default: False
          - specified: True
          - user_defined_value: ordereddict([('condor_host_execution_id', 1)])
          - final_value: ordereddict([('condor_host_execution_id', 1)])
     Param 10: supplemental_config
          Param 1: some_additional_parameter
               - required: False
               - use_default: False
               - specified: True
               - user_defined_value: 
               - final_value: 
Component: HTCondor-CE
 No of expected params: 4
  - Param 1 :condor_host_execution_id
     - required: True
     - specified: True
     - user-defined-value: 1
     - Final Value: 1
  - Param 2 :uid_domain
     - required: True
     - specified: False
     - use-default: True
     - default-value: cern.ch
     - Final Value: cern.ch
  - Param 3 :gss_assist_gridmap_cache_expiration
     - required: True
     - specified: False
     - use-default: True
     - default-value: 7200
     - Final Value: 7200
  - Param 4 :per_job_history_dir
     - required: True
     - specified: False
     - use-default: True
     - default-value: /var/lib/gratia/condorce_data
     - Final Value: /var/lib/gratia/condorce_data
{'gss_assist_gridmap_cache_expiration': 7200, 'condor_host_execution_id': 1, 'per_job_history_dir': '/var/lib/gratia/condorce_data', 'uid_domain': 'cern.ch'}
Component: HTCondor-Batch
Traceback (most recent call last):
  File "/Users/mayanksharma/cernbox/Projects/WLCG/simple_grid/simple_grid_yaml_compiler/simple_grid_yaml_compiler/yaml_compiler.py", line 185, in <module>
    main()
  File "/Users/mayanksharma/cernbox/Projects/WLCG/simple_grid/simple_grid_yaml_compiler/simple_grid_yaml_compiler/yaml_compiler.py", line 181, in main
    execute_compiler(site_level_configuration_file, output, schema)
  File "/Users/mayanksharma/cernbox/Projects/WLCG/simple_grid/simple_grid_yaml_compiler/simple_grid_yaml_compiler/yaml_compiler.py", line 163, in execute_compiler
    phase_5_output_file = phase_5(phase_4_output, runtime_vars, yaml)
  File "/Users/mayanksharma/cernbox/Projects/WLCG/simple_grid/simple_grid_yaml_compiler/simple_grid_yaml_compiler/yaml_compiler.py", line 94, in phase_5
    augmented_config = processor_config_schemas.process_config_schema(lightweight_component[component_section], config_schema_file, default_data_runtime_file, meta_info)
  File "/Users/mayanksharma/cernbox/Projects/WLCG/simple_grid/simple_grid_yaml_compiler/simple_grid_yaml_compiler/compiler/processor_config_schemas.py", line 94, in process_config_schema
    final_config_for_component = get_final_config_values_for_component(component, expected_params, defaults_for_expected_params, meta_info)
  File "/Users/mayanksharma/cernbox/Projects/WLCG/simple_grid/simple_grid_yaml_compiler/simple_grid_yaml_compiler/compiler/processor_config_schemas.py", line 35, in get_final_config_values_for_component
    print " No of expected params: " + str(len(expected_params))
TypeError: object of type 'NoneType' has no len()

Desktop (please complete the following information):

Additional context Add any other context about the problem here.