Clean up main.py -- create separate config for filenames and folders?

rjhanes commented 3 years ago

Ported from https://github.nrel.gov/aeberle/celavi/issues/160

This functionality is being implemented in the configs branch.

Latest commit is 7290b0a , which parameterizes the CostGraph initialization and all filename/directory references with information from the yaml file. Testing is still underway, I will update once I've fully tested the current commit and yaml file.

Current yaml file reads as:

flags:
  compute_locations: False  # if compute_locations is enabled (True), compute locations from raw input files (e.g., LMOP, US Wind Turbine Database)
  run_routes: False  # if run_routes is enabled (True), compute routing distances between all input locations
  use_computed_routes: True  # if use_computed_routes is enabled, read in a pre-assembled routes file instead of generating a new one
  initialize_costgraph: True  # create cost graph fresh or use an imported version
  pickle_costgraph: True  # save the newly initialized costgraph as a pickle file

data_filtering:
  enable_data_filtering: False  # If true, dataset will be filtered to the states below

  # If you specify True above, you need to list the states to filter here.
  # Default behavior is not to pass any states through the filter.
  # If enable_data_filtering is False, this list is ignored.

  states_to_filter:
    - TX
    - IA
    - CO

scenario_parameters:
    start_year: 2000
    end_year: 2050
    timesteps_per_year: 12
    max_dist: 300

data_directories:
    inputs: inputs/
    raw_locations: inputs/raw_location_data/
    us_roads: inputs/precomputed_us_road_network/
    preprocessing_output: preprocessing/
    lookup_tables: lookup_tables/
    lci: pylca_celavi_data/
    outputs: outputs
    routing_output: preprocessing/routing_intermediate_files

input_filenames:
    locs: locations_computed.csv
    step_costs: step_costs.csv
    fac_edges: fac_edges.csv
    transpo_edges: transpo_edges.csv
    route_pair: route_pairs.csv
    avg_blade_masses: avgblademass.csv
    routes_custom: routes.csv
    routes_computed: routes_computed.csv
    transportation_graph: transportation_graph.csv
    node_locs: node_locations.csv
    power_plant_locs: uswtdb_v4_1_20210721.csv
    landfill_locs: landfilllmopdata.csv
    other_facility_locs: other_facility_locations_all_us.csv
    lookup_facility_type: facility_type.csv
    turbine_data: number_of_turbines.csv

output_filenames:
    costgraph_pickle: netw.obj
    costgraph_csv: netw.csv

costgraph_parameters:
    sc_begin: manufacturing
    sc_end: 
      - landfilling
      - cement co-processing
      - next use
    cg_verbose: 1
    save_cg_csv: True
    finegrind_cumul_initial: 1.0
    finegrind_initial_cost: 165.38
    finegrind_revenue: 242.56
    finegrind_learnrate: -0.05
    finegrind_material_loss: 0.3
    coarsegrind_cumul_initial: 1.0
    coarsegrind_initial_cost: 121.28
    coarsegrind_learnrate: -0.05
    cg_update_timesteps: 12

rjhanes commented 3 years ago

Updated config file reads:

flags:
  compute_locations: False  # if compute_locations is enabled (True), compute locations from raw input files (e.g., LMOP, US Wind Turbine Database)
  run_routes: True  # if run_routes is enabled (True), compute routing distances between all input locations
  use_computed_routes: True  # if use_computed_routes is enabled, read in a pre-assembled routes file instead of generating a new one
  initialize_costgraph: True  # create cost graph fresh or use an imported version
  pickle_costgraph: True  # save the newly initialized costgraph as a pickle file

data_filtering:
  enable_data_filtering: False  # If true, dataset will be filtered to the states below

  # If you specify True above, you need to list the states to filter here.
  # Default behavior is not to pass any states through the filter.
  # If enable_data_filtering is False, this list is ignored.

  states_to_filter:
    - TX
    - IA
    - CO

scenario_parameters:
    start_year: 2000.0
    end_year: 2050.0
    timesteps_per_year: 12
    max_dist: 300

data_directories:
    inputs: inputs/
    raw_locations: inputs/raw_location_data/
    us_roads: inputs/precomputed_us_road_network/
    preprocessing_output: preprocessing/
    lookup_tables: lookup_tables/
    lci: pylca_celavi_data/
    outputs: outputs
    routing_output: preprocessing/routing_intermediate_files

input_filenames:
    locs: locations_computed.csv
    step_costs: step_costs.csv
    fac_edges: fac_edges.csv
    transpo_edges: transpo_edges.csv
    route_pairs: route_pairs.csv
    avg_blade_masses: avgblademass.csv
    routes_custom: routes.csv
    routes_computed: routes_computed.csv
    transportation_graph: transportation_graph.csv
    node_locs: node_locations.csv
    power_plant_locs: uswtdb_v4_1_20210721.csv
    landfill_locs: landfilllmopdata.csv
    other_facility_locs: other_facility_locations_all_us.csv
    lookup_facility_type: facility_type.csv
    turbine_data: number_of_turbines.csv

output_filenames:
    costgraph_pickle: netw.obj
    costgraph_csv: netw.csv

costgraph_parameters:
    sc_begin: manufacturing
    sc_end: 
      - landfilling
      - cement co-processing
      - next use
    cg_verbose: 1
    save_cg_csv: True
    finegrind_cumul_initial: 1.0
    finegrind_initial_cost: 165.38
    finegrind_revenue: 242.56
    finegrind_learnrate: -0.05
    finegrind_material_loss: 0.3
    coarsegrind_cumul_initial: 1.0
    coarsegrind_initial_cost: 121.28
    coarsegrind_learnrate: -0.05
    cg_update_timesteps: 12

rjhanes commented 3 years ago

Tested ab1e8cc on medium-data branch in the interests of time. I found some errors in the config parameter formatting that I corrected in c533774.

Testing summary:

locations_computed generated, is consistent with other files
number_of_turbines generated, contains capacity projections, is consistent with other files
step_costs generated, is consistent with other files
routes_computed generated, is consistent (this one is in the repo since it's small enough)
Did not test filtering functionality
Used fixed component lifetimes (did not test Weibull distribution)
Context generated correctly
Components generated correctly
I kept the model running until ~2005 and saw no unexpected or obviously erroneous output.

Config file updated for commit c533774 reads: (Must be saved as config.yaml in the topmost celavi-data directory for the code to find it)

flags:
  compute_locations: True  # if compute_locations is enabled (True), compute locations from raw input files (e.g., LMOP, US Wind Turbine Database)
  run_routes: False  # if run_routes is enabled (True), compute routing distances between all input locations
  use_computed_routes: True  # if use_computed_routes is enabled, read in a pre-assembled routes file instead of generating a new one
  initialize_costgraph: True  # create cost graph fresh or use an imported version
  enable_data_filtering: False  # If true, dataset will be filtered to the states below
  pickle_costgraph: True  # save the newly initialized costgraph as a pickle file
  generate_step_costs: True # set to False if supply chain costs for a facility type vary regionally
  use_fixed_lifetime: True # set to False to use Weibull distribution for lifetimes

scenario_parameters:
    start_year: 2000.0
    end_year: 2050.0
    timesteps_per_year: 12
    max_dist: 300 #km
    # If you specify enable_data_filtering = True above, you need to list the states to filter here.
    # Default behavior is not to pass any states through the filter.
    # If enable_data_filtering is False, this list is ignored.
    states_to_filter:
    - IA

data_directories:
    inputs: inputs/
    raw_locations: inputs/raw_location_data/
    us_roads: inputs/precomputed_us_road_network/
    preprocessing_output: preprocessing/
    lookup_tables: lookup_tables/
    lci: pylca_celavi_data/
    outputs: outputs/
    routing_output: preprocessing/routing_intermediate_files/

input_filenames:
    locs: locations_computed.csv
    step_costs: step_costs.csv
    fac_edges: fac_edges.csv
    transpo_edges: transpo_edges.csv
    route_pairs: route_pairs.csv
    avg_blade_masses: avgblademass.csv
    routes_custom: routes.csv
    routes_computed: routes_computed.csv
    transportation_graph: transportation_graph.csv
    node_locs: node_locations.csv
    power_plant_locs: uswtdb_v4_1_20210721.csv
    landfill_locs: landfilllmopdata.csv
    other_facility_locs: other_facility_locations_all_us.csv
    standard_scenario: StScen20A_MidCase_annual_state.csv
    lookup_facility_type: facility_type.csv
    lookup_step_costs: step_costs_default.csv
    turbine_data: number_of_turbines.csv

output_filenames:
    costgraph_pickle: netw.obj
    costgraph_csv: netw.csv

costgraph_parameters:
    sc_begin: manufacturing
    sc_end: 
    - landfilling
    - cement co-processing
    - next use
    cg_verbose: 1
    save_cg_csv: True
    finegrind_cumul_initial: 1.0
    finegrind_initial_cost: 165.38
    finegrind_revenue: 242.56
    finegrind_learnrate: -0.05
    finegrind_material_loss: 0.3
    coarsegrind_cumul_initial: 1.0
    coarsegrind_initial_cost: 121.28
    coarsegrind_learnrate: -0.05
    cg_update_timesteps: 12

discrete_event_parameters:
    component_list:
    - nacelle
    - blade
    - tower
    - foundation
    seed: 13
    min_lifespan: 120 # Units: timesteps
    blade_weibull_L: 240
    blade_weibull_K: 2.2
    component_fixed_lifetimes:
      nacelle : 30
      blade : 20
      foundation : 50
      tower : 50

rjhanes commented 3 years ago

closed with merge of PR #13

NREL / celavi

Clean up main.py -- create separate config for filenames and folders? #2