search

EIAgov / BlueSky

Apache License 2.0
39 stars 4 forks source link

readme

BlueSky Prototype

Table of Contents

 

Overview

Project BlueSky is an EIA initiative to develop an open source, next generation energy systems model, which will eventually be used to produce the Annual Energy Outlook (AEO) and International Energy Outlook (IEO). Our outlooks are focused on projecting realistic outcomes under a given set of assumptions, accounting for rapid technological innovation, new policy, changing consumer preferences, shifting trade patterns, and the real-world friction associated with the adoption of novel or risky technology. To address these challenges, the next generation model is designed to be modular, flexible, transparent, and robust.

The BlueSky Prototype is the first step towards creating this next generation model. Our objective in releasing the prototype is to give the modeling community an early opportunity to experiment with the new framework and provide feedback. Feedback gathered from stakeholders will be used to develop a full-scale version of the model beginning in 2025. See the contribution file for information on how to give feedback.

There are four key features associated with the BlueSky Prototype:

  1. A computationally efficient, modular structure that allows each sector to flexibly capture the underlying market behavior using different governing dynamics. The protype contains three test modules representing electricity, hydrogen, and residential demand. Both the electricity and hydrogen models employ least-cost linear optimization as a governing dynamic, while the residential sector model adjusts demand levels based on prices from the other two sectors. The prototype includes an 'integrator' module that can either execute an individual module or combine the modules into a single nonlinear optimization problem using two different methods. This feature will allow the next generation model to be modular and flexible.

  2. Well-documented Python code. In the prototype, we have tried to write Python code that is easy to follow. We have included both detailed, high-level descriptions of each module in markdown, as well as code documentation using Sphinx, which we use to aggregate doc strings embedded in the code to create documentation. Most open-source models develop coding and documentation styles prior to their initial release. Even for models that are easy to apply, the underlying code and documentation are not always clear. We are inviting the community to provide feedback at this early stage so that code and documentation developed for the production version are as transparent as possible. This feature will allow the next generation model to be transparent.

  3. An example data pipeline documented with Snakemake. Energy system models are data intensive and require a multitude of data transformations to convert raw data from a known source into formatted model input data. We are experimenting with Snakemake, a tool that allows us to create a well-documented data management framework to organize, pre-process, and post-process model data. In the prototype, Snakemake is used in a standalone application to create a geospatial crosswalk involving both electricity operations and weather data. We are considering the use of Snakemake across the model to comprehensively manage input data. This feature will allow the next generation model to be transparent.

  4. Two methods to characterize model sensitivity. One method uses an efficient way to approximate derivatives of model outputs with respect to inputs for systems of nonlinear equations using complex variables (Lai et al., 2005). The method is broadly applicable to any module where the equations are continuously differentiable and has been implemented in applications beyond energy modeling. A second method is directly applicable to convex optimization problems and applies approximation methods for sensitivity analysis on optimization models (Castillo et al., 2006). These methods will allow us to ensure robust model results by testing model sensitivity. Additional methods to quantify model sensitivity and uncertainty will be implemented in the future.

Prototype diagram

The BlueSky Prototype is focused on demonstrating the four features above. The feedback gathered from community experimentation will be used to improve future versions of the next generation model. Given the prototype's limited scope, it does NOT provide results that are relevant to the real world. To emphasize this point, we have omitted any reference to a specific country, region, or timeframe in the input data.

 

Setup

Follow these steps to set up and run the BlueSky Prototype model.

Prerequisites

To set up the appropriate Python environment, you will need to install the following:

Steps

  1. Clone the Bluesky repository
  2. Set up the environment

Clone the BlueSky repository

Clone the BlueSky GitHub repository to your local computer. See the tabs below for instructions using Git or VS-Code.

Clone with git

## Prerequisites **Git**: Ensure that you have Git Bash installed on your machine. You can download it from [git-scm.com](https://git-scm.com/). ## Steps to Clone a Repository 1. **Open Your Terminal or Command Prompt**: - On **Windows**, you can use Git Bash or Command Prompt. - On **macOS** or **Linux**, open the Terminal. git bash image 2. **Navigate to Your Desired Directory**: Use the `cd` command to change to the directory where you want to clone the repository. For example: ```bash cd path/to/your/directory ``` 3. **Get the Repository URL**: - Go to the GitHub repository page you want to clone. - Click on the **Code** button, then copy the repository URL (either HTTPS or SSH). 4. **Clone the Repository**: Use the `git clone` command followed by the repository URL. For example: ```bash git clone https://github.com/username/repository.git ``` or if using SSH: ```bash git clone git@github.com:username/repository.git ``` 5. **Access the Cloned Repository**: After cloning, navigate into the repository folder: ```bash cd repository ``` ## Additional Notes - If you're using HTTPS, you may be prompted to enter your GitHub username and password.

Clone with Visual Studio Code

## Prerequisites **Visual Studio Code**: Ensure that you have VS-Code installed on your machine. ## Steps to Clone a Repository using Visual Studio Code 1. Open VS-Code and **click** `Clone Git Repository` *circled below in $`\textcolor{red}{\text{RED}}`$ vs code image 1 2. Paste the repository link into the search bar when prompted and hit **ENTER** *circled below in $`\textcolor{red}{\text{RED}}`$ ```bash https://github.com/EIAgov/BlueSky.git ``` vs code image 2 3. The *File Explorer* should pop up and ask where you would like to save the cloned repository. 4. **Click** on the folder destination and hit save. The repository will be cloned to your specified location.

Setup Environment with Anaconda Prompt

  1. Open Anaconda Prompt

    Launch the Anaconda Prompt to begin the setup process.

  2. Navigate to the Working Directory

    Use the cd command to navigate to the working directory for the BlueSky Prototype. For example:

    cd path\to\your\BlueSkyPrototype

  3. Run the Environment Creation Script

    Run the batch file to set up the Conda environment:

    envs\env-setup

    This will create and set up the necessary Conda environment for the project.

Setup Instructions with Screenshots

Click to expand setup instructions ## Steps 1. **Open Anaconda Prompt** Launch the Anaconda Prompt to begin the setup process. Image for anaconda prompt 2. **Navigate to the Working Directory** Use the `cd` command to navigate to the working directory for the BlueSky Prototype. For example: ```bash cd path\to\your\BlueSkyPrototype ``` Image to navigate to directory 3. **Run the Environment Creation Script** Run the batch file to set up the Conda environment: ```bash envs\env-setup.bat ``` This will create and set up the necessary Conda environment for the project. Image to show environment set up process

 

Usage

The BlueSky Prototype can be executed either as a Dash app that provides a graphical user interface (GUI), or run from the command line. All scripts should be run from the top-level directory of the project via the app or main.py file, as explained below.

Note that the Prototype can be run with different configuration options to indicate the modules, regionality, temporal resolution, solver options, and other module-specific settings. These options are encoded in run_config.toml. When running the model from the command line, run_config.toml must be modified directly and contains detailed descriptions of the configuration settings. When running the model from the GUI, the same configuration options can be set via the interface without accessing the file directly. For GUI users, the configuration options are described in detail here.

One of the key features of the BlueSky Prototype is the ability to run modules individually or select between two solve options for the integrated runs. The table below describes the different modes when running the model. More details on these methods can be found in the Integrator README.

Mode Methods File Demonstrates
standalone runner.py Apply a unified (single optimization) method to one of the modules (Electricity, Hydrogen, Residential)
gs-combo gaussseidel.py Apply an iterative (Gauss-Seidel) method to 3-way combination of modules (Electricity, Hydrogen, Residential)
unified-combo unified.py Apply a unified (single optimization) method to 3-way combination of modules (Electricity, Hydrogen, Residential)

Ways to Run the Model

Dash App Graphical User Interface (GUI) Steps

  1. Open Anaconda Prompt

    Launch the Anaconda Prompt to begin the setup process.

  2. Navigate to the Working Directory

    Use the cd command to navigate to the working directory for the BlueSky Prototype. For example:

    cd path\to\your\BlueSky

  3. Run the Dash App

    To launch the BlueSky Prototype app, use the following command:

    .\run_dash_app

    This will start the Dash application.

  4. Access the Web Interface

    • Once the app is running, a web browser should automatically open.
    • If the GUI doesn’t show up immediately, refresh the browser page.

You're all set! The BlueSky Prototype model should now be running, and you can interact with it through the browser interface.

Dash app main screen

Usage Instructions with Screenshots

Click to expand usage instructions ## Steps 1. **Open Anaconda Prompt** Launch the Anaconda Prompt to begin the setup process. Image for anaconda prompt 2. **Navigate to the Working Directory** Use the `cd` command to navigate to the working directory for the BlueSky Prototype. For example: ```bash cd path\to\your\BlueSkyPrototype ``` Image to navigate to directory 3. **Run the Dash App** To launch the BlueSky Prototype app, use the following command: ```bash .\run_dash_app ``` This will start the Dash application. cmd Image with run dash bat 4. **Access the Web Interface** - Once the app is running, a web browser should automatically open. - If the GUI doesn’t show up immediately, refresh the browser page. Image of dash app look and feel You're all set! The BlueSky Prototype model should now be running, and you can interact with it through the browser interface.

Dash App features

features of dash app config settings and save button

Dash Features description

  1. The Edit Configuration Settings section allows the user the ability to overide default settings, which are found in run_config_template.toml. Make sure to select Save to save any changes to the default settings.

  2. Save button allows users to save model configuration inputs for select model runs. This action overwrites the run_config.toml file, which is used as the run file for model runs.

  3. Configuration message This message notifies the user that they have successfully saved and updated the run_config.toml file used for model runs.

  4. Mode Selection and Run This is where the user selects which model mode they would like to try for a given run. After making the selection, they must click the Run button. A blue loading feature will let the user know that model is running. A blue bar will appear when the model is finished.

  5. Output message This red message notifies the user that the model successfully ran the mode selected and created a time stamp log and model outputs folder in the outputs folder in the main directory.

  6. Code Documentation Button When selected, this button opens the html code documentation generated by Sphinx in a new browser tab.

To return to default settings, refresh the web browsers and click the save button to return to the template default settings.

Command Prompt Steps

See the tab below for instructions on how to run the model using the command prompt.

Be sure to have the 'bsky' environment activated in a terminal, navigate to the 'BlueSky' directory in your terminal, and then run the following code: ```bash python main.py --help ``` This should return a description of ```main.py```, as well as the options available within the command line interface. Now, check to see if we can run and solve a model by executing: ```bash python main.py --mode standalone ``` If this reports an objective value, you have successfully run the model in standalone! The file [```main.py```](main.py) processes user options as directed by the configuration file [```run_config.toml```](src/integrator/run_config.toml) and builds/solves an instance of an integrated (multiple modules) or standalone (single module) model. Any runs initiated from the top level ```main.py``` file will initiate logging and report to output folder. The log file contains (currently very limited) updates on run progress, solver output, etc. The default logging level is INFO. That may be increased by setting the command line argument --debug when running main.py. It is also possible to set the option for ```default_mode``` in ```run_config.toml```. Then, run ```main.py``` without specifying a mode: ```bash python main.py ```

 

Documentation

Documentation for each of the modules is available using the links below. We also include documentation for other products within the repository, including a sample data workflow tool, a sample unit test setup, and a sample output data viewer.

Users can access code documentation following these instructions for HTML, Markdown, and PDF documentation. Alternatively, code documentation can be generated by the user within the GUI by clicking the 'code documentation' button, which will run Sphinx locally and generate an html file in your open browser. See the instructions on working with the GUI for more details

Support

See the contribution file for information on feedback. Please reach out to BlueSky team for assistance: nextgenerationmodel@eia.gov

Authors

U.S. Energy Information Administration

License

The BlueSky code, as distributed here, is governed by the Apache 2.0 license

Project status

Currently under development and review.