WikiWatershed / model-my-watershed

The web application front end for Model My Watershed.
https://modelmywatershed.org
Apache License 2.0
57 stars 31 forks source link

MapShed GMS File Creation #1212

Closed rajadain closed 8 years ago

rajadain commented 8 years ago

The code that creates the final GMS file is in ProcessData sub in Class1.vb of the MapShed source code:

'************************************************************************
' TITLE: ProcessData
' AUTHOR: David Lehning and Ken Corradini
' DESCRIPTION: Gets the selected shapes for the basins layer and clips
'              data layers, then generates run files for GWLF-E
'
' INPUTS: AverageHygAwcKf, CalcAEU, CalcCN, CalcGrowSeason, CalcGWN, CalcKVCoef, CalcLSCP, CalcRecessCoef, CalcSepticSystems, 
' CalcSoilP, ClearBasins, CompareSeptTractNo, GetExtractionData, GetPointSourceData, MakeDirectories, ProcessSoilData, ProcessWxYears
' ReadPointSourceTable, ReInitVariables, ResetMenuSettings, StreamLength, TransMonthData, UnpavedRdArea
' OUTPUTS: GWLF-E gms file
' NOTES: None
'
' Change Log: 
' Date          Changed By      Notes
' ************************************************************************
Public Sub ProcessData()
    ' Dimension generic shapefile and raster instances
    ' Dimension generic Field variables
    ' Dimension Basins variables
    ' Dimension Point Source variables
    ' Dimensions the preprocessing continuation message
    ' Dimension Time variables
    ' Dimension other non-specifically used variables
    ' Check to make sure the Basins layer currently active and selected layer
    ' Get the number of layers in the map
    ' Gets the Point Source ID - Layer is assumed to be the first layer in the view's TOC
    ' Get the selected polygon (watershed) to use as the clipping polygon
            ' Get the name of the selected layer
            ' Check to make sure that the selected layer is the basins layer
                ' Get the Basins layer
                ' Add Fields if necessary
                ' Check to see if fields are already present
                ' If fields aren't found, then add them.
                ' Get the basins file name
                ' Find the Id field in the basins layer
        ' Dimension all layer specific variables needed for data processing
        ' - Soils
        ' - Streams
        ' - Weather
        ' - Physiographic Regions
        ' - County
        ' - Septic Systems
        ' - Urban Areas
        ' - Unpaved Roads
        ' - Flowline
        ' - Water Extraction
        ' - AFOs
        ' - Land Use
        ' - DEM
        ' - Soil Phosphorus
        ' - Groundwater Nitrogen
        ' Dimension Selected layer items
        ' Dimension PreProcessing data form
        ' Reset the layer flags to 0 - Layer flags indicate if a layer is present in the view (layer flag = 1)
        ' Get the number of layers in the map
        ' Loop through the layers set each layers shapefile/raster object ("*Sf"), flag ("*Flag"), and position ("*Idx") 
            ' Set point layers
                ' Current the current layer
                ' Get the name of the selected layer
                ' Locate the weather layer
                ' Locate the Point Sources layer
                ' Locate the Water Extraction layer
                ' Locate the afos layer
            ' Set polygon layers
                ' Current the current layer
                ' Get the name of the selected layer
                ' Locate the soils layer
                ' Locate the Physiographic Provinces layer
                ' Locate the Coiunties layer
                ' Locate the Septic Systetms layer
                ' Locate the Urban Areas layer. Earlier versions of MapShed required the Urban Areas layer to be a 
            ' Set polyline layers
                ' Current the current layer
                ' Get the name of the selected layer
                ' Locate the streams layer
                ' Locate the unpaved layer
                ' Locate the flowline layer
            ' Set raster layers
                ' Current the current layer
                ' Get the name and path of the selected layer
                ' Locate the Landuse layer
                ' Locate the DEM layer
                ' Locate the SoilP layer
                ' Locate the GWN layer
                ' Locate the Urban Areas layer
        ' Make sure that the mandatory layers were found
        ' If the flowline layer is detected, obtain the ID field index to be used later when setting the attenuation 
            ' Set the attenuation flow distance to 0 since flowline layer not present.
        ' Create runfile and clipping directories and set ClipPath
        ' Get the number of selected basins
        ' Create the weather files
        ' Get the Id of the first selected basin
        ' If user selected to dissolve all basins into one, then set the run ID for each new analysis starting at 0 and 
        ' Define and open the preprocessing data form
        ' User canceled the preprocessing dialog, reset aggregate ID if necessary and stop data processing
        ' Determine if user wants to continue with the input data processing
        ' Get the initial start time of processing
        ' Refresh the map's view
        ' If aggregate basins desired then dissolve boundaries into one shape
            ' Loop through number of basins then merge newest basin into a "master" shape
            ' Determine the dissolved basins area
        ' For each Basin, perform the input data analysis
            ' If running on multiple basins, get basin area, stream flow adjustment factor, and attenuation flow distance
                ' Get shape index
                ' Return the Id for the selected shape
                ' Return the selected polygon
                ' Get the extent of the polygon
                ' Get the basin area
                ' Get the basins Streamflow Volume Adjustment Factor
                    ' If Streamflow Volume Adjustment Factor is 0, then set to 1.  A value of 0 means that it was not 
                ' Calculate the Attenuation Flow Distance
                    ' Get the Attenuation Flow Distance for each flowline
            ' Set the status bar
            ' Reinitialize all arrays and variables
            ' Get the latitude and longitude
            ' Locate the two closest Wx stations and get avg Latitude
            ' Obtain the monthly average length of day
            ' Dimension each clipped layer's full path
            ' Set the shapefile clip paths
            ' Set the raster clip paths
            ' Clip all mandatory layers
            ' - Clip Soil layer
            ' - Clip Stream layer
            ' - Clip Land Use layer
            ' - Clip DEM layer
            ' Clip non-mandatory layers if they exist in the view's TOC (*flag = 1)
            ' - Clip Soil Phosphorus layer
            ' - Clip Groundwater Nitrogen layer
            ' - Clip Physiographic Province layer
            ' - Clip County layer
            ' - Clip Septic Systems layer
            ' - Clip Point Source layer
            ' - Clip Unpaved Roads layer
            ' - Clip Water Extraction layer
            ' Obtain the Available Water Capacity raster data set from the clipped soils shape file
            ' Obtain the KFactor raster data set from the clipped soils shape file
            ' Derive slope from DEM
            ' If Flow Accumulation is selected as the LS Method, derive D8 (deterministic eight-node) grids and LS using 
            ' Adds and calculates the numeric hydrologic group field to the soils shape file.
            ' Calculate area weighted AvgAwc, AvgHyg, AvgKf
            ' Grid the streams shapefile
            ' Obtain the numerical value of hydrologic soil group from the clipped soils shape file
            ' Determines stream length, Awc, Hyg, Kf, Slope, C, P, & CN for each landuse; ag area of slope > 3% and > 3 
            ' Urban Areas layer analysis
                ' Calculate Total Developed Land
                ' Clip the Urban Areas raster
                ' 
                        ' Get the no data and header information
                        ' Get the unique values for the clipped urban areas grid in order to determine municipalities
                        ' Sort the UAsClipValues list
                        ' Begin line writing to the urban areas data list. This list contains all lines which are later 
                        ' Loop through each municipality present and determine its total area; land use areas; area 
                            ' Obtain the municipalities name from the look up table
                            ' Define the temporary urban areas municipality grid and land use grid paths
                            ' Get the header information from the clipped grid and create the new municipality grid
                            ' Reclassify all cells not equal to current municipality id to NoData.  This is needed to 
                            ' Generate the Land Use data
                            ' Write the individual Urban Area header line
                            ' Write the individual areas
                            ' Write the remaining Data lines
                ' If no Urban Area grid is present, write the following line to the GWLF-E input .gms file.
            ' Write GWLF-E run files
            ' Clean out clipped shape files and grids
            ' Kill previous clips
                ' Could not delete files
                ' Could not delete files
        ' Reset the status bar and cursor
        ' Get the current date and time
        ' Display the "run complete" message
        ' Resets status bar
        ' Clears the weather ID list
            ' Clean out clipped shape files and grids
            ' Kill previous clips
                ' Could not delete files
                ' Could not delete files
    ' Update View's preview map

not including function calls.

The specific WriteGwlfData subroutine which is called by ProcessData has the outline:

'************************************************************************
' TITLE: WriteGwlfData
' AUTHOR: David Lehning and Ken Corradini
' DESCRIPTION: Assembles data needed to write the GWLF-E run files 
'
' INPUTS: StrmClipPath, SoilClipPath
' OUTPUTS: New .gms file
' NOTES: Calls StreamLength, CalcSoilP, CalcGWN, CalcRecessCoef, TransMonthData, CalcSepticSystems
'
' Change Log: 
' Date          Changed By      Notes
' ************************************************************************
Private Sub WriteGwlfData(ByVal StrmClipPath As String, ByVal SoilClipPath As String)
    ' Determine landuses on which manure spreading is likely
    ' Get the total stream length in the watershed
    ' Get the default Soil P (later will add the check for Soil P Grid clip) => Non-mandatory layer
    ' Get the Soil Phosphorus from the clipped grid
    ' Soil Test P the estimate of available soil P
    ' Soil Total P is the estimate of concentration of total P
    ' Calculate soil test P (STP) based on estimate of Psed
    ' Determine dissolved P in runoff based on soil test P (STP)
    ' Determine dissolved P in runoff for turfgrass/golf courses
    ' Get the default Groundwater N and P (Later will add check for GW N P Grid clip) => Non-mandatory layer
    ' Get the Groundwater Nitrogen from the clipped grid
    ' If groundwater nitrogen is less than 0.34, then set to 0.34
    ' Set the AEU to 0 (Later will add check for Animal Density clip) => Non-mandatory layer
    ' Get the AFOs data and readjust AEU if AFOs layer present
    ' Animal feeding operations data
    ' Obtain the average animal weights in Kg for each animal
    ' Reset AEU variables
    ' Set flags
    ' Open the AFOs shape file if AFOs layer present
            ' Get the fields index
            ' Loop through the AFOs, obtain the points within the selected basin, and get the AFO values
                ' Get the shape
                ' Get the point
                ' Check to see if point is within the basin polygon
                ' Get AFOs data if within basin polygon
                    ' If AWMS (Poultry) then calculate its AEU
                        ' Calculate AEU
                    ' If AWMS (Livestock) then calculate its AEU
                        ' Calculate AEU
                    '  If Runoff control then calculate its AEU
                        ' Obtain AEU for each farm animal
                        ' Calculate AEU
                    ' If Phytase in Feed then calculate its AEU
                        ' Obtain AEU for each farm animal
                        ' Calculate AEU
                    ' Obtain AEU for each farm animal
        ' Obtain AEU for each farm animal
        ' Get the total AEU, Total livestock and poultry AEU
        ' If the total livestock AEU > 0 then recalculate the AEU based on the basin area
        ' Reset the percentages of the animal waste management systems
        ' Estimate the number of broilers and layers from the chickens layer
        ' Set the Pathogen variables
        ' Calculate the Septic Failure rate
        ' Set the Pathogen variables
        ' Calculate the Septic Failure rate
    ' Determine the N and P value for manure spreading and runoff
    ' Calculate the sediment delivery ratio based on area
    ' Assemble the landuse lines for transport file
    ' Redefine values as arrays aligned with new landuse values
    ' Construct the rural land use lines
    ' Calculate the total basin area in hectares based on the area for each land use category
    ' Reset average CN for rural land uses
    ' Loop through each rural land use category and calculate the average ET for the growing (ETGrow) and dormant (ETDorm) seasons
        ' Calculate the average CN for rural land uses
        ' Calculate the average K factor for rural land uses
        ' Calculate the average slope for rural land uses
        ' Assemble the rural land use lines
        ' Total the average ET for the growing (ETGrow) and dormant (ETDorm) seasons
    ' Default urban landuse lines
    ' - Low Density Mixed
    ' - Medium Density Mixed
    ' - High Density Mixed
    ' - Low Density Residential
    ' - Medium Density Residential
    ' - High Density Residential
    ' Calculate Total Developed Land
    ' If urban areas are 0 then zero out the lines
            ' Calculate the average K factor for urban land uses
            ' Calculate the average slope for urban land uses
            ' Calculate the average CN for urban land uses
        ' Total the average ET for the growing (ETGrow) and dormant (ETDorm) seasons
    ' Calculate average CN for the basin
    ' Calculate sediment A factor for use in stream bank erosion calculations in GWLF
    ' Calcuate the percent urban area
    ' Calculate sediment A factor with the DEM calculated slope
    ' Set the default Tile Drainage density to zero
    ' Assemble monthly values for the transport file
    ' Begin writing the GWLF-E gms file
    ' Create and write the first line. Contains the number of rural and urban land uses and the basin ID
    ' Create and write the coefficient line
    ' Recalculate urban length to all non-ag stream length
    ' Write the default Antecedent Rain+Melt Moisture Condition For Days -1 To -5
    ' Write the monthly data
    ' Write the landuse lines to the transport file
    ' Write the layer flag data
    ' Set the default Point Source flag and TotN and TotP to zero (Later will process Point Source layer clip) => Non-mandatory layer
    ' Set the default Septic System flag and TotN and TotP to zero (Later will process Septic System layer clip) => Non-mandatory layer
    ' The EstSS flag is true when user selects to use the land cover map to estimate population on septic systems
        ' Calculate septic system values
        ' Estimate septic system values
    ' Assemble and write nutrient loading information
    ' Assemble and write manure spreading information
    ' Wrute the landuse lines to the nutrient file
    ' Write contaminants
    ' Write N, P, and Sed code lines (used to identify the following N, P, and Sed values)
    ' Write N, P and Sed values to file
    ' Write manure spreading values
    ' Write the Point Source monthly N, P and Flow values
    ' Write the Septic Flag
    ' Write the monthly septic data
    ' Write NitrSepticLoad, PhosSepticLoad, NitrPlantUptake, PhosPlantUptake
    ' Write Tile Drainage concentrations
    ' Assemble the data for scenario reduction editor
    ' BMP1 default coefficients for N, P, and Sed
    ' BMP2 default coefficients for N, P, and Sed
    ' BMP3 default coefficients for N, P, and Sed
    ' BMP4 default coefficients for N, P, and Sed
    ' BMP5 default coefficients for N, P, and Sed
    ' BMP6 default coefficients for N, and P 
    ' BMP7 default coefficients for N, P, and Sed
    ' BMP8 default coefficients for N, P, and Sed
    ' Vegetated Buffer Strips for N, P, Sed, and Pathogen
    ' Streambank Fencing for N, P, Sed, and Pathogen
    ' Streambank Stabilization for N, P, and Sed
    ' Unaved Roads for N, P, and Sed
    ' AWMS (Livestock) for N, P, and Pathogen
    ' AWMS (Poultry) for N, P, and Pathogen
    ' Runoff Controls for N, P, and Pathogen
    ' Phytase in Feed for P
    ' Constructed Wetlands for N, P, Sed, and Pathogen
    ' Bioretention Areas for N, P, Sed, and Pathogen
    ' Detention Basins for N, P, Sed, and Pathogen
    ' Assemble the Scenario File lines with the above variables
    ' Conversion of Septic System to Secondary Treatment Plant - (not used in the model - PRedICT only)
    ' Conversion of Septic System to Tertiary Treatment Plant - (not used in the model - PRedICT only)
    ' Conversion of Primary Treatment to Secondary Treatment - (not used in the model - PRedICT only)
    ' Conversion of Primary Treatment to Tertiary Treatment - (not used in the model - PRedICT only)
    '  Conversion of Secondary Treatment to Tertiary Treatment - (not used in the model - PRedICT only)
    '  FC concentration for treatment plands - (not used in the model - PRedICT only)
    ' Default cost information for BMPs - (not used in the model - PRedICT only)
    ' Optimization variables (Optimization routines not used at this time) - (not used in the model - PRedICT only)
    ' Assemble the Scenario File lines with the above variables
    ' Prep area variables for the Scenario File
    ' Set all defaults for the scenario editor.  They are all hard coded values except where noted.
    ' The "Line" designations for each line refer to the "Scenario File" portion of the gms file and are referenced in that block of code
    ' Line 3
    ' Line 4
    ' Line 5
    ' Line 6
    ' End of line 7
    ' Line 8
    ' Line 9
    ' Line 10
    ' Line 11
    ' The total acres of row crops on 3% slope
    ' AWMS (Livestock)
    ' AWMS (Poultry)
    ' Runoff Control
    ' Phytast in Feed
    ' AEUs
    ' Streams in Ag Areas
    ' Total Stream Length
    ' Unpaved Road Length
    ' Stream Miles with Vegetated Buffer Strips
    ' Grazing and non-grazing animal values set to 0
    ' Line 12
    ' Line 13
    ' Line 14
    ' Line 36
    ' Line 37
    ' Line 38
    ' Line 39
    ' Line 40
    ' Line 41
    ' Line 42
    ' Line 43
    ' Line 44
    ' Line 45
    ' Line 46
    ' Initial scenario name
    ' Initial scenario project name
    ' Write the Scenario File
    ' Write the animal and pathogen lines
    ' Write the nutrient retention data
    ' Write the weather data
    ' Write the urban area data
    ' Close and save the newly created gms file

Translate this and any called VB subroutines to Python pseudocode.

rajadain commented 8 years ago

I have created a GMS Key Spreadsheet which details each field in the final GMS file. It has the following columns:

This spreadsheet was made using the original documentation and the original source code in the FileShare.