Export tool to update the soil_rechr_init and soil_moist_init values

[jdduh]

For any given HRU, the soil_rechr_max value cannot be larger than the soil_moist_max value. During the export, please 1) set soil_rechr_max to soil_moist_max if soil_rech_max is larger than soil_moist_maxc and 2) set soil_rechr_init and soil_moist_init to 25% of soil_rechr_max and soil_moist_max respectively.

Prompt a warning message if 1) occurs. The message could be something like "Warning! Parameter inconsistency detected. Some HRUs' soil_rechr_max values are larger than soil_moist_max values. The soil_rechr_max values are replaced by soil_moist_max values."

[lbross]

I suggest handling these items when the parameters are calculated from the Profile Builder rather than during the export. This gives better audit capability for the parameters and they aren't "secretly" calculated when the parameters are exported. It will also keep the export code cleaner.

We can create a soil_moist_max_orig column for the original values and populate soil_rechr_max as directed. The soil_moist_max_orig column won't be exported but will be available for reference. We can add soil_rechr_init and soil_moist_init columns to the parameter output table for the hru and calculate it as directed above. If these are added to the bagis_parameters.txt control file then they will automatically be exported with no code changes to the export process. Also, the user can view them in the parameter viewer, if desired.

[jdduh]

I agree and will modify the models in the toolbox.

[jdduh]

Combine all 4 parameters in one model and send warning message when internal inconsistency is detected.

[lbross]

Will the warning message be in the BAGIS-H profile builder of the BAGIS-P parameter export? Need a sample model that generates a warning message so we can update the gp model framework to check these messages and pass them through.

[jdduh]

An updated soil toolbox is posted on ftp://basins.geog.pdx.edu/BAGIS/BAGIS-P%20Models/methods/. The new model is Soil_Moist_Recharge_Parameters. It contains a python script tool that sends a WARNING message (using the arcpy.AddWarning method) at the end of the process. BAGIS-P needs to get the warning message for display. Please note that arcpy allows adding 3 types of messages: 1) AddMessage, 2) AddWarning, 3) AddError. I think the warning message is more appropriate for our purpose.

[jdduh]

Here is an example of the warning message of the new soil model.

[lbross]

I cannot successfully run this model using ArcToolbox from Masoud's computer. Here is the error I get: Messages Executing: soilMoistMax3 D:\Momeni\AOIs\teton_aoi D:\Momeni\AOIs\teton_aoi\zones\elev_15 elev_15 D:\NRCS\GIS\Static\BAGIS_DATABIN_09222015.gdb\LF_us_130evt D:\NRCS\GIS\Static\BAGIS_DATABIN_09222015.gdb\STATSGO_muid D:\Momeni\AOIs\teton_aoi\zones\elev_15\param.gdb\Profile_eWSF_PRMS_params soil_moist_max soil_rechr_max soil_moist_init soil_rechr_init Start Time: Tue Nov 03 15:10:29 2015 Executing (Lookup (3)): Lookup D:\NRCS\GIS\Static\BAGIS_DATABIN_09222015.gdb\STATSGO_muid AWCL C:\Users\lbross\Documents\ArcGIS\Default.gdb\wc_l Start Time: Tue Nov 03 15:10:40 2015 ERROR 999999: Error executing function. The table was not found. [VAT_wc_l] The operation was attempted on an empty geometry. ERROR 010067: Error in executing grid expression. Failed to execute (Lookup (3)). Failed at Tue Nov 03 15:10:43 2015 (Elapsed Time: 3.00 seconds) Failed to execute (soilMoistMax3). Failed at Tue Nov 03 15:10:44 2015 (Elapsed Time: 15.00 seconds)

Attached is an image of the arguments I supplied:

[lbross]

I also tried running through BAGIS-P and it failed but with a different error message: Messages Executing: soilMoistMax3 D:\Momeni\AOIs\teton_aoi D:\Momeni\AOIs\teton_aoi\zones\elev_15 elev_15 D:\Momeni\AOIs\teton_aoi\param\paramdata.gdb\evt_072509 D:\Momeni\AOIs\teton_aoi\param\paramdata.gdb\muid D:\Momeni\AOIs\teton_aoi\zones\elev_15\param.gdb\Profile_eWSF_PRMS_params soil_moist_max soil_rechr_max soil_moist_init soil_rechr_init Start Time: Tue Nov 03 15:20:38 2015 Executing (Lookup (3)): Lookup D:\Momeni\AOIs\teton_aoi\param\paramdata.gdb\muid AWCL D:\Momeni\AOIs\teton_aoi\Default.gdb\wc_l Start Time: Tue Nov 03 15:20:53 2015 Succeeded at Tue Nov 03 15:20:57 2015 (Elapsed Time: 4.00 seconds) Executing (Lookup (4)): Lookup D:\Momeni\AOIs\teton_aoi\param\paramdata.gdb\muid AWCH D:\Momeni\AOIs\teton_aoi\Default.gdb\WC_h Start Time: Tue Nov 03 15:20:57 2015 Succeeded at Tue Nov 03 15:21:01 2015 (Elapsed Time: 4.00 seconds) Executing (Raster Calculator: get avg AWC): RasterCalculator "("D:\Momeni\AOIs\teton_aoi\Default.gdb\wc_l" + "D:\Momeni\AOIs\teton_aoi\Default.gdb\WC_h") / 2" D:\Momeni\AOIs\teton_aoi\Default.gdb\AWC Start Time: Tue Nov 03 15:21:01 2015 (Raster(r"D:\Momeni\AOIs\teton_aoi\Default.gdb\wc_l") + Raster(r"D:\Momeni\AOIs\teton_aoi\Default.gdb\WC_h")) / 2 Succeeded at Tue Nov 03 15:21:06 2015 (Elapsed Time: 5.00 seconds) Executing (lookup (2)): Lookup D:\Momeni\AOIs\teton_aoi\param\paramdata.gdb\muid ROCKDEPH D:\Momeni\AOIs\teton_aoi\Default.gdb\rockdeph Start Time: Tue Nov 03 15:21:06 2015 Succeeded at Tue Nov 03 15:21:10 2015 (Elapsed Time: 4.00 seconds) Executing (Lookup): Lookup D:\Momeni\AOIs\teton_aoi\param\paramdata.gdb\muid ROCKDEPL D:\Momeni\AOIs\teton_aoi\Default.gdb\rockdepl Start Time: Tue Nov 03 15:21:10 2015 Succeeded at Tue Nov 03 15:21:14 2015 (Elapsed Time: 4.00 seconds) Executing (Raster Calculator: get bedrock depth): RasterCalculator "("D:\Momeni\AOIs\teton_aoi\Default.gdb\rockdeph" + "D:\Momeni\AOIs\teton_aoi\Default.gdb\rockdepl") / 2" D:\Momeni\AOIs\teton_aoi\Default.gdb\rockDepth Start Time: Tue Nov 03 15:21:14 2015 (Raster(r"D:\Momeni\AOIs\teton_aoi\Default.gdb\rockdeph") + Raster(r"D:\Momeni\AOIs\teton_aoi\Default.gdb\rockdepl")) / 2 Succeeded at Tue Nov 03 15:21:18 2015 (Elapsed Time: 4.00 seconds) Executing (Lookup (5)): Lookup D:\Momeni\AOIs\teton_aoi\param\paramdata.gdb\evt_072509 root_depth D:\Momeni\AOIs\teton_aoi\Default.gdb\Reclass_land1 Start Time: Tue Nov 03 15:21:18 2015 Succeeded at Tue Nov 03 15:21:22 2015 (Elapsed Time: 4.00 seconds) Executing (Cell Statistics: min. of plant root depth and soil depth): CellStatistics D:\Momeni\AOIs\teton_aoi\Default.gdb\rockDepth;D:\Momeni\AOIs\teton_aoi\Default.gdb\Reclass_land1 D:\Momeni\AOIs\teton_aoi\Default.gdb\CellSta_rock1 MINIMUM DATA Start Time: Tue Nov 03 15:21:22 2015 Succeeded at Tue Nov 03 15:21:27 2015 (Elapsed Time: 5.00 seconds) Executing (Cell Statistics: min. of rooting depth and 18): CellStatistics 18;D:\Momeni\AOIs\teton_aoi\Default.gdb\CellSta_rock1 D:\Momeni\AOIs\teton_aoi\Default.gdb\CellSta_1 MINIMUM DATA Start Time: Tue Nov 03 15:21:27 2015 Succeeded at Tue Nov 03 15:21:32 2015 (Elapsed Time: 5.00 seconds) Executing (Raster Calculator: AWC * (min of rooting depth and 18)): RasterCalculator ""D:\Momeni\AOIs\teton_aoi\Default.gdb\AWC" * "D:\Momeni\AOIs\teton_aoi\Default.gdb\CellSta_1"" D:\Momeni\AOIs\teton_aoi\Default.gdb\rastercalc1 Start Time: Tue Nov 03 15:21:32 2015 Raster(r"D:\Momeni\AOIs\teton_aoi\Default.gdb\AWC") * Raster(r"D:\Momeni\AOIs\teton_aoi\Default.gdb\CellSta_1") Succeeded at Tue Nov 03 15:21:36 2015 (Elapsed Time: 4.00 seconds) Executing (Zonal Statistics as Table: rechr_max Mean): ZonalStatisticsAsTable D:\Momeni\AOIs\teton_aoi\zones\elev_15\elev_15.gdb\grid Value D:\Momeni\AOIs\teton_aoi\Default.gdb\rastercalc1 D:\Momeni\AOIs\teton_aoi\Default.gdb\ZonalSt_rechrmean NODATA MEAN Start Time: Tue Nov 03 15:21:37 2015 Succeeded at Tue Nov 03 15:21:39 2015 (Elapsed Time: 2.00 seconds) Executing (rechr_max Copy_Field_To_Params_Table): CopyFieldToParamsTable D:\Momeni\AOIs\teton_aoi\zones\elev_15\param.gdb\Profile_eWSF_PRMS_params D:\Momeni\AOIs\teton_aoi\Default.gdb\ZonalSt_rechrmean MEAN soil_rechr_max Start Time: Tue Nov 03 15:21:39 2015 Executing (Join Field: join field to parameters table): JoinField D:\Momeni\AOIs\teton_aoi\zones\elev_15\param.gdb\Profile_eWSF_PRMS_params HRU_ID D:\Momeni\AOIs\teton_aoi\Default.gdb\ZonalSt_rechrmean VALUE MEAN Start Time: Tue Nov 03 15:21:40 2015 Succeeded at Tue Nov 03 15:21:40 2015 (Elapsed Time: 0.00 seconds) Executing (Add Field: Add a new field to param table): AddField D:\Momeni\AOIs\teton_aoi\zones\elev_15\param.gdb\Profile_eWSF_PRMS_params soil_rechr_max FLOAT # # # # NULLABLE NON_REQUIRED # Start Time: Tue Nov 03 15:21:40 2015 Adding soil_rechr_max to D:\Momeni\AOIs\teton_aoi\zones\elev_15\param.gdb\Profile_eWSF_PRMS_params... Succeeded at Tue Nov 03 15:21:41 2015 (Elapsed Time: 1.00 seconds) Executing (Calculate Field: Assign values from joined field to new field): CalculateField D:\Momeni\AOIs\teton_aoi\zones\elev_15\param.gdb\Profile_eWSF_PRMS_params soil_rechr_max [MEAN] VB # Start Time: Tue Nov 03 15:21:41 2015 Succeeded at Tue Nov 03 15:21:42 2015 (Elapsed Time: 1.00 seconds) Executing (Delete Field: Delete joined field): DeleteField D:\Momeni\AOIs\teton_aoi\zones\elev_15\param.gdb\Profile_eWSF_PRMS_params 'MEAN' Start Time: Tue Nov 03 15:21:42 2015 Dropping MEAN from D:\Momeni\AOIs\teton_aoi\zones\elev_15\param.gdb\Profile_eWSF_PRMS_params... Succeeded at Tue Nov 03 15:21:42 2015 (Elapsed Time: 0.00 seconds) Succeeded at Tue Nov 03 15:21:42 2015 (Elapsed Time: 3.00 seconds) Executing (Raster Calculator: AWC * rooting depth): RasterCalculator ""D:\Momeni\AOIs\teton_aoi\Default.gdb\AWC" * "D:\Momeni\AOIs\teton_aoi\Default.gdb\CellSta_rock1"" D:\Momeni\AOIs\teton_aoi\Default.gdb\rastercalc Start Time: Tue Nov 03 15:21:43 2015 Raster(r"D:\Momeni\AOIs\teton_aoi\Default.gdb\AWC") * Raster(r"D:\Momeni\AOIs\teton_aoi\Default.gdb\CellSta_rock1") Succeeded at Tue Nov 03 15:21:47 2015 (Elapsed Time: 4.00 seconds) Executing (Zonal Statistics as Table: moist_max Mean): ZonalStatisticsAsTable D:\Momeni\AOIs\teton_aoi\zones\elev_15\elev_15.gdb\grid Value D:\Momeni\AOIs\teton_aoi\Default.gdb\rastercalc D:\Momeni\AOIs\teton_aoi\Default.gdb\ZonalSt_moistmean NODATA MEAN Start Time: Tue Nov 03 15:21:47 2015 Succeeded at Tue Nov 03 15:21:49 2015 (Elapsed Time: 2.00 seconds) Executing (moist_max Copy_Field_To_Params_Table): CopyFieldToParamsTable D:\Momeni\AOIs\teton_aoi\zones\elev_15\param.gdb\Profile_eWSF_PRMS_params D:\Momeni\AOIs\teton_aoi\Default.gdb\ZonalSt_moistmean MEAN soil_moist_max Start Time: Tue Nov 03 15:21:49 2015 Executing (Join Field: join field to parameters table): JoinField D:\Momeni\AOIs\teton_aoi\zones\elev_15\param.gdb\Profile_eWSF_PRMS_params HRU_ID D:\Momeni\AOIs\teton_aoi\Default.gdb\ZonalSt_moistmean VALUE MEAN Start Time: Tue Nov 03 15:21:50 2015 Succeeded at Tue Nov 03 15:21:51 2015 (Elapsed Time: 1.00 seconds) Executing (Add Field: Add a new field to param table): AddField D:\Momeni\AOIs\teton_aoi\zones\elev_15\param.gdb\Profile_eWSF_PRMS_params soil_moist_max FLOAT # # # # NULLABLE NON_REQUIRED # Start Time: Tue Nov 03 15:21:51 2015 Adding soil_moist_max to D:\Momeni\AOIs\teton_aoi\zones\elev_15\param.gdb\Profile_eWSF_PRMS_params... Succeeded at Tue Nov 03 15:21:51 2015 (Elapsed Time: 0.00 seconds) Executing (Calculate Field: Assign values from joined field to new field): CalculateField D:\Momeni\AOIs\teton_aoi\zones\elev_15\param.gdb\Profile_eWSF_PRMS_params soil_moist_max [MEAN] VB # Start Time: Tue Nov 03 15:21:51 2015 Succeeded at Tue Nov 03 15:21:52 2015 (Elapsed Time: 1.00 seconds) Executing (Delete Field: Delete joined field): DeleteField D:\Momeni\AOIs\teton_aoi\zones\elev_15\param.gdb\Profile_eWSF_PRMS_params 'MEAN' Start Time: Tue Nov 03 15:21:52 2015 Dropping MEAN from D:\Momeni\AOIs\teton_aoi\zones\elev_15\param.gdb\Profile_eWSF_PRMS_params... Succeeded at Tue Nov 03 15:21:52 2015 (Elapsed Time: 0.00 seconds) Succeeded at Tue Nov 03 15:21:53 2015 (Elapsed Time: 4.00 seconds) Executing (Validate soil_rechr_max <= soil_moist_max): CalculateField D:\Momeni\AOIs\teton_aoi\zones\elev_15\param.gdb\Profile_eWSF_PRMS_params soil_rechr_max new_value VB "Dim new_value\n\nIF [soil_rechr_max] > [soil_moist_max] THEN\n new_value = [soil_moist_max] \n\nELSE\n new_value = [soil_rechr_max] \nEND IF" Start Time: Tue Nov 03 15:21:53 2015 Succeeded at Tue Nov 03 15:21:53 2015 (Elapsed Time: 0.00 seconds) Executing (Add Field: Add "soil_rechr_init" Field): AddField D:\Momeni\AOIs\teton_aoi\zones\elev_15\param.gdb\Profile_eWSF_PRMS_params soil_rechr_init FLOAT # # # # NULLABLE NON_REQUIRED # Start Time: Tue Nov 03 15:21:53 2015 Adding soil_rechr_init to D:\Momeni\AOIs\teton_aoi\zones\elev_15\param.gdb\Profile_eWSF_PRMS_params... Succeeded at Tue Nov 03 15:21:54 2015 (Elapsed Time: 1.00 seconds) Executing (Calculate Field: soil_rechr_init): CalculateField D:\Momeni\AOIs\teton_aoi\zones\elev_15\param.gdb\Profile_eWSF_PRMS_params soil_rechr_init "[soil_rechr_max] * 0.25" VB # Start Time: Tue Nov 03 15:21:54 2015 Succeeded at Tue Nov 03 15:21:54 2015 (Elapsed Time: 0.00 seconds) Executing (Add Field: Add "soil_moist_init" Field): AddField D:\Momeni\AOIs\teton_aoi\zones\elev_15\param.gdb\Profile_eWSF_PRMS_params soil_moist_init FLOAT # # # # NULLABLE NON_REQUIRED # Start Time: Tue Nov 03 15:21:55 2015 Adding soil_moist_init to D:\Momeni\AOIs\teton_aoi\zones\elev_15\param.gdb\Profile_eWSF_PRMS_params... Succeeded at Tue Nov 03 15:21:55 2015 (Elapsed Time: 0.00 seconds) Executing (Calculate Field: soil_moist_init): CalculateField D:\Momeni\AOIs\teton_aoi\zones\elev_15\param.gdb\Profile_eWSF_PRMS_params soil_moist_init "[soil_moist_max] * 0.25" VB # Start Time: Tue Nov 03 15:21:55 2015 Succeeded at Tue Nov 03 15:21:56 2015 (Elapsed Time: 1.00 seconds) Executing (Add Message Tool): AddMessageTool "soil_moist_init and soil_rechr_init are set to 25% of soil_moist_max and soil_rechr_max values." Start Time: Tue Nov 03 15:21:56 2015 ERROR 000576: Script associated with this tool does not exist. Failed to execute (Add Message Tool). Failed at Tue Nov 03 15:21:56 2015 (Elapsed Time: 0.00 seconds) Failed to execute (soilMoistMax3). Failed at Tue Nov 03 15:21:59 2015 (Elapsed Time: 1 minutes 21 seconds)

[lbross]

For some reason, it did run yesterday,even though the results were all -99, so I can ask the following question(s). I identified the type of message that is sent as esriGPMessageTypeWarning. Should the framework treat any warning of this type the same? That is, pop a message box and record in the parameter log.

The best place for you to troubleshoot the model/AOI may be on Masoud's computer. Both items are there and available remotely.

[jdduh]

It's the look up tool! Madeline put a note somewhere saying that the tool doesn't work on 10.0. She had developed a workaround method to create a raster based on an attribute field of a vector featureclass. Now I have to figure out a solution if we want to test the model on 10.

[lbross]

Questions on how to handle the warning message:

1. Should the framework treat any message of type esriGPMessageTypeWarning. Should the framework treat any warning of this type the same? That is, pop a message box and record in the parameter log.
2. The example message is prefixed with BAGIS-P Message. Should the framework filter for only messages with this prefix? If so, may want to change it to just BAGIS message to be more generic?
3. Should the prefix be excluded from the warning message that is displayed/saved?
4. Should the warning message display in red in the parameter log? Currently error messages do display in red

[jdduh]

1. Please check only the Warning messages. There are usually not that many warning messages when the models are executed.
2. Yes, I have changed the prefix to just "BAGIS:" You can modify the 10.0 model for your testing.
3. Please include the prefix.
4. Display in red.

The JH_Coef model also adds BAGIS specific messages (for example, "August is the hottest month"). I have also changed the message type to warning and added the BAGIS prefix.

[lbross]

All BAGIS-P coding is complete for this issue. Leaving issue open to address issue with LookUp tool in ArcGIS 10.0 that prevents the Soil_Moist_Recharge_Parameters model from generating results on ArcGIS 10.0. Model completes successfully but returns no data.

[lbross]

Revisit to make sure comments are being handled correctly. Download new models/profiles from basins. Use Sprague. The 25% message should always show up in red on the log file.

[lbross]

Made some minor code changes to support passing the messages to the parameter calculation summary log. BAGIS-P was configured to pop message boxes while the parameters were calculating if a warning message was generated by the model but this might slow down workflow because the calculation stops until the message is acknowledged. I disabled the message boxes.

My design only accounted for a single message but the Soil_Moist_Recharge_Parameters model passes two. I updated the code to append all messages together into a single string with line breaks between each message. Attaching a sample log to this issue (change the file extension to .html). If you don't like the way this looks, I can update the code to store a collection of messages and print them more elegantly in the log.

Profile_soils_params_log.txt

Version 1.9.5 of the BAGIS-P add-in includes these fixes.

One of the messages passed by Soil_Moist_Recharge_Parameters appears to be a debug message so that model should be updated.

[jdduh]

Looks good. I will remove the debug message from the model.

[lbross]

@jdduh Can we close this issue? I think we were leaving it open because of the bug with the LookUp tool in ArcGIS 10.0 that prevented the model from returning valid results. Now that we are on 10.2.2, this should be irrelevant.

[lbross]

@lbross to research what search string is for a BAGIS message. @jdduh to verify that it is working as designed.

[lbross]

The search string for a BAGIS messages is 'BAGIS'. The JH_Coeff model successfully writes a message to the log. This model prefixes the message with 'BAGIS: '.

BAGIS-P trims the prefix from the message before saving it to the log. It is expecting the ":" so it trims the search string + 2 characters from the beginning of the message. This code is well-designed so if you want to change the search string or how the message is saved to the log, it would be a simple fix in only one place.

I believe there is also a message in the soil recharge model but it failed when I tried to run it so I couldn't verify. Opened issue #30 for this error. I pulled all the latest models from basins before testing. The sample log posted on the Jan 14 comment is representative of how the log looks. Just change the suffix from .txt to .html to view.

@jdduh: Let me know if we should change anything with the warning messaging.