Support Minnesota WEPS work

[akrherz]

A TODO list based on discussions with our collaborators in Minnesota wishing to use WEPS along side our DEP runs in the state.

See TODO list now in the comment below

[akrherz]

An alternative SWEEP, see David James email 21 Sep 2017

[bdalzell]

Yes, I've been working with the SWEEP stand alone .exe for the past several months and have a good sense of the required inputs. (which are similar to WEPS) It seems that the IA State mesonet site will be a good source for wind speed data (and I see that your email is listed as contact for that site). The data appear to be in 20to30 minute intervals, but they are not uniform so we will need to find a way to condition them to make the data suitable for SWEEP input file.

[akrherz]

@bdalzell Great!

Currently, I produce a 0.125x0.125 degree lat/lon gridded hourly analysis of various weather variables. This information does not include wind gusts though and only 'steady state' winds. Do you think this will be insufficient for your needs?

[bdalzell]

The SWEEP documentation doesn't make any mention of requiring data on wind gusts so I think this hourly data set is a great place to start. I'm in favor of making the most of existing available data sets - at least for this initial effort. Once we have the initial model running, I can do some additional testing so see how sensitive the results are to the time resolution of the wind data.

[akrherz]

@bdalzell Do you have a document describing the SWEEP weather file format?

[bdalzell]

Hi Daryl, (apologies for the delay, my hard drive died and I'm still piecing my digital life back together. luckily, there was not much data on there - just all the programs)

Here is the SWEEP user guide, the weather information is on page 24.

Perhaps more helpful for you is an example of the text input file required to run SWEEP. I added ".txt" to the file name so GitHub would recognize it. I think the simplest way to link SWEEP with DEP is to generate this text file directly, update the weather and crop data as needed, then run the .exe. (I've been toying around with this with an R script.)

The wind data appear near the bottom of the file (9th row from the end, by my count). It is the very long row beginning with the value 18.004. Data points are separated by spaces.

Also important is the value for "nstep" located 24 rows from the bottom.

In this example, the wind data are in 20 minute intervals (nstep = 72). Wind speed is m/s.

Norman_195.in.txt SWEEPUserGuide.pdf

[akrherz]

@bdalzell Thanks, but oh my. This model only runs for a single day? I guess I am now confused about how this would be used for this project? Could you kindly elaborate on your vision for that?

[bdalzell]

@akrherz When run in its full form, the full WEPS model requires data on: climate, soils, farm management, and crops

the WEPS model then simulates: hydrology (soil moisture), management disturbances on soil surface, crop growth and residue (and more).

Since you are already running WEPP, it doesn't seem efficient (to me) to try to completely parameterize and run WEPS independent of WEPP.

That's where the SWEEP daily erosion model comes in. I think there are a lot of data from WEPP that can be used to build the SWEEP input text file.

My plan was to: 1) build the initial SWEEP input file (this would be manual in the beginning, but I'd like to find some way to automate this - or semi-automate). 2) look at wind speed data to determine if it exceeds threshold values, if above the threshold - then: 3) update the key parts of the SWEEP input file based on WEPP data (soil moisture, crop residue). 4) run SWEEP and report the results.

some additional notes: SWEEP runs fairly quickly, I set up a batch to run through R: a test run of 20 fields took about 8 seconds.

The threshold wind speed is fairly high so SWEEP would lie dormant most of the time. (though I still need to work out the best way to determine the threshold value across multiple soil types).

Here are some old slides that summarize my thinking on this. DEP_WEPSoverview.pptx

What do you think?

[akrherz]

Thanks for the detailed response. I worry about the automation aspect and the spatial scale. What is your ideal for number of 'fields' to run each day via automation? Are you hoping that the automation runs on my end or yours? I wouldn't worry about the thresholding aspect, just run all needed runs each day.

[bdalzell]

I'm planning to run a sensitivity analysis (manually) to work out how many 'fields' need to be run. Since you guys are reporting results at the HUC12 scale, I plan to randomly select points within a HUC12 and will run them for a test day with high wind (and low crop residue).

An alternative would be to look at a county-scale (or something larger than HUC12). This could potentially reduce the number of fields that we'd need to generate input files for.

Based on what I've done so far, there doesn't appear to be a lot of variability between fields so I have my fingers crossed that it'll be a manageable number.

I worry about the automation too - but want to get the data handshake between WEPP and SWEEP worked out first to be certain that we can actually run SWEEP in this manner. One potential result of these efforts will be the conclusion that we need more expertise and dedicated developer effort to automate the field parameterization process. But if we can at least identify the pathway forward - I think that'll be a strong first step.

I got my PC back yesterday afternoon with a new hard drive - I'm getting software installed now and I hope to begin my sensitivity analysis work today.

[bdalzell]

@akrherz Can you provide me with an example WEPP output file that includes data on soil (texture, moisture) and plant growth / residue? I want to begin thinking about how to collect these data in order to generate the WEPS file. Thanks!

[akrherz]

@bdalzell Uffties, sorry again for the lengthy delays. Hope springs eternal that I am back at this. With regards to WEPP outputs. I am unsure I have anything on soil texture. The soil moisture (water balance) files I have are modified from what the baseline output from WEPP is, so they are custom to our project :( The plant growth and residue are outputs that we currently do not routinely output. So I am striking out big-time here. Perhaps if you installed WEPP on your windows PC and then interrogated the resulting text files from an example run, that would help your learning process?

[bdalzell]

@akrherz I'm confused about how you can run WEPP without soil texture data. Surely, there must be an underlying database that is used to generate the WEPP input files? Is there a way to change WEPP in order to generate those plant growth and residue outputs? I will try to look into WEPP directly, but was hoping to avoid re-inventing the wheel by relying on existing DEP expertise. I have one other lead on running WEPS at a broader scale, so I'm looking into that too. Will let you know when I learn something.

[akrherz]

@bdalzell your question to me was to provide "example WEPP output file that includes data on soil (texture, moisture)", with regards to texture, I only have input files to WEPP, but they are in the WEPP soil file format, which I didn't think SWEEP would be able to understand. For this dataset, it may be easier for Gelder/James to generate what you need using their databases, which I don't have access to.

Yeah, we can output plant growth and residue, but I am nervous about doing that each day as it adds to the runtime and things are getting tight timeline wise with execution.

We have a local team meeting tomorrow and will hit on some of these issues there to see what others think.

[bdalzell]

Thanks @akrherz !
I understand that the data format will be different for WEPP vs WEPS, I was hoping to find a way to extract the key data, then re-format them in a WEPS-friendly way.

My other lead (at NRCS) is out this week, but I hope to have more information from her next week. I'll be curious to see how they handle the crop growth / residue data in their efforts.

[bdalzell]

@akrherz I've made some additional headway with extracting WEPP data that will be necessary for generating input for WEPS/SWEEP. More specifically, I can find most of the data I need in the WEPP output files: "grph_0.txt" and "p0.sol" When I run WEPP on my desktop, the output files are for one year (of long-term average climate conditions). It is not yet clear to me how these files will look when they are updated daily as you do with DEP. Do you have any thoughts on this? Or can you provide me examples of the the grph_0.txt file? (no rush on this, I'll be out of the office for the rest of the week). -Brent

[akrherz]

@bdalzell Good morning. So for the Daily Erosion Project, we run the command line version of WEPP on Linux, so there are some differences with the workflow and files. But I believe the WEPP GUI eventually runs the same command CLI fortran code that we use, so there should be some conformity here.

So the grph_0.txt and p0.sol files, I am unsure of. Could you kindly email those to me and I can see what their structure and what our equivalents are?

When we run DEP/WEPP, we provide a run file that specifies a number of options for which output files and their structure should be generated. So it is just a matter of matching up what the run file is that the WEPP GUI generates for you.

[bdalzell]

@akrherz , Here are the samples of the grph_0.txt and p0.sol files. Github is unhappy about the ".sol" file extension, so I renamed it to ".txt"

I also included a p0_annotated.txt file which includes some comments from me to explain the data fields.

For SWEEP, there is decent documentation about different options when running via command line. I did a little searching for similar WEPP instructions but have been unsuccessful so far. Both models are structured similarly and I'm aware of other efforts to link them together so, hopefully, there should be similar sets of command line options.

For my part, I've been working on pulling key data out of the grph_0.txt file and inserting it into the SWEEP input file (after performing necessary unit conversions).

grph_0.txt p0_annotated.txt p0.txt

FYI, there is a team phone call next Tuesday, I believe Brian G. will join. If you are available, it my be helpful for you to join. Alternatively, we can continue to communicate this way.

thanks. -Brent

[akrherz]

@bdalzell I plan to be on the call tomorrow and will summarize what we discuss on the call here.

[bdalzell]

@akrherz That sounds great. Thank you very much!

[akrherz]

@bdalzell Oye, I suck. I can hear and see the call, but my audio keeps failing. Would you like to have a call between the two of us in the near future?

[bdalzell]

@akrherz No worries. I appreciate that you joined the call and I'm encouraged that you think you'll be able to generate the grph_0.txt file from your DEP runs. In the short term, I'd like to get my hands on examples of: grph_0.txt, and p0.sol that are generated by DEP. That'll allow me to get one step closer to something that should integrate with DEP.

A phone call sometime will certainly be helpful, but it's probably not immediately necessary. I'll be out for vacation next week (and busy during the week I return). How about a phone call sometime around mid-late September?

[akrherz]

Please find an example graph file attached from a manual run I did just now. graph.txt

Here's the soil file for that same run 070200110604_25.txt

Whenever a phone call works for you, I should be good for!

[bdalzell]

awesome, thanks!

[akrherz]

@bdalzell kindly came down to ISU to visit us and our items of interest for the afternoon meeting on 3 December 2019.

• [ ] Reviewing our current soils files, it is unclear what the non-commented out SSURGO MUKEY line is doing. We'd like to include some more comments with aux data to support this work.
• [x] Found a linux version of SWEEP.
• [x] create grph, sweepin, sweepout folders within file schema tree to store output for the current 3 HUC12s of interest.
• [x] implement a script that proctors the daily execution of this system. The steps to include
• [x] editing in the hourly wind information,
• [x] running of SWEEP, and
• [ ] databasing the result.
• [x] modify WEPP src to only dump out the last year of daily data for 1 OFE for graph file.

TODO items from our 4 Dec meeting.

• [ ] We aux soil information somehow provided in the DEP SOL files (likely as comments). very fine sand and calcium carbonate.
• [ ] resolve provision of field information as static inputs. Wind breaks, direction of tillage.
• [ ] daryl figures out the databasing and web interface to show these results.
• [x] add to daily production runs, needs R and R package readr installed.
• [ ] find a real day's data that actually produces erosion. Does it even exist?
• [ ] @bdalzell comes up with some sort of validation dataset to check against.
• [ ] backfill runs to 2007 (grph file will be trouble for this).
• [ ] Resolve WEPP's code modification hack for last year run.

[akrherz]

A conference call with the WEPS/SWEEP folks today was very helpful. A main takeway I found is that perhaps WEPS is evolving the Soil layer GMD/GSD values to more erosion friendly values than what baseline SURGO provides. These are two knobs that I don't think we have done any testing with on our end.