Infiltration Editor Meeting

[FLO-2DKaren]

@rpachaly I figured out a great project that will engage your engineering skills and your python skills. Let's do a meeting and see if we can start a plan.

[rpachaly]

Awesome! I'm available after 1PM your time. Just let me know the best time for you.

[FLO-2DKaren]

@rpachaly

OK I'll set it for 1pm and if you can't make it exactly at that time, just let me know. The meeting can be reused or restarted at anytime.

Raincell and Infiltration Editor Meeting Jun 14, 2023, 1:00 – 1:30 PM (America/New_York)

Please join my meeting from your computer, tablet or smartphone. https://meet.goto.com/820619765 You can also dial in using your phone. Access Code: 820-619-765 United States: +1 (571) 317-3122

• One-touch: tel:+15713173122,,820619765#

Get the app now and be ready when your first meeting starts: https://meet.goto.com/install

[rpachaly]

1PM works for me!

[FLO-2DKaren]

Soils-LC-AZ-Eloy.zip

[FLO-2DKaren]

Load the infiltration data and build a small project to use it on.

[FLO-2DKaren]

More Ideas Soil Use Curve Number Generator like code to sample SURRGO for: soil - chorizon and any table that links the chorizon ID to the MUKey NDOT_Rainfall_Loss_Parameters JEFULLER.pdf

Use JE Fuller methods to calculate DTHETA, PSIF, and XKSAT.

Landuse Land use data is the main problem. Curve number generator uses the rasters on NCLD. That might be OK for a base resolution but it kind of sucks for roads, and paved impervious areas.
Let's explore using OpenStreetMap for our impervious overlay. It's good data for practically the whole world.

• Buildings
• Paved Surfaces
• Highways

[rpachaly]

@FLO-2DKaren Hi Karen, I'll take a look in this issue while we wait for the green ampt issue approval.

[rpachaly]

Let me see if I understood this issue correctly for the SOIL:

1. SSURGO data: Download relevant soil data from NRCS SSURGO database - hzdept, hzdepb, sandtotal, silttotal, claytotal, fragsize, fragvol - from the area of interest (a.k.a computational domain) following the Curve Number Generator code.
2. JE Fuller methods: Use the equations presented on the section 2.4 to calculate DTHETA, PSIF, and XKSAT for the downloaded data.
3. Use this calculated DTHETA, PSIF, and XKSAT to compute the Green-Ampt.

Basically, I need to add a new method on the Calculate Green-Ampt dialog that performs JE Fuller methods for estimating DTHETA, PSIF, and XKSAT instead of selecting directly the DTHETA, PSIF, and XKSAT from the soil layer. Is that correct?

[rpachaly]

LAND USE:

I was looking for anything that could help us to identify the roads, buildings, and paved surfaces and I've found this plugin QuickOSM. Take a quick look in this youtube video. Basically, this plugin can obtain the streets from OpenStreetMap. However, it does not return a raster but a shapefile. This is a problem, but I was checking and they have the number of lanes in the attribute table. Maybe we can apply a buffer based on the lanes and then convert it to a raster. After that, we can set these areas with a very high percent impervious area. What do you think? I've never seen this plugin before, but it looks promising.

[FLO-2DKaren]

@rpachaly I was just looking at this a few days ago. I think it is a good resource.

I wonder why the streets are coming down as polylines. Because if you look at the OSM map, they are clearly defined as polygons. I bet we can get them as polygons a different way.

[FLO-2DKaren]

Hi Robson,

That's kind of what I was thinking for the SSURGO or STATSGO data. Put it into our own processor like the Curve Number Generator but for Green-Ampt data.

[rpachaly]

@rpachaly I was just looking at this a few days ago. I think it is a good resource.

I wonder why the streets are coming down as polylines. Because if you look at the OSM map, they are clearly defined as polygons. I bet we can get them as polygons a different way.

Yeah, I think it could be possible to do that.

Hi Robson,

That's kind of what I was thinking for the SSURGO or STATSGO data. Put it into our own processor like the Curve Number Generator but for Green-Ampt data.

Awesome! I'll start working on this today. I'll add a button to the Compute Green-Ampt to run this new method for computing the Green-Ampt parameters. We can change the place of this button later, but the code for calculating everything will be working.

Also, did you receive any information regarding the New Green-Ampt Method issue?

[FLO-2DKaren]

@rpachaly We have a meeting today and we will get the OK and which method to proceed with.

[rpachaly]

Just letting you know that I'm working on this and a lot of progress was made.

I'm able to download the soil data required for calculating DTHETA, PSIF, and XKSAT using the JE Fuller methods. However, the Curve Number Plugin it is able to download data using a Post Request and a Wfs Request. I don't know why, but my post request misses some features (take a look on the picture). I'll try the wfs in the next days to see if I can get better results. I believe when this issue is solved, calculating the G&A parameters would be easy.

[FLO-2DKaren]

@rpachaly If you get stuck on some of that soil data, we can ask for help from JEFuller.

[FLO-2DKaren]

@rpachaly For next week, we need to allow the plugin to read all words regardless of capitalization for wet, saturated, dry, or normal.

[rpachaly]

Ok! This will be the first thing that I'll do tomorrow.

[rpachaly]

read all words regardless of capitalization for wet, saturated, dry, or normal.

@FLO-2DKaren Hi Karen,

Just finished and pushed the branch. I also added an exception for when there is no wet, saturated, dry or normal on the saturation field. I ran some tests here and all good. Let me know if everything is working as expected.

I'll continue working on the JEFuller method for infiltration.

[rpachaly]

I'm facing a hard time here. There are two problems here:

1. No data available: I think the solution here is to just set the Global Infiltration Parameters in areas where no data is available. Sometimes we do have the polygon but NULL data for the required fields. What do you think?
2. This is where I'm stuck right now. When I prepare the query for getting the data from SSURGO database, I need to join the chorizon, component, and mupolygon tables. This is returning multiple values of hzdept, hzdepb, sandtotal, silttotal, claytotal, fragsize, fragvol for the same mupolygonkey. What should I do in this case? Averaging out the parameters makes sense? I've been studying the SSURGO database, but I'm not sure about the best way to proceed with this.

[FLO-2DKaren]

1. No Data Available. Generic Green-Ampt parameters don't always match. The tables for HEC-RAS are different than the Green Ampt tables for AZDOT and NVDOT. So let's ask JE Fuller and see what they say.
2. JE Fuller only did the top 3 inches of soil for their variables. Make sure you are only looking at the codes for the top 3 inches.
3. If there are more than one value for the top 3 inches, we can average or we can take the confining value. I think we want to limit the infiltration not increase it. I think we want the more restrictive value.
4. JE Fuller might have used STATSGO for their data. It seems to be soil data specific to states. I would rather use more generic data since we hope to offer it as a method in other countries.

[rpachaly]

Some preliminary results and updates here:

results.pptx

In this presentation, I'm bringing the results comparing WPOINT, FCAPAC, SAT, PSIF, XKSAT, DTHETANrm, DTHETADry from the soil layer provided and the FE Fuller method. In my opinion, we're getting satisfactory results for WPOINT, FCAPAC, SAT, and PSIF. However, the XKSAT and the DTHETAs are still not good. They are deviating too much from the soil data. The code isn't totally ready, I need to organize some blocks, add some progress bars and check it again to see if the calculations are correct.

What do you think about my testing procedure? It would be nice to have another soil layer containing all these fields to test in a different location.

• I'm taking the first soil horizon to perform the calculations. Sometimes it is deeper than 3 inches. If there are multiple data for the same horizon, I'm taking the average of these data.
• The comparison was performed considering the same symbology.
• I'm using data from SSURGO.

When I get almost done, I would like to have a meeting to go through all the steps that I performed. Just to check carefully if everything is making sense and that the code is clear.

[FLO-2DKaren]

@rpachaly Hi Robson,

You are using the AZ DOT data. It is a bit older than the NV report and it's DTHETA columns are mixed up.

Let's use the Nevada Data or the Maricopa County Data. That way we know we're comparing the same process.

Maricopa County https://flo-2d.sharefile.com/d-sf1379a05d6b044a39486311e03ef4e26

Nevada GreenAndAmpt_STATSGO2.zip

[FLO-2DKaren]

@rpachaly We can have a meeting at any time. We don't have to wait for you to finish anything. Sometimes it's helpful to bounce ideas in realtime.

[rpachaly]

Hi Karen,

I agree with you! Having a meeting and brainstorming is always good. I just started working here. When you are available, just sent me the link that I'll join the meeting.

[FLO-2DKaren]

@rpachaly Here is a link.

Green-Ampt Method Jul 5, 2023, 12:30 – 1:00 PM (America/New_York)

Please join my meeting from your computer, tablet or smartphone. https://meet.goto.com/137639309 You can also dial in using your phone. Access Code: 137-639-309 United States: +1 (224) 501-3412

• One-touch: tel:+12245013412,,137639309#

Get the app now and be ready when your first meeting starts: https://meet.goto.com/install

[FLO-2DKaren]

@rpachaly

Notes from meeting.

Look at how curve number generator message that happens when the SSURGO site is down. Curve number generator used to limit the size of the download layer. We need to test this on our side. CNG transforms the data to the project CRS. Let's do that too.

Find out how JE Fuller decided which data to use when there is more than one. Find out what they do when the first layer is less than 3 inches.

Testing Run testing to compare to Nevada Shapefile.

Unit conversion There should be a metric switch for one of the tables in the plugin. Final units for English are inches or inches per hour Final units for Metric are mm or mm/hr

Let's add module documentation for all the definitions

[FLO-2DKaren]

@rpachaly Thanks Robson, I'm excited for this new code.

[rpachaly]

You're welcome Karen! Thanks for the brainstorming.

[rpachaly]

Hi Karen,

I didn't find any reference on how to calculate the soil's rock outcrop field. Do you have any material that shows how to calculate this field from the soil data? Or is there any other place where I can get this data?

[FLO-2DKaren]

I think it is in a table called map unit composition. I haven't been able to access that table but I see it in the soil report. Map Unit Composition I got this database here that has a lot of rock outcrop. Maybe you can find it with an access sql query.

wss_aoi_2023-07-07_14-55-08.zip

[rpachaly]

Hi Karen,

Some updates here. I've finished the portion of the code for the SSURGO data. I just need to run some tests here, but I already covered the great majority of the points that we discussed last week:

• Layers are converted to the project CRS

  I convert to WGS84 to create the post request on the NRCS database, then I convert it back again to the project’s CRS.

• Units

  Metric are switched based on the CRS project units.

• Holes

  The holes were filled based on the closest feature. This was very complicated because the holes are not always inside a feature. In this case, the fill the holes processing algorithm works fine. However, sometimes they are in-between two or more features. That’s why I used the approach of selecting the closest feature.

• Huge computational domain

  Bigger CDs crash the code due to the post request. Here is the solution that I’ve thought: Let’s break the computational domain in smaller ones, and then execute the code. I think this is not an easy task to perform, but creating this sort of algorithm for FLO 2D could be useful in other parts of the code. What do you think?

• Rock outcrop

  I still didn’t figure out a way of obtaining this data.

At this point I didn't find a good approach to use the STATSGO2 dataset to calculate the G&A parameters. I was reading, and as far as I understood, the STATSGO data is derived from SSURGO data. SSURGO provides detailed and local-level soil information, while STATSGO provides a more generalized view of soil properties at a regional scale. In my opinion, the SSURGO data is better for our purpose.

[FLO-2DKaren]

@rpachaly Hi Robson,

It might be as simple as using a key word search of the Soil Description.  I was seeing rockoutcrop in the soil description.  Even the Percentage was in there.  Can you try that?

[rpachaly]

I also see the rockoutcrop on the table, but when I perform the query it returns an empty dictionary. I'll be focusing on this today. If the data is there, there is a way of getting it.

[FLO-2DKaren]

@rpachaly Add this to your search terms. PERC_ROCK or Percent Rock

I also found this training manual from NDOT.

Green_And_Ampt_Self_Help_Training.pdf

[FLO-2DKaren]

I don't think the Curve Number Generator is working properly for Rock Outcrop.

Here is a test area.

The muname is RockLand

The curve number is 62 which is not appropriate. This area is like 95% rock coverage.

NLCD doesn't seem to recognize rock outcrop either.

[FLO-2DKaren]

I read this in the Self Help Training.

Soils imperviousness is included in the NDOT soils data shapefiles as PERC_ROCK or Percent Rock. This terminology is consistent with the original NRCS soils data. For the purposes of consistency with Green and Ampt reference documentation, Soils (Natural) Imperviousness is used in this document.

[rpachaly]

Yeah, I think that getting rockoutcrop data from NRCS is going to be a pain in the neck. But I'm getting the rockoutcrop data through the component table, just need to adjust some stuff in the code. Then we can test to see if the data worth it. After that I'll look into this percent rock.

[rpachaly]

The rockoutcrop data is from the STATSGO2 dataset. I cannot combine it with the SSURGO data.

[rpachaly]

The rockoutcrop data is from the STATSGO2 dataset. I cannot combine it with the SSURGO data.

Forget this.

[rpachaly]

From the AOI that you sent me:

20% rock outcrop in this feature from the NRCS dataset.

What do you think?

[rpachaly]

I don't think the Curve Number Generator is working properly for Rock Outcrop.

Here is a test area.

The muname is RockLand

The curve number is 62 which is not appropriate. This area is like 95% rock coverage.

NLCD doesn't seem to recognize rock outcrop either.

I just ran the code for this location and no rock outcrop was found.

[rpachaly]

[FLO-2DKaren]

Haha Soooo Annoying. The polygon is literally named RockLand

[rpachaly]

Can we have a meeting if you're available?

[FLO-2DKaren]

@rpachaly I just opened this one. QGIS Green Ampt Development Meeting Tue, Jul 11, 2023 3:30 PM - 4:00 PM (EDT)

Please join my meeting from your computer, tablet or smartphone. https://meet.goto.com/804029693

[FLO-2DKaren]

@rpachaly Soil Data Once you are satisfied with your process, test it in another two states. Colorado Alabama

Landuse Data Assess the possibility of a base layer from OSM. Initial saturation condition Dry, Normal, Wet Vegetative Cover % Initial Abstraction - mm or in

Impervious Roadways building Paving

[rpachaly]

Hi Karen

I tested the methodology on Alabama and Colorado states. Here are the projects:

ALABAMA (meters) alabama_project.zip

COLORADO (feet) colorado_project.zip

It is interesting Colorado has rock outcrop data. I don't have a lot of experience in the G&A parameters to evaluate if they are in the correct range - that's why I'm sending the projects here.

There is another thing. The SSURGO database has KSAT data. Their definition is: "The amount of water that would move vertically through a unit area of saturated soil in unit time under unit hydraulic gradient.". This is the saturated hydraulic conductivity. The JE Fuller method calculates the saturated hydraulic conductivity and then multiplies it by a factor of 0.5 to obtain the hydraulic conductivity. This is stated on the NDOT JE Fuller methods:

"The Saxton and Rawls (2006) equations yield Ks, or saturated hydraulic conductivity, not effective hydraulic conductivity. Ks was multiplied by the correction factor of 0.5 to obtain Conductivity based on work of Bouwer (1966). This correction factor is consistent with the methods used in the 2014 ADOT Hydrology Manual."

If we are not satisfied with the hydraulic conductivity data, we can use SSURGO ksat * 0.5 and use the other parameters from the JE Fuller method (dthetas & PSIF).

[FLO-2DKaren]

@rpachaly

I am also very new to developing the Green-Ampt parameters. I trust the JE Fuller methodology. They have been working on that method for over a decade.

We can apply whatever methods JE Fuller use for xksat, psif, and dtheta.

When I review your data, I will use generic lookup table to spot check a range of values. The main thing I'm hoping for is general estimates being in a good range and the max and min values to be reasonable.

[FLO-2DKaren]

@rpachaly Notes from my initial inspection. I couldn't download the Alabama data. Maybe send it to me directly.

1. Add a Floor value so that xksat doesn't go below 0.01 in/hr or 2.54 mm/hr

XKSAT(natural) = XKSAT(full saturation) * KsCF If XKSAT(natural) < 0.01 Then XKSAT(natural) = 0.01

2. Add a Soil Description for the final soil layer.
   

3. XKSAT numbers seem low. I see 0.33 to 0.5 for sandy loam in the Maricopa data. Our data is less than 0.08 for sandy loam.

[rpachaly]

I sent the Alabama project to your email. The Maricopa data isn't correct. I'm going to send you the correct data.