Adding new Feature/pavd

[drnimbusrain]

Some cleaning and testing needed, but eventually this draft PR will close #66 and add the ability of canopy-app to read 3D inputs of observed GEDI Plant Area Volume Density (PAVD) profiles for a user-set range of latitude and longitude bounds. Previously canopy-app could only be driven by prescribed plant shape distribution functions.

Note: The PAVD input data example file is based on preliminary GEDI gridded/raster data provided from Scott Goetz and Pat Burns of NAU. The following is needed to move this PR forward:

• [x] Use the updated version of their gridded PAVD dataset on the NAU HPC Portal
• [x] Verification and comparisons of PAVD highest height bins against updated 1 km GEDI FCH (also used in canopy dataset blended with Potapov FCH for gap filling and in high latitudes) from NAU HPC Portal. May need to establish an altitude cutoff threshold for real vs negligible PAVD.
• [x] Integrated PAVD (i.e., PAI) comparison against updated, gridded 1 km VIIRS PAI is needed to validate spatial distribution and column values.

[drnimbusrain]

@angehung5 Thanks for the updated daily LAI in SE files. Can you remove the 'time' column from all updated txt files? canopy-app doesn't use the timestamp in txt files currently, and you can see the CI fails when trying to run .txt files.

Also, can you update these .NC files here at the feature/pavd branch for new PAVD product too?

[drnimbusrain]

@angehung5 Have you updated these SE files for PAVD projection? Seems from quick ncview they are still shifted:

How about the global hourly canopy-app files? Thank you!

[angehung5]

@drnimbusrain Sorry I grabbed the wrong files. Just fixed. They should work now. Global hourly files should be ready tonight.

[drnimbusrain]

Great Wei-Ting, thanks! I will continue development here for canopy-app.

Let's move forward with the updated PAVD max bin height vs FCH (at least with the Potapov dataset until we hear back from Pat) and the updated integrated PAVD (PAI) vs. VIIRS PAI regression comparison.

Thank you!

On Fri, Oct 27, 2023, 11:11 PM Wei-Ting Hung @.***> wrote:

@drnimbusrain https://github.com/drnimbusrain Sorry I grabbed the wrong files. Just fixed. They should work now.

— Reply to this email directly, view it on GitHub https://github.com/noaa-oar-arl/canopy-app/pull/91#issuecomment-1783678801, or unsubscribe https://github.com/notifications/unsubscribe-auth/AGLFYNRFRBAXRQS4ANDGCALYBRZVNAVCNFSM6AAAAAA4T3STROVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTOOBTGY3TQOBQGE . You are receiving this because you were mentioned.Message ID: @.***>

[angehung5]

@drnimbusrain Comparison using updated GEDI files with coordinates corrected.

Maxh is significantly lower than CH over tropical regions, and generally higher than CH by less than 5m. Since PAVD has a vertical interval of 5m, I think difference < 5m is acceptable.

Simply plotted everything here. We definitely need to look into this...maybe separate into different veg type. Any suggestions?

[drnimbusrain]

Thank you Wei-Ting!

Few questions for you:

1. The CH shown here is updated NAU product too, right?

2. Seems to me that in areas of tallest forests is where Maxh << CH. Is this roughly true, and can you show CH plot next to this?

3. For the GEDI vs VIIRS PAI, I think there may be a temporal issue. Since the GEDI PAI (based on integrated PAVD) is annual, do you average the VIIRS PAI over time? We may need to think about this comparison more temporally.

On Wed, Nov 8, 2023, 5:27 PM Wei-Ting Hung @.***> wrote:

@drnimbusrain https://github.com/drnimbusrain Comparison using updated GEDI files with coordinates corrected.

[image: GEDI_maxh_vs_rh95_1km] https://user-images.githubusercontent.com/107704243/281559269-13bc324e-60d8-43a0-b76d-bcc1d205e8a3.jpg

Maxh is significantly lower than CH over tropical regions, and generally higher than CH by less than 5m. Since PAVD has a vertical interval of 5m, I think difference < 5m is acceptable.

[image: GEDI_pai_vs_viirs_1km] https://user-images.githubusercontent.com/107704243/281559296-b78bba15-fae9-4f37-8b79-687845b6729d.jpg [image: GEDI_pai_scatter] https://user-images.githubusercontent.com/107704243/281559314-be26c013-a434-482a-a8bc-90e2e010b9bd.jpg

Simply plotted everything here. We definitely need to look into this...maybe separate into different veg type. Any suggestions?

— Reply to this email directly, view it on GitHub https://github.com/noaa-oar-arl/canopy-app/pull/91#issuecomment-1802778867, or unsubscribe https://github.com/notifications/unsubscribe-auth/AGLFYNUWNCS5TBBKDFMDLNTYDQBMHAVCNFSM6AAAAAA4T3STROVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQMBSG43TQOBWG4 . You are receiving this because you were mentioned. Message ID: @.***>

[angehung5]

1. Yes, all results are based on the most updated NAU products.

2. Yes, maxh << CH at tall forest regions. Also maxh is relatively smooth since PAVD has an interval of 5m. But again, we should check with Pat about how they define maxh.

   

3. VIIRS LAI 2020 annual average is used in this comparison while PAVD is a multi-year average across 2019-2023.

[drnimbusrain]

@angehung5 OK, starting to test this PAVD branch/PR in canopy-app again. Can you double check the PAVD data in the .NC files in this branch and make sure they are the updated product from Pat?

Seeing same issues in numerous SE test grid cells for PAVD, where there isn't much shape to profiles, which extend to really high altitudes (> 50 m).

See many values of low PAVD at relative high height bins:

This results in disagreement with much lower GEDI CHs for corresponding grid cells (although I constrain the PAVD profile inside the valid CH values). For, example a single grid cell (i = 25 and j = 25) in the SE test domain:

You can see the result is a calculated shape function from PAVD that will be linear and have no real shape distribution. This is of course problematic for all calculations in canopy-app using such profiles.

[drnimbusrain]

@angehung5 Maybe you can plot up some of the updated PAVD profiles for points across the SE test domain. What do these profiles look like, and are there any real shape? From our proposal, seems the PAVD may be able to capture some taller forests shape distributions:

[angehung5]

@drnimbusrain

It works fine on my end. GEDI CH=20.80148198 m at i=25, j=25. Please keep it in mind that GEDI PAVD is generally higher than CH, except for tall forests (see my next comment).

Are you using the SE files in the feature/pavd branch? Seems those files are not updated, while the files in the develop branch are up to date. I tried to update the files in the feature/pavd branch but somehow the system does not take the change.

[angehung5]

Just add some numbers

[drnimbusrain]

@angehung5 You can run the current feature/pavd branch and it prints out the following PAVD values for the grid cell:

Lat = 32.157498906278789 Lon = 272.81250000000000 VIIRS PAI = 3.4859482822898071 FCH = 6.3751093731480140

pavd_orig = 7.5710922479629517E-002 6.4598932862281799E-002 4.3507631868124008E-002 2.5484789162874222E-002 1.1120960116386414E-002 3.6568134091794491E-003 1.5977938892319798E-003 6.2849238747730851E-004 3.1753323855809867E-004 3.0350771339726634E-005 0.0000000000000000 0.0000000000000000 0.0000000000000000 0.0000000000000000 ...

I think you are showing a different grid point, because in Fortran the arrays go from 1, and you may be plotting in Python from array starting at 0. In that case, when I run array i = 26 and j = 26 in canopy-app, I get your results (you could check ij = 24 in your plots to see the grid cell I was looking at above).

ij = 26

Seems there still a significant issue in many grid cells in the updated PAVD, not consistent with FCH.

[drnimbusrain]

After discussion with @angehung5 we will move this forward with the preliminary evaluations and since we have not heard back from NAU about the comparisons.

Few things left to do:

• [x] Update the CH (from Potapov to new RH-95 NAU product) in all example .nc, .txt, and point_file*.txt consistently in the repos three example SE domains.
• [x] Add additional supplementary 3D text files (consistent with the three times) with only a single point column for lat, lon, pavd and associated 14 mid-point lev variables. These new point files should be consistent with time and intial lat/lon row in three SE example text files. We can name them, _point_file_3d_20220630.sfcf023.txt, point_file_3d_20220701.sfcf000.txt, and point_file_3d20220701.sfcf001.txt
• [x] Add capability of canopy-app to read in new 3D point text files with lat, lon, lev and pavd when correct options in NL and conditions (single point text examples etc.) are met in canopy-app.

[angehung5]

I would suggest sticking to RH-95 for now, which is generally used for canopy height estimation. We can update to RH-98 later if needed (data is already on Hopper).

[drnimbusrain]

@angehung5 Thanks for the updates, but there is some issue with the changes and CI...See the details here.

[angehung5]

Fixed. Now they should work.

Below shows the difference between Potapov and NAU product (~ +- 2m), FYI.

[drnimbusrain]

@angehung5 Thank you for fixing that, as it likely had to do with the CI python example run for the point based files. Seems when you uploaded the new point files with updated CH you had another vtype that wouldn't run through the options set in the python CI.

So I checked the first item based on now updated CH from NAU for all SE .nc and .txt files, correct? If so, I will continue to test these in canopy-app.

Just the other two items regarding new supplementary pavd .txt files for being able to drive canopy-app with point/column data as well.

Thanks again,

[angehung5]

@drnimbusrain Point PAVD test file added. Please go ahead and check the second item if they work.

[drnimbusrain]

@drnimbusrain Point PAVD test file added. Please go ahead and check the second item if they work.

thank you @angehung5 Let me see how best to use this extracted PAVD profile as text file in canopy-app. I wonder if we just append it into the main point text files. That begs the question then if we can/should add it to the 2D text file examples as well. Although, not sure of likelihood of any user driving canopy-app this way, or having a grid of PAVD in the text file like this.

What do you think?

[drnimbusrain]

Or, maybe the point_pavd file does need to be read in separately, and serves as an example of how a multi-dimension pavd text file could be read in (across multip grid points) as supplementary to the main text example files.

I will start with trying this way now with your files, but just wondering your thoughts here.

[angehung5]

Having a supplementary pavd file might be easier, so users who want to use parameterized shape function don't need to prepare a big input file.

[drnimbusrain]

OK, I have added the draft capability to read in PAVD from text files, and providing new point text canopy files from @angehung5 with PAVD and associated levels as example. Here is the results for comparing using the shape distribution functions from Massman vs. observed PAVD (converted to shape distribution).

Shape distribution function:

Observed PAVD:

@angehung5 As you can see the value ranges make sense, but given the coarse PAVD shape profile constrained to FCH, and not fully consistent with the FCH in the examples (i.e., FCH ~ 7 m, Max PAVD bin height ~ 17 m) lead to quite different shapes in the biogenic emissions and photolysis attenuation (both very strongly tied to the foliage shape distributions). What do you think? This is just an option for now, with known uncertainties, but maybe open the PR for full review now for more testing? Or, do you think there is any way to better address this PAVD uncertainties first?

[drnimbusrain]

Barry suggested redistributing the profile within the FCH, and not simply truncating it. I will work on that suggestion next and report back on tests.

[drnimbusrain]

@angehung5 I changed the methods of how I am using the PAVD. Given its uncertainty, and relatively coarse vertical resolution from NAU (5 m), I think one way we can use it more consistently for now is to identify the observed PAVD height of maximum vegetation, and still apply the distributions from Massman (i.e., the standard deviations above/below the maximum) based on input vegetation type.

Below is my single point test for the updated Lang FCH dataset and PAVD point profile example you provided:

Foliage Distribution Based Height Maximum

PAVD Based Vegetation Height Maximum

You can see that even though we are limiting the PAVD application for now, it works much better, and the novel aspect is we are using the observed GEDI PAVD height of maximum foliage density to constrain the shape functions that are still applied. I would need to briefly test the gridded impacts with new Lang FCH as well, but what do you think?

[drnimbusrain]

@angehung5 If you could provide a review of this as well, when you have the time. If OK, do you think we should add a sensitivity test to your paper for using the observed PAVD max from GEDI as well? You can see that it has an affect on the wind speed profile compared to default profile from Massman. Thanks!

[drnimbusrain]

@angehung5 If you could provide a review of this as well, when you have the time. If OK, do you think we should add a sensitivity test to your paper for using the observed PAVD max from GEDI as well? You can see that it has an affect on the wind speed profile compared to default profile from Massman. Thanks!

@angehung5 Can you please give this a review ASAP?

[angehung5]

@drnimbusrain A quick comparison of Massman shape function and GEDI PAVD. Seems the wind reduction based on PAVD is generally lower compared with Massman. The foliage vertical distribution is different as well, especially at Lower Teakettle, probably because PAVD values are actually averages over certain area (0.01 degree resol) and maybe affected by surrounding vegetations. It should also be noted that the vertical resolution of PAVD profile is much coarser.

[drnimbusrain]

@drnimbusrain A quick comparison of Massman shape function and GEDI PAVD. Seems the wind reduction based on PAVD is generally lower compared with Massman. The foliage vertical distribution is different as well, especially at Lower Teakettle, probably because PAVD values are actually averages over certain area (0.01 degree resol) and maybe affected by surrounding vegetations. It should also be noted that the vertical resolution of PAVD profile is much coarser.

Thank you @angehung5 for doing the review and test!

First, to clarify, I am only using the height of PAVD maximum from GEDI, and not the actual entire PAVD profile. So, the GEDI PAVD is used to only set the height of maximum foliage density and uses the vegtype set standard deviations above and below the maximum based on Massman, so it really is a hybrid approach right now. With the major issues of the gridded PAVD observations (coarse vertical res, inconsistencies with FCH we have right now from NAU, with no response back from them whatsoever, this is the best we have right now. Either way, using Massman, or hybrid PAVD/Massman approach has uncertainty in where the peak foliage density is set, but seems Massman approach is better right now.

Thus, can you also plot the foliage profiles for the different methods for each site to see how they vary? I wonder how you are setting the PAVD for these plots.

[angehung5]

The shapes at Chestnut and GOA look ok, although the height of maximum density is lower than Massman. Willow Creek and Lower Teakettle have the same issue that the maximum height of PAVD is comparable to the canopy height but PAVD values peak near the surface and are relatively low near the canopy top.

PAVD at Lower Teakettle for example, FH=32m

lat,lon,lev01,pavd01,lev02,pavd02,lev03,pavd03,lev04,pavd04,lev05,pavd05,lev06,pavd06,lev07,pavd07,lev08,pavd08,lev09,pavd09,lev10,pavd10,lev11,pavd11,lev12,pavd12,lev13,pavd13,lev14,pavd14
37.00,240.99,2.50,0.0682,7.50,0.0632,12.50,0.0449,17.50,0.0274,22.50,0.0210,27.50,0.0166,32.50,0.0120,37.50,0.0076,42.50,0.0043,47.50,0.0020,52.50,0.0007,57.50,0.0002,62.50,0.0000,67.50,0.0000

Agreed gridding should be the main reason here. Since the maximum heights of PAVD are reasonable (consistent with canopy height), satellite does see the forests. However, the surrounding shorter vegetations introduce a much lower height of maximum PAVD, which we can see at Lower Teakettle.