LSEG_NLDAS_MAP File Format

[rburghol]

File is text, with 1 format parameter per line. Note, in the c++ code, the signifier ">>" loads the next line of the file into the given variable, for example, the line LandSegNLDASFile >> NumLandSeg; reads the next line from the file LandSegNLDASFile, and puts the value into the integer variable NumLandSeg. By looking through the code for each occurence of ">>", we can map our the contents of the config file.

• Example files from Gopal
  • P6_LRSegs_050615.txt
  • P6_LRSegs_050615_Dissolve_FIPS_NHL_Intersect_NLDAS2-X_fmt.txt
• .txt config file containing all VA basin segments and their corresponding grids

[NumLandSeg = # of land segments in file]
[LandSeg = 1st Land Segment]
[NumNLDAS=# of NLDAS cells for this land segment)\]
[NLDAS cell 1 - formatted x(column)y(row)]
[NLDAS cell 2]
...
[NLDAS cell NumNLDAS]
... the above pattern (minus the first line NumLandSeg ) repeats for NumLandSeg times

[rburghol]

Hey @alexwlowe @katealbi11 @kylewlowe @snhandelsman -- I just looked through the code and started mapping out the basics of the file to the best of my knowledge. If someone is game to look into this while we wait to hear from Gopal, I am available for email tech support. That is not to say we don't have plenty to do documenting, downloading the data, figuring out the coverages outside of Virginia, etc.

[alexwlowe]

That's interesting @rburghol!

Do you know what format the LandSeg parameter is named? Does each land seg have unique names, codes, numbers, etc? Also, are the land segments the same as the cbp phase 6 land segs?

[rburghol]

No clue as to the format :), but I put in an email to Gopal. As for the general concept, the land segments should absolutely correspond to phase 6 land segments, just that, we don't year quiteknow the answer there. What I can see, I think, is that there is no weighting information, as the code appears to simply consider each overlapping cell equally. If you were bored and wanted to try your hand at creating a config file go ahead! (you'd have to figure out how to reference the NLDAS2 grid cell based on what ou can infer from the code), NLDAS2_ASCII_to_LSegs would be the command (plus the 11 command line arguments!)

[alexwlowe]

Take 1 at taking a stab at getting NLDAS2_ASCII_to_LSegs to run

I created the config file mimicking the format above and called it LandSegNLDASFile: Then I ran the function using the 11 parameters and got the following error:

Based on the error, I'm assuming I messed up somewhere before LandSegNLDASFile was even read, I will do some looking in the function to try to trouble shoot.

Update: I believe the error occurred at line 101. It looks like the DPT file is created on line 91 using LandSeg, which I believe ties into the LandSegNLDASFile, so I think it actually is that file that is breaking it.

[rburghol]

It is saying it can't open ./1991010100-1991010123/N51029.DPT - Could the problem be that the directory does not exist?

In other words, if you created a directory named "1991010100-1991010123" maybe it would work?

[alexwlowe]

Thanks @rburghol! That helped a lot, past that error!

Take 2 with actual progress Here is the line that I ran and error I received: It looks like I probably need to change my ASCII directory, and maybe the way my grid is formatted in the LandSegNLDASFile

[alexwlowe]

Take 3

The function created 6 files titled N51029.(three letter code) which is I think formatted They all have data similar to the image below: It looks like it is formatted year, month, day, hour, value

Update: If you run 1 land segment with multiple grids, the function does not create more files, but the values within the files change (I'm assuming this is because it now takes into account data from multiple grids).

If you run 2 land segments and list both land segments, all the outputs in the files turn into zeros.

If you say 2 land segments but only list one of them (following config file), it only creates files for the land seg you specified, and the values are the same as if you were to say there was only 1 land segment.

[rburghol]

This looks super encouraging!! ~What was the syntax of the command that you ran?!?!(never mind, I see your commands in the docs! I want to try to replicate it before meeting with Gopal tomorrow.)~

By what I can see here, I think we can avert the error that you got when specifying multiple landsegs, just create a config file for each land seg, and be done with it. That is kid of a clunky workaround, but not the worst thing ever done in the history of data processing :).

[rburghol]

I tried this from your screen capture in the /backup/meteorology directory, and got close. But, the data in /backup/meteorology is not quite the same as that in Alex's home directory. But clearly you know how to make it work, so I think we're on the case!

For reference and future copy-pasteable docs:

mkdir 1991010100-1991010123
NLDAS2_ASCII_to_LSegs ./ ./ 1991 01 01 00 1991 01 01 23 LSEG_NLDAS_MAP

[alexwlowe]

Hey Rob,

Since there is currently only the raw NLDAS data in the meteorology directory, you are going to have to run the GRIB to ASCII function first

I believe this will work

mkdir 1991010100-1991010123
NLDAS2_GRIB_to_ASCII /backup/meteorology ./ 1991 01 01 00 1991 12 31 23 1 372 101
NLDAS2_ASCII_to_LSegs ./ ./ 1991 01 01 00 1991 01 01 23 LSEG_NLDAS_MAP

And the lines in LSEG_NLDAS_MAP should be

1
N51029
1
x372y101

[rburghol]

Thanks! Sorry to ask a question before I read the other doc fully!

[rburghol]

Alex - I have gone through this a bunch of times now, and we have some small tweak that needs to occur:

• The command works if I run it from your home directory. (Try 1)
• The command does not work if I run it from /backup/meteorology or anywhere else, like my home directory. (Try 2)

Try #1: Fails with Error Opening File: .//1991/x374y101zET.txt

rob@deq4:/home/alexwl$ cd /backup/meteorology/
rob@deq4:/backup/meteorology$ NLDAS2_ASCII_to_LSegs ./ /backup/meteorology/grid_met_csv 1991 01 01 00 1991 01 01 23 LSEG_NLDAS_MAP

Start: Tue Jan  1 00:00:00 1991

Stop : Tue Jan  1 23:00:00 1991

File Opened: /backup/meteorology/grid_met_csv/1991010100-1991010123/N51029.DPT
File Opened: /backup/meteorology/grid_met_csv/1991010100-1991010123/N51029.PET
File Opened: /backup/meteorology/grid_met_csv/1991010100-1991010123/N51029.PRC
File Opened: /backup/meteorology/grid_met_csv/1991010100-1991010123/N51029.RAD
File Opened: /backup/meteorology/grid_met_csv/1991010100-1991010123/N51029.TMP
File Opened: /backup/meteorology/grid_met_csv/1991010100-1991010123/N51029.WND
N51029 1 x374y101
N51029 Tue Jan  1 00:00:00 1991

Error Opening File: .//1991/x374y101zET.txt

Try 2: Succeeds from Alex's home directory.

rob@deq4:/backup/meteorology$ cd /home/alexwl/
rob@deq4:/home/alexwl$ NLDAS2_ASCII_to_LSegs ./ /backup/meteorology/grid_met_csv 1991 01 01 00 1991 01 01 23 LSEG_NLDAS_MAP

Start: Tue Jan  1 00:00:00 1991

Stop : Tue Jan  1 23:00:00 1991

File Opened: /backup/meteorology/grid_met_csv/1991010100-1991010123/N51029.DPT
File Opened: /backup/meteorology/grid_met_csv/1991010100-1991010123/N51029.PET
File Opened: /backup/meteorology/grid_met_csv/1991010100-1991010123/N51029.PRC
File Opened: /backup/meteorology/grid_met_csv/1991010100-1991010123/N51029.RAD
File Opened: /backup/meteorology/grid_met_csv/1991010100-1991010123/N51029.TMP
File Opened: /backup/meteorology/grid_met_csv/1991010100-1991010123/N51029.WND
N51029 1 x374y101
N51029 Tue Jan  1 00:00:00 1991

File Opened: .//1991/x374y101zET.txt
File Opened: .//1991/x374y101zPP.txt
File Opened: .//1991/x374y101zRH.txt
File Opened: .//1991/x374y101zRN.txt
File Opened: .//1991/x374y101zVP.txt
File Opened: .//1991/x374y101zTT.txt
File Opened: .//1991/x374y101zWD.txt
$ 1991 1 1 1
% 1991 1 1 1

So, I conclude that there is something in your directory, maybe in the "-R" folder? That appears to be the only place where the file `` exists.

Looping @jdkleiner in on this as I will be out next week -- I won't be totally unreachable, but I'll need his help.

[alexwlowe]

Hey Rob,

There are two reasons why this may not have worked anywhere but my home directory.

1. The timeseries data that is extracted form NLDAS2_GRIB_to_ASCII needed to run the NLDAS2_ASCII_to_LSegs is only in my directory at the moment. We have not batch run this function for all of the grids/timeframe we need yet because we were still downloading the raw NLDAS data.. Based on the error message you received, I think this is the reason.

2. The LSEG_NLDAS_MAP config file is only in my home directory. If you did not create a copy of the file wherever you were running the function, it probably wouldn't work. I did not test to see if it works to run the function in a different directory than the file, but I would assume it needs to be in the same directory unless you specify a path.

[rburghol]

1. I appear to have successfully ran NLDAS_GRIB_to_ASCII
2. I definitely had a copy of the LSEG_NLDAS_MAP file in the directory I was in

What I suspect is that something in "-R..." directory may br required?

On Mon, Jun 28, 2021 at 9:33 AM alexwlowe @.***> wrote:

Hey Rob,

There are two reasons why this may not have worked anywhere but my home directory.

1.

The timeseries data that is extracted form NLDAS2_GRIB_to_ASCII needed to run the NLDAS2_ASCII_to_LSegs is only in my directory at the moment. We have not batch run this function for all of the grids/timeframe we need yet because we were still downloading the raw NLDAS data.. Based on the error message you received, I think this is the reason. 2.

The LSEG_NLDAS_MAP config file is only in my home directory. If you did not create a copy of the file wherever you were running the function, it probably wouldn't work. I did not test to see if it works to run the function in a different directory than the file, but I would assume it needs to be in the same directory unless you specify a path.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/HARPgroup/cbp6/issues/247#issuecomment-869689071, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABC4AIVQQYPV2RC6UHTOROLTVB2ZPANCNFSM47FZIYLA .

--

Robert W. Burgholzer 'Making the simple complicated is commonplace; making the complicated simple, awesomely simple, that's creativity.' - Charles Mingus

[alexwlowe]

That's interesting Rob, I'll do some more testing and looking around my directories today to try and figure out what's up. I'll update this comment with what I find.

Update: I was able to run the function in the /backup/meteorology directory for Jan 1st, 2017 using the following commands.

mkdir /backup/meteorology/out/lseg_csv/2017010100-2017010123
NLDAS2_ASCII_to_LSegs ./out/grid_met_csv ./out/lseg_csv 2017 01 01 00 2017 01 01 23 LSEG_NLDAS_MAP

I think that you have to make sure that the time series directory (./out/grid_met_csv) has the timeseries data for the grid you specify in the config folder, and that the land seg directory (./out/lseg_csv) has a directory within it that outlines the timeframe that the function is running (2017010100-2017010123 in this case). I also think the LSEG_NLDAS_MAP config file needs to be in your working directory (/backup/meteorology), or you have to specify it's path in the function.

Hopefully this helps!

[alexwlowe]

Area weighting update:

It appears that the NLDAS_ASCII_to_LSegs determines the land segment values by finding the average value of all of the grids that are located within the land segment (specified by the LSEG_NLDAS_MAP config file). This means that it does not take into account how much actual area from each grid is in the land segment.

To test this hypothesis, I used Gopal's example file to run NLDAS_ASCII_to_LSegs for land segment N51550 (There are 11 total NLDAS grids that are in this land segment). The outputted temperature file (N51550.TMP) contained values that are the average of the 11 temperature values of the NLDAS grids. I assume this means that the function does not take into account how much land area of each grid is actually contained within the land segment, because not all 11 of the grids were fully within the land segment. Some of them are partially in N51550, partially in a different land segment.

Hopefully this all makes sense, I know sometimes my writing isn't completely coherent!

[rburghol]

I think we need to make sure someone else can run the script from /backup/meteorology to verify.

[alexwlowe]

We were all able to run NLDAS2_ASCII_to_LSegs from /backup/meteorology!

[alexwlowe]

Quick Note:

The function seems to break after the first 168 grid IDs from the config file have been run. The error message says that the grid .txt file cannot be opened, even if the .txt file for the grid is in the correct directory and has data. I do not know if this is just because the function can only crunch so many files worth of numbers at one time or if there is possibly a way around the issue. If not, then the function can just be run multiple times with separate config files until all land segments and corresponding grids are accounted for.

p.s. there is now a config file called LSEG_NLDAS_ALL located in the /backup/meteorology directory that contains a list of all of the land segments in VA's basins and their corresponding grid IDs! It is in the correct format to run the NLDAS2_ASCII_to_LSegs function.