how do I load site data to the Pecan server

[fabeit]

I am trying to add met data to the pecan server via the BETY website following these instructions https://github.com/PecanProject/pecan/wiki/How-to-insert-new-Input-data.

However I am not able to upload my own .nc files because it seems I am restricted to whatever it is in one of the listed machines.

How do I upload some of these .nc files I have taken from http://hydrology.princeton.edu/data/pgf/1.0deg/3hourly/ ? Also should I crop these global files to adjust the region where I want to run ED?

Thanks

[mdietze]

Ok, so this is mostly a question about how you move files from your computer onto your VM. Any methods that you would use to move files to a remote server should work (sftp, rsync, etc.). Also, it is possible to set up a shared folder between a VM and the host computer that is running it: https://www.virtualbox.org/manual/ch04.html#sharedfolders. For something as large as a global met product the latter is a much better options since then you won't need two copies.

We don't have met converters on hand for the Princeton product, so right now you'll probably want to do any clipping and converting by hand (though I hope to have global met working in the next few weeks). PEcAn uses netCDF files in CF format as it's internal standard (https://github.com/PecanProject/pecan/wiki/Adding-an-Input-Converter#met-data) so probably the simplest path is to nco and cdo tools to update the variable names and units to CF and then use PEcAn's met2model.ED2 to convert the files into the ED2 format. Slightly more ambitious (but very helpful to the overall PEcAn project) would be to write a met2CF function in PEcAn's data.atmosphere module to do the conversion from the native format to CF (see, for example, the met conversion functions met2CF.Ameriflux, met2CF.NARR, met2CF.PalEON, and met2CF.csv). Indeed, met2CF.PalEON and met2CF.NARR are very similar to what you need for the Princeton file and you should be able to modify one of those to handle this job. If you do take this route, please please send us the code (preferably by doing a Github pull request so you get credit) since the overarching design philosophy for PEcAn is that no one else should have to write the same processing code twice!

[fabeit]

Michael, I do not mind riding in sharing the code but I am a bit overwhelmed with all these procedures. Are you sure their Princeton data is not CF compliant?

Also, my goal was to run the ED model on the Pecan server because it will be way too slow on my laptop. That is why I was asking how to upload the data. So the web interface only allows to run models in places where site data is already on the server, is this correct? I am just a bit confused because I am able to create a site where ever I want but I cannot add the necessary data to run the model.

[mdietze]

The readme in the link you sent earlier says it contains the following variables: Precipitation (prcp) Air temperature at 2m above ground (tas) Downward longwave at surface (dlwrf) Downward shortwave at surface (dswrf) Surface pressure (pres) Specific humidity (shum) Windspeed (wind) Which are not CF but are very similar to the variable names used in our NARR scripts

Yes, PEcAn only allows you to do a run if site data is available.

If you want to run ED2 on a server, we should have that conversation offline (my email is dietze at bu dot edu). There are multiple servers out there running PEcAn (3 at BU, 1 at NCSA, 1 at U Illinois, 1 at Brookhaven) but the only one that is 'public' is the demo server at NCSA (the one linked off pecanproject.org), but that server is not for production research (results are not saved, files cannot be uploaded).

[fabeit]

Sorry just a bit late reply. Ok I have created a shared folder from the VM to the host and changed all the paths in the ED2IN file. I first tried to modify an ED2IN file I got from ED 2.1 but I get this error message "fortran runtime error: Bad data for namelist object nl%nstraty but I have the default value 1, 4. Then I got the one from testrun.ED.r46, I test it without changing any values and it worked. So I modified paths and the location of the run to use my met data. First it told me that it was missing "ED MET DRIVER HEADER", so I copied the one inside sites/ebifarm but I realized that was set up for a specific location so I get the error "polygon lies outside the met driver domain' Polygon long: 1.6500E+1 Lat: 2.50 but the site I entered is 16.50 and 2.5. And the subroutine is init_met_drivers. I have been told that ED can read directly .nc files from people who have used ED. Does the standalone ED inside Pecan also need the conversion of nc to CF standard? Or do I have to just manually update the content of the "ED MET DRIVER HEADER" file? I am ready to rip my hair out, been trying to run this simple simulation for ED in one single point but I can't! Please help!!!

[mdietze]

There's a lot of questions in there so I'll try and break them down.

1) There's a few things in there that are about ED2, not PEcAn. For example, if you are modifying ED2IN files by hand to set file paths to input files, you are no longer using PEcAn and you might get additional feedback by posting issues to https://github.com/EDmodel/ED2/

2) The version of the ED2IN file you are using HAS TO MATCH the version of ED2 that you are running (2.1 is not sufficient since ~7 years of development has occurred under that release number). The bad namelist error was due to a mismatch. Unhelpfully, when you get that error it is most often the variable before or after the one that threw the error which is the one that's actually wrong.

3) If you've moved ED meteorology files from elsewhere then you have to set the path to the .h5 files within the ED_MET_DRIVER_HEADER file. I personally think this is silly, since there's no info in the ED_MET_DRIVER_HEADER that we couldn't put in the HDF5 files themselves, but that's how it works right now.

4) The geographic info in the ED_MET_DRIVER_HEADER and in the file names themselves have to match

5) If you are running ED2 outside of PEcAn then the met files have to have an .h5 extension. Due to a convergence in netCDF4 and HDF5 format it is in theory possible to just rename .nc files to .h5, however internally in those files the met has to have the CORRECT VARIABLE NAMES AND UNITS. Alternatively, if the meteorology is in netCDF CF (PEcAn's standard) you can use met2model.ED2 to convert the met into ED2 format. This code will also write the ED_MET_DRIVER_HEADER

[ashiklom]

Closing because this is an old discussion, and no further comments have been posted.