Run tide + met

[jpolton]

Experiment set up with example met forcing (ERA5: pressure and winds) + tides

[jpolton]

Created a new branch 'feature/met'

Add wiki notes on how to prep 2013/2014 ERA5 forcing data

Added new experiment EXP_barotropicTide_ERA5 that runs with tides and met. See Run tides with met forcing wiki page.

[jpolton]

Estuary at i=113, j=114 looks too narrow to let the water easily flow in and out. This leads to a hydrological jump at time step =261, (dt=20s) starting at 1st Jan 2013.

Water in the estuary has a sea level of -5m. At the mouth it is -2m. So there is an unphysical steep SSH gradient at the mouth. This seems to result in a blow up.

FIX: modify the bathymetry to either open the channel, or close the channel in this region.

I will leave this as an exercise for @mpayopayo

[mpayopayo]

@jpolton I'm getting some error (missing parentheses in lines containing print in OFFICIAL_Generate_NEMO_Forcing_NEWERA.py) when trying to add atmospheric forcing.

    print tout[0], tout[-1], tout.shape, out.shape, LON.shape
             ^
SyntaxError: Missing parentheses in call to 'print'. Did you mean print(tout[0], tout[-1], tout.shape, out.shape, LON.shape)?

I'm using the pynemo3 environment because I don't have any other environment-I'm new to python. Can the problem come from there? what packages do you have in your nrct_env environment so that I can have the same?

[jpolton]

@jpolton I'm getting some error (missing brackets in lines containing print in OFFICIAL_Generate_NEMO_Forcing_NEWERA.py) when trying to add atmospheric forcing. I'm using the pynemo3 environment because I don't have any other environment-I'm new to python. Can the problem come from there? what packages do you have in your nrct_env environment so that I can have the same?

Not sure what happened to my file edits there. The issue with print statements is a version 2 to version 3 thing. python 3 requires brackets: print("This"). Code SCRIPTS/OFFICIAL_Generate_NEMO_Forcing_NEWERA.py modified. The conda environment used is vanilla. You should NOT need anything fancy. I just grabbed an old one I had lying around

[mpayopayo]

@jpolton I pulled your branch, so shall i modify the script in my code and then it will be incorporated when pushing changes? still not quite sure of how git works

[jpolton]

@jpolton I pulled your branch, so shall i modify the script in my code and then it will be incorporated when pushing changes? still not quite sure of how git works

I don't quite understand. Yes pull the changes from the repo. That should fix the problem you identified. If your changes make it even better than commit them and push it back.

[mpayopayo]

@jpolton I'm getting an error saying the following at the 1st step:

    import netcdftime
ModuleNotFoundError: No module named 'netcdftime'

I've tried cloning pynemo3 under a different name to then add that module but I'm getting unexpected errors. Could you please save a txt of your nrct_env environment so that i can create it from that txt file?

[jpolton]

conda activate nrct_env
conda list -e 
# This file may be used to create an environment using:
# $ conda create --name <env> --file <this file>
# platform: linux-64
basemap=1.0.7=np19py27_0
ca-certificates=2019.6.16=hecc5488_0
cairo=1.12.18=6
certifi=2016.2.28=py27_0
curl=7.45.0=0
cython=0.26=py27_0
fontconfig=2.11.1=6
freetype=2.5.5=1
geos=3.3.3=0
hdf5=1.8.15.1=3
libgfortran=1.0=0
libiconv=1.14=0
libnetcdf=4.3.3.1=3
libpng=1.6.17=0
libxml2=2.9.4=0
libxslt=1.1.29=0
lxml=3.8.0=py27_0
matplotlib=1.4.3=np19py27_2
ncurses=5.9=10
netcdf4=1.1.9=np19py27_0
numpy=1.9.2=py27_0
openssl=1.0.2l=0
pip=9.0.1=py27_1
pixman=0.32.6=0
py2cairo=1.10.0=py27_2
pyjnius=1.4=py27_0
pyparsing=2.0.3=py27_0
pyqt=4.11.4=py27_4
python=2.7.13=0
python-dateutil=2.6.1=py27_0
pytz=2017.2=py27_0
qt=4.8.7=3
readline=6.2=2
requests=2.14.2=py27_0
scipy=0.16.0=np19py27_0
seawater=3.3.4=py_1
setuptools=36.4.0=py27_1
sip=4.18=py27_0
six=1.10.0=py27_0
sqlite=3.13.0=0
thredds_crawler=1.0.0=py27_0
tk=8.5.18=0
wheel=0.29.0=py27_0
zlib=1.2.11=0

I've just tested this python SCRIPTS/OFF... and it works. However looking at this module list it is python2 not python3! So at some near future point it should be further updated to python3. However, in the mean time, hopefully you can build the nrct_env.

[mpayopayo]

@jpolton I copied the output of that into a ncrt_env.txt and tried creating the environment with conda create --name ncrt_env --file ncrt_env.txt. It fails and I get the following error:

$conda create --name nrct_env --file nrct_env.txt

PackagesNotFoundError: The following packages are not available from current channels:

  - ca-certificates==2019.6.16=hecc5488_0
  - pyjnius==1.4=py27_0
  - thredds_crawler==1.0.0=py27_0
  - seawater==3.3.4=py_1

Current channels:

  - https://repo.continuum.io/pkgs/main/linux-64
  - https://repo.continuum.io/pkgs/main/noarch
  - https://repo.continuum.io/pkgs/free/linux-64
  - https://repo.continuum.io/pkgs/free/noarch
  - https://repo.continuum.io/pkgs/r/linux-64
  - https://repo.continuum.io/pkgs/r/noarch
  - https://repo.continuum.io/pkgs/pro/linux-64
  - https://repo.continuum.io/pkgs/pro/noarch

I tried also specifying the channel

$ conda create -c conda-forge --name nrct_env --file nrct_env.txt

PackagesNotFoundError: The following packages are not available from current channels:

  - pyjnius==1.4=py27_0
  - thredds_crawler==1.0.0=py27_0

Current channels:

  - https://conda.anaconda.org/conda-forge/linux-64
  - https://conda.anaconda.org/conda-forge/noarch
  - https://repo.continuum.io/pkgs/main/linux-64
  - https://repo.continuum.io/pkgs/main/noarch
  - https://repo.continuum.io/pkgs/free/linux-64
  - https://repo.continuum.io/pkgs/free/noarch
  - https://repo.continuum.io/pkgs/r/linux-64
  - https://repo.continuum.io/pkgs/r/noarch
  - https://repo.continuum.io/pkgs/pro/linux-64
  - https://repo.continuum.io/pkgs/pro/noarch

Any idea of how to sort this? I can maybe do manually the ones that exist in the channel, but how do I get the others?

Another alternative is if you export is as .yml?

It seems that it gives trouble generating environments from the .txt generated with conda list -e https://github.com/conda/conda/issues/10636

This is the information about the environment in case it helps:

$conda info

       user config file : /login/marpay/.condarc
 populated config files : /login/marpay/.condarc
          conda version : 4.4.10
    conda-build version : 3.4.1
         python version : 3.6.4.final.0
       base environment : /packages/lmodmodules/apps/anaconda/3-5.1.0  (read only)
           channel URLs : https://repo.continuum.io/pkgs/main/linux-64
                          https://repo.continuum.io/pkgs/main/noarch
                          https://repo.continuum.io/pkgs/free/linux-64
                          https://repo.continuum.io/pkgs/free/noarch
                          https://repo.continuum.io/pkgs/r/linux-64
                          https://repo.continuum.io/pkgs/r/noarch
                          https://repo.continuum.io/pkgs/pro/linux-64
                          https://repo.continuum.io/pkgs/pro/noarch
          package cache : /work/marpay/marpay-conda/pkgs
       envs directories : /work/marpay/envs
                          /login/marpay/.conda/envs
                          /packages/lmodmodules/apps/anaconda/3-5.1.0/envs
               platform : linux-64
             user-agent : conda/4.4.10 requests/2.18.4 CPython/3.6.4 Linux/3.10.0-1062.12.1.el7.x86_64 centos/7 glibc/2.17
                UID:GID : 18725:18063
             netrc file : None
           offline mode : False

[mpayopayo]

@jpolton when manually I have the same issue. pyjnius==1.4=py27_0 and thredds_crawler==1.0.0=py27_0 are not available in the current channels. what channels have those packages?

[mpayopayo]

@jpolton I tried also doing pip install pyjnius and said succesfully installed pyjnius then did conda install -c conda-forge thredds_crawler and it also said succesfully installed.

However when doing conda list -e the package pyjnius does not appear while thredds_crawler does appear. I've tried also doing both import pyjnius under a python terminal and import jnius under a python terminal and both give errors: >>> import pyjnius

  File "<stdin>", line 1, in <module>
ImportError: No module named pyjnius

>>> import jnius

  File "<stdin>", line 1, in <module>
  File "/work/marpay/envs/nrct_envt/lib/python2.7/site-packages/jnius/__init__.py", line 43, in <module>
    from .reflect import *  # noqa
  File "/work/marpay/envs/nrct_envt/lib/python2.7/site-packages/jnius/reflect.py", line 20, in <module>
    class Class(with_metaclass(MetaJavaClass, JavaClass)):
  File "/work/marpay/envs/nrct_envt/lib/python2.7/site-packages/six.py", line 808, in __new__
    return meta(name, bases, d)
  File "jnius/jnius_export_class.pxi", line 120, in jnius.MetaJavaClass.__new__
TypeError: Error when calling the metaclass bases
    type() argument 1 must be string, not unicode

[mpayopayo]

@jpolton I did pip install pyjnius and conda install -c conda-forge thredds_crawler and it doesn't give any error, in both cases it says succesfully installed. but when doing conda list -e the package pyjnius does not appear. When I try to import pyjnius under a python terminal it also says no module named pyjnius. And when trying to import jnius I get

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/work/marpay/envs/nrct_envt/lib/python2.7/site-packages/jnius/__init__.py", line 43, in <module>
    from .reflect import *  # noqa
  File "/work/marpay/envs/nrct_envt/lib/python2.7/site-packages/jnius/reflect.py", line 20, in <module>
    class Class(with_metaclass(MetaJavaClass, JavaClass)):
  File "/work/marpay/envs/nrct_envt/lib/python2.7/site-packages/six.py", line 808, in __new__
    return meta(name, bases, d)
  File "jnius/jnius_export_class.pxi", line 120, in jnius.MetaJavaClass.__new__
TypeError: Error when calling the metaclass bases
    type() argument 1 must be string, not unicode

it doesn't seem to need those modules though. The alternative to import pyjnius would be to do import autoclass

The python script SCRIPTS/OFFICIAL_Generate_NEMO_Forcing_NEWERA.py runs fine with env nrct_env (python2) but you need to change to python3 to do python create_LSM.py (pynemo3 works fine).

I also made slurm script for the last step and will modify wiki

[mpayopayo]

ONce the bathymetry is modified and the Taw estuary is closed, it ran for a month (January2013) without blowing up. The wiki is modified accordingly

[micdom]

@mpayopayo @jpolton the wiki about the creation of met-forcing (Download ERA5 forcing data 1. Extract ERA5 data) is a bit unclear, with the mention to python2, python3 and python environment (source activate nrct_env). I've managed to extract the ERA5 files and I've updated the wiki with the instructions for a very simple python environment to use (with python3 and I deleted the notes about python2 and 3). I'll need to update just a couple of lines in OFFICIAL_Generate_NEMO_Forcing_NEWERA.py and push it.

[micdom]

@mpayopayo @jpolton the generation of the met forcing can work also with the PyNNEMO environment, I added that in the wiki. I hope I didn't mess anything in OFFICIAL_Generate_NEMO_Forcing_NEWERA.py, it works for me...