Build domain_cfg file

[jpolton]

Generate static files that define the domain. Work flow documentation is in wiki and SCRIPTS folder

[jpolton]

Try trimming the English Channel:

cp gebco_in.nc fixed_bathy.nc

ipython

import netCDF4
import numpy as np
dset = netCDF4.Dataset('gebco_in.nc','r')
dout = netCDF4.Dataset('fixed_bathy.nc','a')

dout.variables['elevation'][0:99,:] = 0
dout.variables['elevation'][0:200,300::] = 0

dset.close()
dout.close()

If happy: mv fixed_bathy.nc gebco_in.nc

Plot the region

import matplotlib.pyplot as plt
din = netCDF4.Dataset('gebco_in.nc','r')
ssh = din.variables['elevation']
lat = din.variables['lat']
lon = din.variables['lon']
X,Y = np.meshgrid(lat,lon)
plt.pcolormesh(Y.T,X.T,ssh);plt.colorbar()
plt.ylim([50.5, 51.8])
plt.xlabel('longitude')
plt.ylabel('latitude')
plt.title('Severn Estuary')
plt.show()

[jpolton]

@mpayopayo I think this is done. Can you give it a try?

The following process is followed to build and get started with this configuration

git clone https://github.com/JMMP-Group/SEVERN-SWOT.git

Then follow descritptions in: https://github.com/JMMP-Group/SEVERN-SWOT/wiki

Specifically: https://github.com/JMMP-Group/SEVERN-SWOT/wiki/2.-Build-domain-configuration-file

Put comments or issues here. (This is to be done after doing the build NEMO and XIOS process)

[mpayopayo]

I don't have access to /projectsa/NEMO/nibrun/ARCHER_DATA/INPUTS/AMM15/coordinates.nc, so asked Nico for access.

[jpolton]

I don't have access to /projectsa/NEMO/nibrun/ARCHER_DATA/INPUTS/AMM15/coordinates.nc, so asked Nico for access.

Well he is long gone. I've copied the file to a temporary place:

scp /projectsa/NEMO/nibrun/ARCHER_DATA/INPUTS/AMM15/coordinates.nc /scratch/jelt/coordinates_tmp.nc

This is not a permanent location, just something to keep you going for now.

[mpayopayo]

got it. thanks

[mpayopayo]

@jpolton doubt on the parameters of configuration in $DOMAIN/s-sig_DOMAINcfg_namelist_cfg

Do jpiglo and jpjglo come from the parent model and then I should pick them from $DOMAIN/namelist.input? or since it is not dynamic nesting it just considers it as global model itself?

&namcfg ! parameters of the configuration !----------------------------------------------------------------------- ! ln_e3_dep = .true. ! =T : e3=dk[depth] in discret sens. ! ! ===>>> will become the only possibility in v4.0 ! ! =F : e3 analytical derivative of depth function ! ! only there for backward compatibility test with v3.6 ! ! cp_cfg = "Severn" ! name of the configuration jp_cfg = 500 ! resolution of the configuration jpidta = 342 ! 1st lateral dimension ( >= jpi ) jpjdta = 273 ! 2nd " " ( >= jpj ) jpkdta = 31 ! number of levels ( >= jpk ) jpiglo = 342 ! 1st dimension of global domain --> i =jpidta jpjglo = 273 ! 2nd - - --> j =jpjdta jpizoom = 1 ! left bottom (i,j) indices of the zoom jpjzoom = 1 ! in data domain indices jperio = 0 ! lateral cond. type (between 0 and 6)

Also, I hadn't realised that make_tools.sh had failed to build DOMAINcfg. I've tried again to build it, but no success. I get "gmake: *** No rule to make target 'closea.o', needed by 'domzgr.o'. Stop." and then it fails. Any idea about how to sort it out?

[jpolton]

1 of 2.

jpiglo = 342 ! 1st dimension of global domain --> i =jpidta jpjglo = 273 ! 2nd - - --> j =jpjdta

I don't entirely know about jpiglo and jpjglo. I've too assumed that they are not used and just copied them to match jpidta and jpjdta as a relic from the days when I knew even less.

[jpolton]

2 of 2:

At the top of the make_tools.sh there is code to download the tools

 cd $NEMO
for ext_name in tools
    do
    ext=`svn propget svn:externals | grep $ext_name | cut -c2-`
    echo $ext
    svn co http://forge.ipsl.jussieu.fr/nemo/svn/$ext
done

This seems to have not worked properly since you don't have all the download files in DOMAINcfg/src E.g.

ls /work/n01/n01/jelt/SEVERN-SWOT/BUILD_EXE/NEMO/4.0.6/tools/DOMAINcfg/src

Perhaps $NEMO was not defined when you ran make_tools.sh? Not sure, but the above svn snippet should have populated the DOMAINcfg/src folder, which is the problem and I can't replicate the issue....

Maybe you have an old make_tools.sh? To check do:

cd SEVERN-SWOT
git status # to check if you are ahead or behind the master branch

If you have edits to make (more comments / fixes etc), create a new branch with them and do a pull request.

EDIT Just checked - you do not appear to be behind the master branch. Hmm

[mpayopayo]

I was checking at the moment. I had defined $NEMO when first running tools. Yesterday I tried again when make_domain_cfg.sh was failing. I also had defined all the paths. The other tools built fine, I think it is only the DOMAINcfg the one giving problems EDIT could it be an option just to copy the files I'm missing from your folder? I know it is a dirty fix...

[jpolton]

@mpayopayo Can you confirm that the $NEMO/tools/DOMAINcfg/scr/*F90 don't download when you do the following:

cd $NEMO
for ext_name in tools
    do
    ext=`svn propget svn:externals | grep $ext_name | cut -c2-`
    echo $ext
    svn co http://forge.ipsl.jussieu.fr/nemo/svn/$ext
done

[mpayopayo]

@jpolton That's correct. No download of DOMAINcfg/src/F90 when I execute that bit of script. I've checked for the other tools built with make_tools.sh and I think I have the same F90 that you have and they built fine.

[jpolton]

@jpolton That's correct. No download of DOMAINcfg/src/F90 when I execute that bit of script. I've checked for the other tools built with make_tools.sh and I think I have the same F90 that you have and they built fine.

Ooo weird. Ok. Copy my src files and try and continue. I'll create a ticket of this problem so it isn't forgotten

[mpayopayo]

@jpolton going back to make_tools.sh line by line only to rebuild DOMAINcfg

and I get:

The problem is the file $TDIR/DOMAINcfg/src/domzgr.f90. I copied yours but then it is overwritten in the make_tools.sh with $DOMAIN/domzgr.f90.melange:

your L2110 !hbatz(:,:) = zenv(:,:) my L2110 (once overwritten as per make_tools.sh) hbatz(:,:) = zenv(:,:)

Once I copy again your file it build the DOMAINcfg without problem

[mpayopayo]

@jpolton I'm executing line by line make_domain_cfg.sh and it doesn't produce any tiles L37 (sbatch job_create.slurm) so there's nothing to rebuild. On the slurm* files I'm getting the following:
I tried both with L30 commented and also uncommented (since now it no longer appears on make_tools.sh). Both cases were with ln_e3_dep =.true. on the namelist_cfg since v4.0, but trying now with .false. I also get the same.

I've also tried changing the jpiglo and jpjglo in the namelist so that they match the values in the parent grid. Could it have to do with not having the "parent" grid coordinate file in the folder? subroutine ctl_opn and from the nesting readme from option 4 here: NEMO set up -nesting readme

[jpolton]

@jpolton I'm executing line by line make_domain_cfg.sh and it doesn't produce any tiles L37 (sbatch job_create.slurm) so there's nothing to rebuild. On the slurm* files I'm getting the following: I tried both with L30 commented and also uncommented (since now it no longer appears on make_tools.sh). Both cases were with ln_e3_dep =.true. on the namelist_cfg since v4.0, but trying now with .false. I also get the same.

I've also tried changing the jpiglo and jpjglo in the namelist so that they match the values in the parent grid. Could it have to do with not having the "parent" grid coordinate file in the folder? subroutine ctl_opn and from the nesting readme from option 4 here: NEMO set up -nesting readme

If this is the first time you've submitted a job to ARCHER2 perhaps it is to do with your permissions to do so? You said that Andrew Coward had you set up on a CLASS account. I recall that this was setup for me to run under the ACCORD accord. See line 7 of https://github.com/JMMP-Group/SEVERN-SWOT/blob/master/BUILD_CFG/DOMAIN/job_create_template.slurm Perhaps you can change that to CLASS?

[mpayopayo]

@jpolton I ran on ARCHER on the ACCORD account on the CME project. Now I have a CLASS account. I've changed that line but I'm getting the same message on the slurm-out. While running it seems to have 2 tasks, not sure if that should be the case: The string of text that I get appears in lib_mpp.f90 so maybe it's something to do with the parallel processing but the ctl_opn subroutine is about files being available. I do have coordinates.nc and bathy_meter.nc on the right folder so I'm a bit confused.

[jpolton]

@jpolton I ran on ARCHER on the ACCORD account on the CME project. Now I have a CLASS account. I've changed that line but I'm getting the same message on the slurm-out. While running it seems to have 2 tasks, not sure if that should be the case: The string of text that I get appears in lib_mpp.f90 so maybe it's something to do with the parallel processing but the ctl_opn subroutine is about files being available. I do have coordinates.nc and bathy_meter.nc on the right folder so I'm a bit confused.

That looks promising. From my experience that seems to be what it does when the job is waiting in the queue: E.g. see also squeue -u $USER

[mpayopayo]

@jpolton yes, that's waiting. But then it stops, there are no domain tiles produced and I get the STOP ctl_opn bad opening message on the slurm*. Do I get 2 jobid because the batch script uses jhet? previously I have not used jhet and I only get one jobid

"STOP ctl_opn bad opening" on lib_mpp.f90 indicates that there is a file missing, but I have not figured out what file it is since I do have coordinates.nc and bathy_meter.nc Any clue?

[jpolton]

Good news: I can run your /work/n01/n01/marpay/SEVERN-SWOT/BUILD_EXE/NEMO/4.0.6/tools/DOMAINcfg/BLD/bin/make_domain_cfg.exe and produce output domain_cfg_0006.nc etc.

Hmm. Having a look at the differences:

DOMAINcfg> diff . /work/n01/n01/marpay/SEVERN-SWOT/BUILD_EXE/NEMO/4.0.6/tools/DOMAINcfg

[mpayopayo]

hum I don't understand, there are differences on the binary of the .exe (but you manage to run mine) and on the namelist, so I'm changing the namelist.

[jpolton]

Also just ran with your bathy_meter.nc file... and your coordinates.nc file is the same...

[jpolton]

Also spotted differences with *xml files (that you didn't have) but it appears that I didn't need them either. The only thing I can think of at the moment is around the n01-ACCORD / n01-CLASS account in the job script.

[mpayopayo]

did you also use my namelist? both with the namelist with my values and your values I also get the "STOP ctl_open bad opening" messages on the slurm out and no tiles. I had already restore the modules

[jpolton]

OK. Will test that now

[jpolton]

No need I think I have it:

ls /work/n01/n01/marpay/SEVERN-SWOT/BUILD_EXE/NEMO/4.0.6/tools/DOMAINcfg/namelist_ref Does not exist. Copy mine

[mpayopayo]

it was that! I have the tiles now. I'll update that on the wiki too

[jpolton]

it was that! I have the tiles now. I'll update that on the wiki too

Hmm. There is some fishy though as you should have had the missing *xml and namelist_ref files when the downloaded the tools/DOMAINcfg files in make_tools.sh. I just tested these svn checkouts in a fresh directory and got the files....

[mpayopayo]

I tried again for ext_name in tools do ext=svn propget svn:externals | grep $ext_name | cut -c2- echo $ext svn co http://forge.ipsl.jussieu.fr/nemo/svn/$ext done

but neither the *.xml nor the namelist_ref download. I don't understand

[jpolton]

You have to do this in a directory where an SVN checkout has already been done.

[jpolton]

mkdir TMP
cd TMP
export NEMO=NEMO
svn co http://forge.ipsl.jussieu.fr/nemo/svn/NEMO/releases/r4.0/r4.0.6 --depth empty $NEMO

  cd $NEMO
  for ext_name in tools
    do
    ext=`svn propget svn:externals | grep $ext_name | cut -c2-`
    echo $ext
    svn co http://forge.ipsl.jussieu.fr/nemo/svn/$ext
  done

[jpolton]

@mpayopayo Did we solve these issues? Can you close this ticket if you are happy with it?