wrfchem-leeds / WRFotron

Tools to automatise WRF-Chem runs with re-initialised meteorology
https://wrfchem-leeds.github.io/WRFotron/
GNU Affero General Public License v3.0
20 stars 6 forks source link

mpirun unable to find real.exe #14

Closed chmltf closed 4 years ago

chmltf commented 4 years ago

The runs I have been submitting do not create the wrfinput_d01 and wrfbdy_d01 files made from real.exe. The executable is copied to the run folder but it still cannot "find" it.

The pre.bash.exxxxxx reads: Name of source model =>NCEP GFS Analysis GRID 4

mpirun was unable to find the specified executable file, and therefore did not launch the job. This error was first reported for process rank 0; it may have occurred for other processes as well.

NOTE: A common cause for this error is misspelling a mpirun command line parameter option (remember that mpirun interprets the first unrecognized command line token as the executable).

Node: d8s0b4 Executable: real.exe

From what I can tell real.exe mostly requires a namelist.input file which is: &time_control ! time start_year = 2014, 2014, ! start year start_month = 01, 01, ! start month start_day = 02, 02, ! start day start_hour = 12, 12, ! start hour start_minute = 00, 00, ! start minute start_second = 00, 00, ! start second end_year = 2014, 2014, ! end year end_month = 01, 01, ! end month end_day = 05, 05, ! end day end_hour = 00, 00, ! end hour end_minute = 00, 00, ! end minute end_second = 00, 00, ! end second interval_seconds = 21600, ! interval between meteorological data files input_from_file = .true., .true., ! whether the nested run will have input files for domains other than domain 1 history_interval = 60, 60, ! history output file interval in minutes (integer only) frames_per_outfile = 1, 1, ! number of output times bulked into each history file restart = .false., ! whether this run is a restart restart_interval = 1440, ! restart output file interval in minutes io_form_history = 2 ! NetCDF io_form_restart = 2 ! NetCDF io_form_input = 2 ! NetCDF io_form_boundary = 2 ! NetCDF debug_level = 400 ! debugging level iofields_filename = "iofields" ,"iofields" ! an option to request particular variables to appear in output io_form_auxinput7 = 2, ! biomass burning-emissions input (wrffirechemi_d01) data format is WRF netCDF
auxinput7_inname = 'wrffirechemid' ! biomass burning filename auxinput7_interval_m = 60, ! biomass burning time interval, minutes frames_per_auxinput7 = 1, ! biomass burning files per time interval io_form_auxinput12 = 0, ! restart file type /

&domains ! domains - dimensions, nesting, parameters time_step = 180, ! timestep, meteorology, seconds time_step_fract_num = 0, ! numerator for fractional time step time_step_fract_den = 1, ! denominator for fractional time step max_dom = 1, ! number of domains e_we = 140, 154, ! westeast dimension e_sn = 140, 154, ! southnorth dimension e_vert = 33, 33, ! vertical dimension num_metgrid_levels = 27, ! number of vertical levels in WPS output num_metgrid_soil_levels = 4, ! number of soil levels or layers in WPS output dx = 30000, 2777.778, ! westeast resolution, metres dy = 30000, 2777.778, ! southnorth resolution, metres grid_id = 1, 2, ! grid ID parent_id = 1, 1, ! parent ID i_parent_start = 1, 70, ! x coordinate of the lower-left corner j_parent_start = 1, 60, ! y coordinate of the lower-left corner parent_grid_ratio = 1, 9, ! nesting ratio relative to the domain’s parent parent_time_step_ratio = 1, 9, ! parent-to-nest time step ratio feedback = 1, ! feedback from nest to its parent domain /

&physics ! physics mp_physics = 10, 10, ! microphysics scheme, 10 = Morrison 2-moment scheme
progn = 1, 1, ! prognostic number density, switch to use mix-activate scheme ra_lw_physics = 4, 4, ! longwave radiation scheme, 4 = RRTMG ra_sw_physics = 4, 4, ! shortwave radiation scheme, 4 = RRTMG radt = 30, 30, ! minutes between radiation physics calls, recommended 1 minute per km of dx (e.g. 10 for 10 km grid); use the same value for all nests sf_sfclay_physics = 5, 5, ! surface layer physics option, 5 = MYNN (Ravan's suggestion) sf_surface_physics = 2, 2, ! land surface physics option, 2 = NOAH bl_pbl_physics = 5, 5, ! boundary layer physics option, 5 = MYNN 2.5 bldt = 0, 0, ! minutes between boundary-layer physics calls, 0 = call every timestep cu_physics = 5, 0, ! cumulus parameterization option, 5 = Grell 3D, 0 = off cudt = 0, ! minutes between cumulus physics calls; should be set to 0 when using all cu_physics except Kain-Fritsch cugd_avedx = 1, ! number of grid boxes over which subsidence is spread, set to 3 for 4km run, 1 for 36km isfflx = 1, ! heat and moisture fluxes from the surface for real-data cases and when a PBL is used ifsnow = 1, ! snow-cover effects icloud = 1, ! cloud effect to the optical depth in radiation surface_input_source = 1, ! where landuse and soil category data come from, 1 = WPS num_soil_layers = 4, ! number of soil levels or layers in WPS output sf_urban_physics = 1, ! activate urban canopy model, 1 = single layer, 2 = multi layer mp_zero_out = 2, ! this keeps moisture variables above a threshold value ≥0 mp_zero_out_thresh = 1.e-8, ! critical value for moisture variable threshold, below which moisture arrays (except for Qv) are set to zero cu_rad_feedback = .true., .false., ! sub-grid cloud effect to the optical depth in radiation cu_diag = 1, 0, ! Additional time-averaged diagnostics from cu_physics slope_rad = 0, 1, ! use slope-dependent radiation topo_shading = 0, 1, ! applies neighboring-point shadow effects num_land_cat = 20, ! number of land categories in input data /

&fdda ! FDDA - options for grid, obs and spectral nudging grid_fdda = 1, 0, ! grid nudging gfdda_inname = "wrffdda_d", ! fdda filenames produced gfdda_end_h = 10000, 0, ! time (hr) to stop nudging after the start of the forecast gfdda_interval_m = 360, 0, ! time interval (in mins) between analysis times if_no_pbl_nudging_uv = 1, 0, ! nudging of u and v in the PBL, 0 = yes, 1 = no if_no_pbl_nudging_t = 1, 0, ! nudging of t in the PBL, 0 = yes, 1 = no if_no_pbl_nudging_q = 1, 0, ! nudging of q in the PBL, 0 = yes, 1 = no if_zfac_uv = 0, 0, ! nudge u and v in all layers, 0 = yes, 1 = limit to k_zfac_uv layers k_zfac_uv = 2, ! model level below which nudging is switched off for u and v if_zfac_t = 0, 0, ! nudge t in all layers, 0 = yes, 1 = limit to k_zfac_t layers k_zfac_t = 2, ! model level below which nudging is switched off for t if_zfac_q = 0, 0, ! nudge q in all layers, 0 = yes, 1 = limit to k_zfac_q layers k_zfac_q = 2, ! model level below which nudging is switched off for q guv = 0.0006, 0.0006, ! nudging coefficient for u and v (s-1) gt = 0.0006, 0.0006, ! nudging coefficient for t (s-1) gq = 0.0006, 0.0006, ! nudging coefficient for q (s-1) if_ramping = 0, ! 0 = nudging ends as a step function, 1 = ramping nudging down at the end of the period dtramp_min = 360, ! time (min) for ramping function io_form_gfdda = 2, ! 2 = NetCDF /

&dynamics ! dynamics - diffusion, damping options, advection options rk_ord = 3, ! time-integration scheme option, 3 = Runge-Kutta 3rd order w_damping = 1, ! vertical velocity damping flag, 1 = with damping diff_opt = 1, 1, ! turbulence and mixing option, 1 = evaluates 2nd order diffusion term on coordinate surfaces km_opt = 4, 4, ! eddy coefficient option, 4 = horizontal Smagorinsky first order closure diff_6th_opt = 0, 0, ! 6th-order numerical diffusion, 0 = none diff_6th_factor = 0.12, ! 6th-order numerical diffusion nondimensional rate base_temp = 290. ! base state temperature (K) damp_opt = 3, ! upper-level damping flag, 3 = Rayleigh damping zdamp = 5000., 5000., ! damping depth (m) from model top dampcoef = 0.2, 0.2, ! damping coefficient khdif = 0, 0, ! horizontal diffusion constant (m2/s) kvdif = 0, 0, ! vertical diffusion constant (m2/s) non_hydrostatic = .true., ! running the model in nonhydrostatic mode moist_adv_opt = 2, 2, ! advection options for moisture, 2 = monotonic chem_adv_opt = 2, 2, ! advection options for chemistry, 2 = monotonic scalar_adv_opt = 2, 2, ! advection options for scalars, 2 = monotonic tke_adv_opt = 2, 2, ! advection options for TKE, 2 = monotonic do_avgflx_em = 1, 1, ! outputs time-averaged masscoupled advective velocities, 1 = on /

&bdy_control ! Boundary condition control spec_bdy_width = 5, ! total number of rows for specified boundary value nudging spec_zone = 1, ! number of points in specified zone relax_zone = 4, ! number of points in relaxation zone specified = .true., ! specified boundary condition nested = .false., .true., ! nested boundary conditions /

&grib2 /

&namelist_quilt ! options for asynchronized I/O for MPI applications nio_tasks_per_group = 0, ! # of processors used for IO quilting per IO group nio_groups = 0 ! number of quilting groups /

&chem ! chemistry kemit = 1, ! number of vertical levels in the emissions input data file chem_opt = 202, 202, 202, ! chemistry option, 201 = MOZART-MOSAIC (4 bins + simplified SOA + no aqeuous chemistry), 202 = MOZART-MOSAIC (4 bins + VBS SOA + aqeuous chemistry). bioemdt = 3.0, 3.0, 3.0, ! timestep, biogenic, minutes photdt = 30., 10., 10., ! timestep, photolysis, minutes chemdt = 3.0, 3.0, 3.0, ! timestep, chemistry, minutes io_style_emissions = 2, ! anthropogenic emissions, files, two 12-h emissions data files used emiss_inpt_opt = 102, 102, 102, ! RADM2 emission speciation adapted after reading data file to follow the RADM2/SORGAM framework (including isoprene) emiss_opt = 10, 10, 10, ! anthropogenic emissions, setting, 10 = MOZART (MOZART + aerosols) emissions chem_in_opt = 1, 1, 1, ! initialize chemistry, 1 = uses previous simulation data phot_opt = 4, 4, 4, ! photolysis option, 1 = Full TUV, 3 = Madronich F-TUV, 4 = New full TUV scheme gas_drydep_opt = 1, 1, 1, ! dry deposition of gas species, 1 = on aer_drydep_opt = 1, 1, 1, ! dry deposition of aerosols, 1 = on bio_emiss_opt = 3, 3, 3, ! includes MEGAN biogenic emissions online based upon the weather, land use data gas_bc_opt = 1, 1, 1, ! gas boundary conditions, 1 = default gas_ic_opt = 1, 1, 1, ! gas initial conditions, 1 = default aer_bc_opt = 1, 1, 1, ! aerosol boundary conditions, 1 = default aer_ic_opt = 1, 1, 1, ! aerosol initial conditions, 1 = default gaschem_onoff = 1, 1, 1, ! gas phase chemistry, 1 = on aerchem_onoff = 1, 1, 1, ! aerosol chemistry, 1 = on wetscav_onoff = 1, 1, 1, ! wet scavenging in stratocumulus clouds, 1 = on cldchem_onoff = 0, 1, 1, ! aqueous chemistry in stratocumulus clouds, 1 = on vertmix_onoff = 1, 1, 1, ! vertical turbulent mixing, 1 = on chem_conv_tr = 1, 0, 0, ! subgrid convective transport, 1 = on conv_tr_wetscav = 1, 0, 0, ! wet scavenging in cumulus clouds, subgrid, 1 = on conv_tr_aqchem = 1, 0, 0, ! aqueous chemistry in cumulus clouds, subgrid, 1 = on seas_opt = 2, ! sea salt emissions, 2 = MOSAIC or MADE/SORGAM sea salt emissions dust_opt = 3, ! dust emissions, 3 = GOCART dust emissions with AFWA modifications dmsemis_opt = 1, ! include GOCART dms emissions from sea surface biomass_burn_opt = 2, 2, 2, ! biomass burning emissions, 2 = MOCART plumerisefire_frq = 30, 30, 30, ! time interval for calling the biomass burning plume rise subroutine scale_fire_emiss = .true., .true., .true., ! must be equal to .true. when running with FINN emissions aer_ra_feedback = 1, 1, 1, ! feedback from the aerosols to the radiation schemes, 1 = on ne_area = 500, ! total number of chemical species, in the chemical name list, best to set to a value larger than all chemical species opt_pars_out = 1, ! include optical properties in output have_bcs_chem = .true., .false., .false., ! gets lateral boundary data from wrfbdy (.true.) or idealized profile (.false.) have_bcs_upper = .false., .false., .false., ! upper boundary bounary condition for chemical species aer_op_opt = 2, 2, 2, ! aerosol optical properties, 1 = volume, 2 = approximate Maxwell-Garnet, 3 = complex volume-mixing, 4 = complex Maxwell-Garnet, 5 = complex core-shell bbinjectscheme = 1, 1, 1, ! 0 = plumerise (biomass_burn_opt), 1 = all ground level (recommended), 2 = flaming evenly in BL, 3 = flaming top BL, 4 = flaming injected at specific height /

lukeconibear commented 4 years ago

Thanks for submitting the issue here.

Is this using the CEMAC method or a manual installation? Which versions of WRFotron and WRFChem are you using?

Do you have real.exe in the run folder? If you run ldd real.exe do the list of linked libraries from compilation show up?

Do you have the default settings within pre.bash of calling ${mpiCommandPre} ./real.exe and within config.bash that mpiCommandPre=mpirun?

chmltf commented 4 years ago

Hi Luke, Thanks for the quick response. I did the manual installation. I am using WRFotron 2.0 and WRFChem 3.7.1. real.exe is in the run folder and you can see the linked libraries in the compiled code from ldd real.exe Yes, the pre.bash contains ${mpiCommandPre} ./real.exe and config.bash mpiCommandPre=mpirun.

lukeconibear commented 4 years ago

Hmm okay. What is the path to the run and I can take a look to see if I can spot anything?

chmltf commented 4 years ago

Here is the path to the run folder /nobackup/chmltf/HONO_13/run/base/2014-01-02_12:00:00-2014-01-05_00:00:00 Thanks.

lukeconibear commented 4 years ago

In the first real without chemistry section of pre.bash around line 82, only the contents of WRFdir/run/* is copied over:

cp -r ${WRFdir}/run/* .

This means that the main executables remain linked to one folder up within ../main/ (which is the case if you're in WRFdir/run):

$ ll *exe
lrwxrwxrwx 1 chmltf chm 17 Jul 29 16:49 ndown.exe -> ../main/ndown.exe
lrwxrwxrwx 1 chmltf chm 16 Jul 29 16:49 real.exe -> ../main/real.exe
lrwxrwxrwx 1 chmltf chm 14 Jul 29 16:49 tc.exe -> ../main/tc.exe
lrwxrwxrwx 1 chmltf chm 15 Jul 29 16:49 wrf.exe -> ../main/wrf.exe

The code then breaks when it cannot find these linked files.

The CEMAC installation method is okay with this, though for the manual installation method you will need to remove these linked executables and copy the executables over directly. You can do this by adding the additional 3 lines below:

cp -r ${WRFdir}/run/* .
rm *.exe
cp -r ${WRFdir}/main/*.exe .
cp -r ${WRFmeteodir}/main/wrf.exe wrfmeteo.exe

This should enable pre.bash to find the executables. Let me know if this works.

chmltf commented 4 years ago

I have added those 3 lines to my pre.bash and submit the same run again (overwriting the previous). The error message is different this time, saying there is a problem with loading a shared library.

Name of source model =>NCEP GFS Analysis GRID 4 real.exe: error while loading shared libraries: libfabric.so.1: cannot open shared object file: No such file or directory

Primary job terminated normally, but 1 process returned a non-zero exit code. Per user-direction, the job has been aborted.


mpirun detected that one or more processes exited with non-zero status, thus causing the job to be terminated. The first process to do so was:

Process name: [[62881,1],0] Exit code: 127

lukeconibear commented 4 years ago

This looks like it's found the real.exe executable but the linked library libfabric isn't recognised, for example:

$ ldd real.exe
        linux-vdso.so.1 =>  (0x00007ffc62f65000)
        libnetcdff.so.6 => not found
        libnetcdf.so.15 => not found
        libmpifort.so.12 => /apps/developers/libraries/intelmpi/2019.4.243/1/intel-19.0.4/intel64/lib/libmpifort.so.12 (0x00007f3517f8c000)
        libmpi.so.12 => /apps/developers/libraries/intelmpi/2019.4.243/1/intel-19.0.4/intel64/lib/release/libmpi.so.12 (0x00007f351709a000)
        libdl.so.2 => /lib64/libdl.so.2 (0x00007f3516e96000)
        librt.so.1 => /lib64/librt.so.1 (0x00007f3516c8e000)
        libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f3516a72000)
        libm.so.6 => /lib64/libm.so.6 (0x00007f3516770000)
        libc.so.6 => /lib64/libc.so.6 (0x00007f35163a3000)
        libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f351618d000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f351834b000)
        libfabric.so.1 => not found

This can happen when the required modules aren't loaded. If I remember rightly you manually compiled WRFChem using intel, intelmpi, and netcdf. If you module load these (as below) the executable should find the paths to these linked libraries. You may want/need to module purge first to remove any other modules which may interfere e.g. openmpi.

$ module purge
$ module list
No Modulefiles Currently Loaded.
$ module load intel
$ module load intelmpi
$ module load netcdf
$ module list
Currently Loaded Modulefiles:
  1) intel/19.0.4          2) intelmpi/2019.4.243   3) netcdf/4.6.3
$ ldd real.exe
        linux-vdso.so.1 =>  (0x00007ffec77a8000)
        libnetcdff.so.6 => /apps/developers/libraries/netcdf/4.6.3/1/intel-19.0.4-intelmpi-2019.4.243/lib/libnetcdff.so.6 (0x00007fa84c735000)
        libnetcdf.so.15 => /apps/developers/libraries/netcdf/4.6.3/1/intel-19.0.4-intelmpi-2019.4.243/lib/libnetcdf.so.15 (0x00007fa84c216000)
        libmpifort.so.12 => /apps/developers/libraries/intelmpi/2019.4.243/1/intel-19.0.4/lib/libmpifort.so.12 (0x00007fa84be57000)
        libmpi.so.12 => /apps/developers/libraries/intelmpi/2019.4.243/1/intel-19.0.4/lib/release/libmpi.so.12 (0x00007fa84af65000)
        libdl.so.2 => /lib64/libdl.so.2 (0x00007fa84ad61000)
        librt.so.1 => /lib64/librt.so.1 (0x00007fa84ab59000)
        libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fa84a93d000)
        libm.so.6 => /lib64/libm.so.6 (0x00007fa84a63b000)
        libc.so.6 => /lib64/libc.so.6 (0x00007fa84a26e000)
        libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007fa84a058000)
        libhdf5_hl.so.10 => /apps/developers/libraries/hdf5/1.8.21/1/intel-19.0.4-intelmpi-2019.4.243/lib/libhdf5_hl.so.10 (0x00007fa849e31000)
        libhdf5.so.10 => /apps/developers/libraries/hdf5/1.8.21/1/intel-19.0.4-intelmpi-2019.4.243/lib/libhdf5.so.10 (0x00007fa849861000)
        libz.so.1 => /lib64/libz.so.1 (0x00007fa84964b000)
        libcurl.so.4 => /lib64/libcurl.so.4 (0x00007fa8493e2000)
        libifport.so.5 => /apps/developers/compilers/intel/19.0.4/1/default/compilers_and_libraries_2019.4.243/linux/compiler/lib/intel64_lin/libifport.so.5 (0x00007fa8491b4000)
        libifcoremt.so.5 => /apps/developers/compilers/intel/19.0.4/1/default/compilers_and_libraries_2019.4.243/linux/compiler/lib/intel64_lin/libifcoremt.so.5 (0x00007fa848e1f000)
        libimf.so => /apps/developers/compilers/intel/19.0.4/1/default/compilers_and_libraries_2019.4.243/linux/compiler/lib/intel64_lin/libimf.so (0x00007fa84887f000)
        libsvml.so => /apps/developers/compilers/intel/19.0.4/1/default/compilers_and_libraries_2019.4.243/linux/compiler/lib/intel64_lin/libsvml.so (0x00007fa846edb000)
        libintlc.so.5 => /apps/developers/compilers/intel/19.0.4/1/default/compilers_and_libraries_2019.4.243/linux/compiler/lib/intel64_lin/libintlc.so.5 (0x00007fa846c69000)
        libirng.so => /apps/developers/compilers/intel/19.0.4/1/default/compilers_and_libraries_2019.4.243/linux/compiler/lib/intel64_lin/libirng.so (0x00007fa8468f7000)
        libfabric.so.1 => /apps/developers/libraries/intelmpi/2019.4.243/1/intel-19.0.4/libfabric/lib/libfabric.so.1 (0x00007fa8466bf000)
        /lib64/ld-linux-x86-64.so.2 (0x00007fa84ccf2000)
        libidn.so.11 => /lib64/libidn.so.11 (0x00007fa84648c000)
        libssh2.so.1 => /lib64/libssh2.so.1 (0x00007fa846262000)
        libssl3.so => /lib64/libssl3.so (0x00007fa846010000)
        libsmime3.so => /lib64/libsmime3.so (0x00007fa845de9000)
        libnss3.so => /lib64/libnss3.so (0x00007fa845abc000)
        libnssutil3.so => /lib64/libnssutil3.so (0x00007fa84588c000)
        libplds4.so => /lib64/libplds4.so (0x00007fa845688000)
        libplc4.so => /lib64/libplc4.so (0x00007fa845483000)
        libnspr4.so => /lib64/libnspr4.so (0x00007fa845245000)
        libgssapi_krb5.so.2 => /lib64/libgssapi_krb5.so.2 (0x00007fa844ff8000)
        libkrb5.so.3 => /lib64/libkrb5.so.3 (0x00007fa844d0f000)
        libk5crypto.so.3 => /lib64/libk5crypto.so.3 (0x00007fa844adc000)
        libcom_err.so.2 => /lib64/libcom_err.so.2 (0x00007fa8448d8000)
        liblber-2.4.so.2 => /lib64/liblber-2.4.so.2 (0x00007fa8446c9000)
        libldap-2.4.so.2 => /lib64/libldap-2.4.so.2 (0x00007fa844474000)
        libssl.so.10 => /lib64/libssl.so.10 (0x00007fa844202000)
        libcrypto.so.10 => /lib64/libcrypto.so.10 (0x00007fa843da0000)
        libkrb5support.so.0 => /lib64/libkrb5support.so.0 (0x00007fa843b90000)
        libkeyutils.so.1 => /lib64/libkeyutils.so.1 (0x00007fa84398c000)
        libresolv.so.2 => /lib64/libresolv.so.2 (0x00007fa843773000)
        libsasl2.so.3 => /lib64/libsasl2.so.3 (0x00007fa843556000)
        libselinux.so.1 => /lib64/libselinux.so.1 (0x00007fa84332f000)
        libcrypt.so.1 => /lib64/libcrypt.so.1 (0x00007fa8430f8000)
        libpcre.so.1 => /lib64/libpcre.so.1 (0x00007fa842e96000)
        libfreebl3.so => /lib64/libfreebl3.so (0x00007fa842c93000)

If you resubmit the job from within a shell that has these modules loaded it should find these required libraries. Let me know if this works.

chmltf commented 4 years ago

Unfortunately it is still aborting the run in the same place.

I've made these changes to the loaded modules and all linked libraries show up for real.exe. I am still using the CEMAC modules, too.

 ldd real.exe
        linux-vdso.so.1 =>  (0x00007ffeaeb41000)
        libnetcdff.so.6 => /apps/developers/libraries/netcdf/4.6.3/1/intel-19.0.4-intelmpi-2019.4.243/lib/libnetcdff.so.6 (0x00007f379ab8c000)
        libnetcdf.so.15 => /apps/developers/libraries/netcdf/4.6.3/1/intel-19.0.4-intelmpi-2019.4.243/lib/libnetcdf.so.15 (0x00007f379a66d000)
        libmpifort.so.12 => /apps/developers/libraries/intelmpi/2019.4.243/1/intel-19.0.4/lib/libmpifort.so.12 (0x00007f379a2ae000)
        libmpi.so.12 => /apps/developers/libraries/intelmpi/2019.4.243/1/intel-19.0.4/lib/release/libmpi.so.12 (0x00007f37993bc000)
        libdl.so.2 => /lib64/libdl.so.2 (0x00007f37991b8000)
        librt.so.1 => /lib64/librt.so.1 (0x00007f3798fb0000)
        libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f3798d94000)
        libm.so.6 => /lib64/libm.so.6 (0x00007f3798a92000)
        libc.so.6 => /lib64/libc.so.6 (0x00007f37986c5000)
        libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f37984af000)
        libhdf5_hl.so.10 => /apps/developers/libraries/hdf5/1.8.21/1/intel-19.0.4-intelmpi-2019.4.243/lib/libhdf5_hl.so.10 (0x00007f3798288000)
        libhdf5.so.10 => /apps/developers/libraries/hdf5/1.8.21/1/intel-19.0.4-intelmpi-2019.4.243/lib/libhdf5.so.10 (0x00007f3797cb8000)
        libz.so.1 => /lib64/libz.so.1 (0x00007f3797aa2000)
        libcurl.so.4 => /lib64/libcurl.so.4 (0x00007f3797839000)
        libifport.so.5 => /apps/developers/compilers/intel/19.0.4/1/default/compilers_and_libraries_2019.4.243/linux/compiler/lib/intel64_lin/libifport.so.5 (0x00007f379760b000)
        libifcoremt.so.5 => /apps/developers/compilers/intel/19.0.4/1/default/compilers_and_libraries_2019.4.243/linux/compiler/lib/intel64_lin/libifcoremt.so.5 (0x00007f3797276000)
        libimf.so => /apps/developers/compilers/intel/19.0.4/1/default/compilers_and_libraries_2019.4.243/linux/compiler/lib/intel64_lin/libimf.so (0x00007f3796cd6000)
        libsvml.so => /apps/developers/compilers/intel/19.0.4/1/default/compilers_and_libraries_2019.4.243/linux/compiler/lib/intel64_lin/libsvml.so (0x00007f3795332000)
        libintlc.so.5 => /apps/developers/compilers/intel/19.0.4/1/default/compilers_and_libraries_2019.4.243/linux/compiler/lib/intel64_lin/libintlc.so.5 (0x00007f37950c0000)
        libirng.so => /apps/developers/compilers/intel/19.0.4/1/default/compilers_and_libraries_2019.4.243/linux/compiler/lib/intel64_lin/libirng.so (0x00007f3794d4e000)
        libfabric.so.1 => /apps/developers/libraries/intelmpi/2019.4.243/1/intel-19.0.4/libfabric/lib/libfabric.so.1 (0x00007f3794b16000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f379b149000)
        libidn.so.11 => /lib64/libidn.so.11 (0x00007f37948e3000)
        libssh2.so.1 => /lib64/libssh2.so.1 (0x00007f37946b9000)
        libssl3.so => /lib64/libssl3.so (0x00007f3794467000)
        libsmime3.so => /lib64/libsmime3.so (0x00007f3794240000)
        libnss3.so => /lib64/libnss3.so (0x00007f3793f13000)
        libnssutil3.so => /lib64/libnssutil3.so (0x00007f3793ce3000)
        libplds4.so => /lib64/libplds4.so (0x00007f3793adf000)
        libplc4.so => /lib64/libplc4.so (0x00007f37938da000)
        libnspr4.so => /lib64/libnspr4.so (0x00007f379369c000)
        libgssapi_krb5.so.2 => /lib64/libgssapi_krb5.so.2 (0x00007f379344f000)
        libkrb5.so.3 => /lib64/libkrb5.so.3 (0x00007f3793166000)
        libk5crypto.so.3 => /lib64/libk5crypto.so.3 (0x00007f3792f33000)
        libcom_err.so.2 => /lib64/libcom_err.so.2 (0x00007f3792d2f000)
        liblber-2.4.so.2 => /lib64/liblber-2.4.so.2 (0x00007f3792b20000)
        libldap-2.4.so.2 => /lib64/libldap-2.4.so.2 (0x00007f37928cb000)
        libssl.so.10 => /lib64/libssl.so.10 (0x00007f3792659000)
        libcrypto.so.10 => /lib64/libcrypto.so.10 (0x00007f37921f7000)
        libkrb5support.so.0 => /lib64/libkrb5support.so.0 (0x00007f3791fe7000)
        libkeyutils.so.1 => /lib64/libkeyutils.so.1 (0x00007f3791de3000)
        libresolv.so.2 => /lib64/libresolv.so.2 (0x00007f3791bca000)
        libsasl2.so.3 => /lib64/libsasl2.so.3 (0x00007f37919ad000)
        libselinux.so.1 => /lib64/libselinux.so.1 (0x00007f3791786000)
        libcrypt.so.1 => /lib64/libcrypt.so.1 (0x00007f379154f000)
        libpcre.so.1 => /lib64/libpcre.so.1 (0x00007f37912ed000)
        libfreebl3.so => /lib64/libfreebl3.so (0x00007f37910ea000)

Here is the pre.bash.eXXXXXX file now:

Name of source model =>NCEP GFS Analysis       GRID   4
[proxy:0:0@d8s0b2.arc4.leeds.ac.uk] HYD_spawn (../../../../../src/pm/i_hydra/libhydra/spawn/intel/hydra_spawn.c:117): execvp error on file real.exe (No such file or directory)
mv: cannot stat ‘rsl*’: No such file or directory
netcdf error
mv: cannot stat ‘wrfbiochemi_d01’: No such file or directory
ncatted: ERROR file /nobackup/chmltf/HONO_13/run/wrfbiochemi_d01_WRFChem3.7.1 not found. It does not exist on the local filesystem, nor does it match remote filename patterns (e.g., http://foo or foo.bar.edu:file).
ncatted: HINT file-not-found errors usually arise from filename typos, incorrect paths, missing files, or capricious gods. Please verify spelling and location of requested file. If the file resides on a High Performance Storage System (HPSS) accessible via the 'hsi' command, then add the --hpss option and re-try command.
ncks: ERROR file wrfinput_d01 not found. It does not exist on the local filesystem, nor does it match remote filename patterns (e.g., http://foo or foo.bar.edu:file).
ncks: HINT file-not-found errors usually arise from filename typos, incorrect paths, missing files, or capricious gods. Please verify spelling and location of requested file. If the file resides on a High Performance Storage System (HPSS) accessible via the 'hsi' command, then add the --hpss option and re-try command.
File err
lukeconibear commented 4 years ago

Okay. From the shell you're submitting the . master.bash 2014 01 ... command on, could you run module list and return the results? Also, do you have conda activated on this same shell, and if so could you also run conda list and return the results?

chmltf commented 4 years ago
module list
Currently Loaded Modulefiles:
  1) licenses              3) intel/19.0.4          5) intelmpi/2019.4.243   7) WRFchem/3.7.1         9) nco/4.8.1
  2) sge                   4) user                  6) netcdf/4.6.3          8) ncl/6.5.0            10) wrfchemconda/2.7
conda list
# packages in environment at /nobackup/WRFChem/anaconda2:
#
# Name                    Version                   Build  Channel
_ipyw_jlab_nb_ext_conf    0.1.0                    py27_0
_libgcc_mutex             0.1                        main
alabaster                 0.7.12                   py27_0
anaconda                  2019.10                  py27_0
anaconda-client           1.7.2                    py27_0
anaconda-navigator        1.9.7                    py27_0
anaconda-project          0.8.3                      py_0
asn1crypto                1.0.1                    py27_0
astroid                   1.6.5                    py27_0
astropy                   2.0.9            py27hdd07704_0
atomicwrites              1.3.0                    py27_1
attrs                     19.2.0                     py_0
babel                     2.7.0                      py_0
backports                 1.0                        py_2
backports.functools_lru_cache 1.5                        py_2
backports.os              0.1.1                    py27_0
backports.shutil_get_terminal_size 1.0.0                    py27_2
backports.tempfile        1.0                        py_1
backports.weakref         1.0.post1                  py_1
backports_abc             0.5                      py27_0
beautifulsoup4            4.8.0                    py27_0
bitarray                  1.0.1            py27h7b6447c_0
bkcharts                  0.2                      py27_0
blas                      1.0                         mkl
bleach                    3.1.0                    py27_0
blosc                     1.16.3               hd408876_0
bokeh                     1.3.4                    py27_0
boto                      2.49.0                   py27_0
bottleneck                1.2.1            py27h035aef0_1
bzip2                     1.0.8                h7b6447c_0
ca-certificates           2019.8.28                     0
cairo                     1.14.12              h8948797_3
cdecimal                  2.3              py27h14c3975_3
certifi                   2019.9.11                py27_0
cffi                      1.12.3           py27h2e261b9_0
cftime                    1.0.4.2          py27hdd07704_0
chardet                   3.0.4                 py27_1003
click                     7.0                      py27_0
cloudpickle               1.2.2                      py_0
clyent                    1.2.2                    py27_1
colorama                  0.4.1                    py27_0
conda                     4.8.3            py27h8c360ce_1    conda-forge
conda-build               3.18.9                   py27_3
conda-env                 2.6.0                         1
conda-package-handling    1.6.0            py27h7b6447c_0
conda-verify              3.4.2                      py_1
configparser              4.0.2                    py27_0
contextlib2               0.6.0                      py_0
cryptography              2.7              py27h1ba5d50_0
curl                      7.65.3               hbc83047_0
cycler                    0.10.0                   py27_0
cython                    0.29.13          py27he6710b0_0
cytoolz                   0.10.0           py27h7b6447c_0
dask                      1.2.2                      py_0
dask-core                 1.2.2                      py_0
dbus                      1.13.6               h746ee38_0
decorator                 4.4.0                    py27_1
defusedxml                0.6.0                      py_0
distributed               1.28.1                   py27_0
docutils                  0.15.2                   py27_0
entrypoints               0.3                      py27_0
enum34                    1.1.6                    py27_1
et_xmlfile                1.0.1                    py27_0
expat                     2.2.6                he6710b0_0
fastcache                 1.1.0            py27h7b6447c_0
filelock                  3.0.12                     py_0
flask                     1.1.1                      py_0
fontconfig                2.13.0               h9420a91_0
freetype                  2.9.1                h8a8886c_1
fribidi                   1.0.5                h7b6447c_0
funcsigs                  1.0.2                    py27_0
functools32               3.2.3.2                  py27_1
future                    0.17.1                   py27_0
futures                   3.3.0                    py27_0
get_terminal_size         1.0.0                haa9412d_0
gevent                    1.4.0            py27h7b6447c_0
glib                      2.56.2               hd408876_0
glob2                     0.7                        py_0
gmp                       6.1.2                h6c8ec71_1
gmpy2                     2.0.8            py27h10f8cd9_2
graphite2                 1.3.13               h23475e2_0
greenlet                  0.4.15           py27h7b6447c_0
grin                      1.2.1                    py27_4
gst-plugins-base          1.14.0               hbbd80ab_1
gstreamer                 1.14.0               hb453b48_1
h5py                      2.9.0            py27h7918eee_0
harfbuzz                  1.8.8                hffaf4a1_0
hdf4                      4.2.13               h3ca952b_2
hdf5                      1.10.4               hb1b8bf9_0
heapdict                  1.0.1                      py_0
html5lib                  1.0.1                    py27_0
icu                       58.2                 h9c2bf20_1
idna                      2.8                      py27_0
imageio                   2.6.0                    py27_0
imagesize                 1.1.0                    py27_0
importlib_metadata        0.23                     py27_0
intel-openmp              2019.4                      243
ipaddress                 1.0.22                   py27_0
ipykernel                 4.10.0                   py27_0
ipython                   5.8.0                    py27_0
ipython_genutils          0.2.0                    py27_0
ipywidgets                7.5.1                      py_0
isort                     4.3.21                   py27_0
itsdangerous              1.1.0                    py27_0
jbig                      2.1                  hdba287a_0
jdcal                     1.4.1                      py_0
jedi                      0.15.1                   py27_0
jinja2                    2.10.3                     py_0
jpeg                      9b                   h024ee3a_2
jsonschema                3.0.2                    py27_0
jupyter                   1.0.0                    py27_7
jupyter_client            5.3.3                    py27_1
jupyter_console           5.2.0                    py27_1
jupyter_core              4.5.0                      py_0
jupyterlab                0.33.11                  py27_0
jupyterlab_launcher       0.11.2           py27h28b3542_0
kiwisolver                1.1.0            py27he6710b0_0
krb5                      1.16.1               h173b8e3_7
lazy-object-proxy         1.4.2            py27h7b6447c_0
libarchive                3.3.3                h5d8350f_5
libcurl                   7.65.3               h20c2e04_0
libedit                   3.1.20181209         hc058e9b_0
libffi                    3.2.1                hd88cf55_4
libgcc-ng                 9.1.0                hdf63c60_0
libgfortran-ng            7.3.0                hdf63c60_0
liblief                   0.9.0                h7725739_2
libnetcdf                 4.6.1                h11d0813_2
libpng                    1.6.37               hbc83047_0
libsodium                 1.0.16               h1bed415_0
libssh2                   1.8.2                h1ba5d50_0
libstdcxx-ng              9.1.0                hdf63c60_0
libtiff                   4.0.10               h2733197_2
libtool                   2.4.6                h7b6447c_5
libuuid                   1.0.3                h1bed415_2
libxcb                    1.13                 h1bed415_1
libxml2                   2.9.9                hea5a465_1
libxslt                   1.1.33               h7d1a2b0_0
linecache2                1.0.0                    py27_0
llvmlite                  0.29.0           py27hd408876_0
locket                    0.2.0                    py27_1
lxml                      4.4.1            py27hefd8a0e_0
lz4-c                     1.8.1.2              h14c3975_0
lzo                       2.10                 h49e0be7_2
markupsafe                1.1.1            py27h7b6447c_0
matplotlib                2.2.3            py27hb69df0a_0
mccabe                    0.6.1                    py27_1
mistune                   0.8.4            py27h7b6447c_0
mkl                       2019.4                      243
mkl-service               2.3.0            py27he904b0f_0
mkl_fft                   1.0.14           py27ha843d7b_0
mkl_random                1.1.0            py27hd6b4f25_0
mock                      3.0.5                    py27_0
more-itertools            5.0.0                    py27_0
mpc                       1.1.0                h10f8cd9_1
mpfr                      4.0.1                hdf1c602_3
mpmath                    1.1.0                    py27_0
msgpack-python            0.6.1            py27hfd86e86_1
multipledispatch          0.6.0                    py27_0
navigator-updater         0.2.1                    py27_0
nbconvert                 5.6.0                    py27_1
nbformat                  4.4.0                    py27_0
ncurses                   6.1                  he6710b0_1
netcdf4                   1.4.2            py27h808af73_0
networkx                  2.2                      py27_1
nltk                      3.4.5                    py27_0
nose                      1.3.7                    py27_2
notebook                  5.7.8                    py27_0
numba                     0.45.1           py27h962f231_0
numexpr                   2.7.0            py27h9e4a6bb_0
numpy                     1.16.5           py27h7e9f1db_0
numpy-base                1.16.5           py27hde5b4d6_0
numpydoc                  0.9.1                      py_0
olefile                   0.46                     py27_0
openpyxl                  2.6.3                      py_0
openssl                   1.1.1d               h7b6447c_2
packaging                 19.2                       py_0
pandas                    0.24.2           py27he6710b0_0
pandoc                    2.2.3.2                       0
pandocfilters             1.4.2                    py27_1
pango                     1.42.4               h049681c_0
parso                     0.5.1                      py_0
partd                     1.0.0                      py_0
patchelf                  0.9                  he6710b0_3
path.py                   11.5.0                   py27_0
pathlib2                  2.3.5                    py27_0
patsy                     0.5.1                    py27_0
pcre                      8.43                 he6710b0_0
pep8                      1.7.1                    py27_0
pexpect                   4.7.0                    py27_0
pickleshare               0.7.5                    py27_0
pillow                    6.2.0            py27h34e0f95_0
pip                       19.2.3                   py27_0
pixman                    0.38.0               h7b6447c_0
pkginfo                   1.5.0.1                  py27_0
pluggy                    0.13.0                   py27_0
ply                       3.11                     py27_0
prometheus_client         0.7.1                      py_0
prompt_toolkit            1.0.15                   py27_0
psutil                    5.6.3            py27h7b6447c_0
ptyprocess                0.6.0                    py27_0
py                        1.8.0                    py27_0
py-lief                   0.9.0            py27h7725739_2
pycairo                   1.18.1           py27h2a1e443_0
pycodestyle               2.5.0                    py27_0
pycosat                   0.6.3            py27h14c3975_0
pycparser                 2.19                     py27_0
pycrypto                  2.6.1            py27h14c3975_9
pycurl                    7.43.0.3         py27h1ba5d50_0
pyflakes                  2.1.1                    py27_0
pygments                  2.4.2                      py_0
pylint                    1.9.2                    py27_0
pyodbc                    4.0.27           py27he6710b0_0
pyopenssl                 19.0.0                   py27_0
pyparsing                 2.4.2                      py_0
pyqt                      5.9.2            py27h05f1152_2
pyrsistent                0.15.4           py27h7b6447c_0
pysocks                   1.7.1                    py27_0
pytables                  3.5.2            py27h71ec239_1
pytest                    4.6.2                    py27_0
python                    2.7.16               h9bab390_7
python-dateutil           2.8.0                    py27_0
python-libarchive-c       2.8                     py27_13
python_abi                2.7                    1_cp27mu    conda-forge
pytz                      2019.3                     py_0
pywavelets                1.0.3            py27hdd07704_1
pyyaml                    5.1.2            py27h7b6447c_0
pyzmq                     18.1.0           py27he6710b0_0
qt                        5.9.7                h5867ecd_1
qtawesome                 0.6.0                      py_0
qtconsole                 4.5.5                      py_0
qtpy                      1.9.0                      py_0
readline                  7.0                  h7b6447c_5
requests                  2.22.0                   py27_0
ripgrep                   0.10.0               hc07d326_0
rope                      0.14.0                     py_0
ruamel_yaml               0.15.46          py27h14c3975_0
scandir                   1.10.0           py27h7b6447c_0
scikit-image              0.14.2           py27he6710b0_0
scikit-learn              0.20.3           py27hd81dba3_0
scipy                     1.2.1            py27h7c811a0_0
seaborn                   0.9.0                    py27_0
send2trash                1.5.0                    py27_0
setuptools                41.4.0                   py27_0
simplegeneric             0.8.1                    py27_2
singledispatch            3.4.0.3                  py27_0
sip                       4.19.8           py27hf484d3e_0
six                       1.12.0                   py27_0
snappy                    1.1.7                hbae5bb6_3
snowballstemmer           2.0.0                      py_0
sortedcollections         1.1.2                    py27_0
sortedcontainers          2.1.0                    py27_0
soupsieve                 1.9.3                    py27_0
sphinx                    1.8.5                    py27_0
sphinxcontrib             1.0                      py27_1
sphinxcontrib-websupport  1.1.2                      py_0
spyder                    3.3.6                    py27_0
spyder-kernels            0.5.2                    py27_0
sqlalchemy                1.3.9            py27h7b6447c_0
sqlite                    3.30.0               h7b6447c_0
ssl_match_hostname        3.7.0.1                  py27_0
statsmodels               0.10.1           py27hdd07704_0
subprocess32              3.5.4            py27h7b6447c_0
sympy                     1.4                      py27_0
tbb                       2019.4               hfd86e86_0
tblib                     1.4.0                      py_0
terminado                 0.8.2                    py27_0
testpath                  0.4.2                    py27_0
tk                        8.6.8                hbc83047_0
toolz                     0.10.0                     py_0
tornado                   5.1.1            py27h7b6447c_0
tqdm                      4.36.1                     py_0
traceback2                1.4.0                    py27_0
traitlets                 4.3.3                    py27_0
typing                    3.7.4.1                  py27_0
unicodecsv                0.14.1                   py27_0
unittest2                 1.1.0                    py27_0
unixodbc                  2.3.7                h14c3975_0
urllib3                   1.24.2                   py27_0
wcwidth                   0.1.7                    py27_0
webencodings              0.5.1                    py27_1
werkzeug                  0.16.0                     py_0
wheel                     0.33.6                   py27_0
widgetsnbextension        3.5.1                    py27_0
wrapt                     1.11.2           py27h7b6447c_0
wrf-python                1.3.2            py27h7eb8c7e_1    conda-forge
wurlitzer                 1.0.3                    py27_0
xarray                    0.11.3                   py27_0
xlrd                      1.2.0                    py27_0
xlsxwriter                1.2.1                      py_0
xlwt                      1.3.0                    py27_0
xz                        5.2.4                h14c3975_4
yaml                      0.1.7                had09818_2
zeromq                    4.3.1                he6710b0_3
zict                      1.0.0                      py_0
zipp                      0.6.0                      py_0
zlib                      1.2.11               h7b6447c_3
zstd                      1.3.7                h0b5b093_0
lukeconibear commented 4 years ago

I guess that having the CEMAC module WRFchem/3.7.1 loaded too is causing the issue. This enables access to the CEMAC pre-compiled executables, which may be interfering with the use of your manually compiled versions. Try unloading at least this module: module unload WRFchem/3.7.1 and see if that fixes the issue?

chmltf commented 4 years ago

I made sure the WRFchem/3.7.1 module was unloaded. Here is the pre.bash.exxxxxx now.

/var/spool/sge_prod/d8s0b1/job_scripts/661484: line 65: ungrib.exe: command not found
[proxy:0:0@d8s0b1.arc4.leeds.ac.uk] HYD_spawn (../../../../../src/pm/i_hydra/libhydra/spawn/intel/hydra_spawn.c:117): execvp error on file geogrid.exe (No such file or directory)
[proxy:0:0@d8s0b1.arc4.leeds.ac.uk] HYD_spawn (../../../../../src/pm/i_hydra/libhydra/spawn/intel/hydra_spawn.c:117): execvp error on file metgrid.exe (No such file or directory)
rm: cannot remove ‘FILE*’: No such file or directory
[proxy:0:0@d8s0b1.arc4.leeds.ac.uk] HYD_spawn (../../../../../src/pm/i_hydra/libhydra/spawn/intel/hydra_spawn.c:117): execvp error on file real.exe (No such file or directory)
mv: cannot stat ‘rsl*’: No such file or directory
/var/spool/sge_prod/d8s0b1/job_scripts/661484: line 115: megan_bio_emiss: command not found
mv: cannot stat ‘wrfbiochemi_d01’: No such file or directory
ncatted: ERROR file /nobackup/chmltf/HONO_13/run/wrfbiochemi_d01_WRFChem3.7.1 not found. It does not exist on the local filesystem, nor does it match remote filename patterns (e.g., http://foo or foo.bar.edu:file).
ncatted: HINT file-not-found errors usually arise from filename typos, incorrect paths, missing files, or capricious gods. Please verify spelling and location of requested file. If the file resides on a High Performance Storage System (HPSS) accessible via the 'hsi' command, then add the --hpss option and re-try command.
ncks: ERROR file wrfinput_d01 not found. It does not exist on the local filesystem, nor does it match remote filename patterns (e.g., http://foo or foo.bar.edu:file).
ncks: HINT file-not-found errors usually arise from filename typos, incorrect paths, missing files, or capricious gods. Please verify spelling and location of requested file. If the file resides on a High Performance Storage System (HPSS) accessible via the 'hsi' command, then add the --hpss option and re-try command.
/var/spool/sge_prod/d8s0b1/job_scripts/661484: line 137: wesely: command not found
/var/spool/sge_prod/d8s0b1/job_scripts/661484: line 139: exo_coldens: command not found
/var/spool/sge_prod/d8s0b1/job_scripts/661484: line 148: anthro_emis: command not found
ln: failed to create symbolic link ‘./anthro_emis.inp’: File exists
Traceback (most recent call last):
  File "sum_sector_emiss_wrfchemi.py", line 22, in <module>
    open_ds = xr.open_dataset(cwd + '/' + file)
  File "/nobackup/cemac/software/apps/wrfchemconda/2.7/1/default/lib/python2.7/site-packages/xarray/backends/api.py", line 321, in open_dataset
    filename_or_obj, group=group, lock=lock, **backend_kwargs)
  File "/nobackup/cemac/software/apps/wrfchemconda/2.7/1/default/lib/python2.7/site-packages/xarray/backends/netCDF4_.py", line 355, in open
    return cls(manager, lock=lock, autoclose=autoclose)
  File "/nobackup/cemac/software/apps/wrfchemconda/2.7/1/default/lib/python2.7/site-packages/xarray/backends/netCDF4_.py", line 314, in __init__
    self.format = self.ds.data_model
  File "/nobackup/cemac/software/apps/wrfchemconda/2.7/1/default/lib/python2.7/site-packages/xarray/backends/netCDF4_.py", line 359, in ds
    return self._manager.acquire().value
  File "/nobackup/cemac/software/apps/wrfchemconda/2.7/1/default/lib/python2.7/site-packages/xarray/backends/file_manager.py", line 172, in acquire
    file = self._opener(*self._args, **kwargs)
  File "/nobackup/cemac/software/apps/wrfchemconda/2.7/1/default/lib/python2.7/site-packages/xarray/backends/netCDF4_.py", line 247, in _open_netcdf4_group
    ds = nc4.Dataset(filename, mode=mode, **kwargs)
  File "netCDF4/_netCDF4.pyx", line 2135, in netCDF4._netCDF4.Dataset.__init__
  File "netCDF4/_netCDF4.pyx", line 1752, in netCDF4._netCDF4._ensure_nc_success
IOError: [Errno 2] No such file or directory: '/nobackup/chmltf/HONO_13/run/base/2014-01-02_12:00:00-2014-01-05_00:00:00/wrfchemi_00z_d01'
/var/spool/sge_prod/d8s0b1/job_scripts/661484: line 163: fire_emis: command not found
[proxy:0:0@d8s0b1.arc4.leeds.ac.uk] HYD_spawn (../../../../../src/pm/i_hydra/libhydra/spawn/intel/hydra_spawn.c:117): execvp error on file real.exe (No such file or directory)
mv: cannot stat ‘rsl*’: No such file or directory
/var/spool/sge_prod/d8s0b1/job_scripts/661484: line 216: mozbc: command not found
cp: cannot stat ‘/nobackup/chmltf/HONO_13/restart/base/wrfrst_d01_2014-01-03_00:00:00’: No such file or directory
rm: cannot remove ‘met_em*’: No such file or directory
lukeconibear commented 4 years ago

At the start of pre.bash, a call to config.bash is made for setup. I have just noticed that within config.bash, you have the line:

module load intel/19.0.4 openmpi/3.1.4  WRFchem/3.7.1 ncl/6.5.0 nco/4.8.1 wrfchemconda/2.7 

This is for the CEMAC installation. Sorry I was unclear about this. For the manual installation, you don't want to reload modules such as intel, openmpi, or WRFchem as these can interfere with your setup. If you are using NCL, NCO, and conda from CEMAC, then these should be okay to keep. You will need NCL and NCO in some form, and others have installed these through their own personal conda environments. For your situation, it is probably easiest to just remove the modules you don't need, so replace the above with the following:

module load ncl/6.5.0 nco/4.8.1 wrfchemconda/2.7 

Hopefully, this should remove the conflicts between CEMAC and manual executables. Let me know how this goes.

I'll update the user guide with this information for changing between CEMAC and manual runs.

chmltf commented 4 years ago

I did do this in the last run. The top of my config.bash reads:

#module load intel/19.0.4 openmpi/3.1.4 WRFchem/3.7.1
module load ncl/6.5.0 nco/4.8.1 wrfchemconda/2.7 

So only the modules on the bottom line are loaded.

lukeconibear commented 4 years ago

Okay. Could you make the updated /nobackup/chmltf/HONO_13/run/base/2014-01-02_12:00:00-2014-01-05_00:00:00 readable as I noticed the above from the /nobackup/chmltf/HONO_13/run/base/2014-01-12_12:00:00-2014-01-15_00:00:00 run.

lukeconibear commented 4 years ago

Similar to the earlier change we made to pre.bash to copy over the main executables directly rather than linking them, I think we need to do the same for the preprocessing executables.

So within pre.bash around line 22, replace:

for aFile in util geogrid ungrib link_grib.csh metgrid

with:

for aFile in util geogrid geogrid.exe ungrib ungrib.exe link_grib.csh metgrid metgrid.exe
chmltf commented 4 years ago

Sorry about that.

The error is similar after making that change.

/var/spool/sge_prod/d10s0b2/job_scripts/661976: line 66: ungrib.exe: command not found
[proxy:0:0@d10s0b2.arc4.leeds.ac.uk] HYD_spawn (../../../../../src/pm/i_hydra/libhydra/spawn/intel/hydra_spawn.c:117): execvp error on file geogrid.exe (No such file or directory)
[proxy:0:0@d10s0b2.arc4.leeds.ac.uk] HYD_spawn (../../../../../src/pm/i_hydra/libhydra/spawn/intel/hydra_spawn.c:117): execvp error on file metgrid.exe (No such file or directory)
rm: cannot remove ‘FILE*’: No such file or directory
[proxy:0:0@d10s0b2.arc4.leeds.ac.uk] HYD_spawn (../../../../../src/pm/i_hydra/libhydra/spawn/intel/hydra_spawn.c:117): execvp error on file real.exe (No such file or directory)
mv: cannot stat ‘rsl*’: No such file or directory
/var/spool/sge_prod/d10s0b2/job_scripts/661976: line 116: megan_bio_emiss: command not found
mv: cannot stat ‘wrfbiochemi_d01’: No such file or directory
ncatted: ERROR file /nobackup/chmltf/HONO_13/run/wrfbiochemi_d01_WRFChem3.7.1 not found. It does not exist on the local filesystem, nor does it match remote filename patterns (e.g., http://foo or foo.bar.edu:file).
ncatted: HINT file-not-found errors usually arise from filename typos, incorrect paths, missing files, or capricious gods. Please verify spelling and location of requested file. If the file resides on a High Performance Storage System (HPSS) accessible via the 'hsi' command, then add the --hpss option and re-try command.
ncks: ERROR file wrfinput_d01 not found. It does not exist on the local filesystem, nor does it match remote filename patterns (e.g., http://foo or foo.bar.edu:file).
ncks: HINT file-not-found errors usually arise from filename typos, incorrect paths, missing files, or capricious gods. Please verify spelling and location of requested file. If the file resides on a High Performance Storage System (HPSS) accessible via the 'hsi' command, then add the --hpss option and re-try command.
/var/spool/sge_prod/d10s0b2/job_scripts/661976: line 138: wesely: command not found
/var/spool/sge_prod/d10s0b2/job_scripts/661976: line 140: exo_coldens: command not found
/var/spool/sge_prod/d10s0b2/job_scripts/661976: line 149: anthro_emis: command not found
lukeconibear commented 4 years ago

Ah I think that error is because ungrib.exe is not being executed because on line 66 of pre.bash there is no ./.

# change
ungrib.exe > ungrib.log

# to 
./ungrib.exe > ungrib.log

For the CEMAC method, these were all removed. For the manual method, the ./ would also be needed in pre.bash for all of the other executables too: geogrid.exe on line 71 metgrid.exe on line 74 real.exe on line 91 megan_bio_emiss on line 116 wesely on line 138 exo_coldens on line 140 anthro_emiss on line 149 fire_emis on line 164 real.exe on line 174 mozbc on line 217 and 222

Then also in main.bash: wrfmeteo.exe on line 34 wrf.exe on line 105

I've added pre.bash.blueprint_manual, main.bash.blueprint_manual, config.bash.blueprint_manual, and main_restart.bash.blueprint_manual with these differences to the repository for reference. I'll also update the user guide.

Sorry for the issues here. I was unaware that the CEMAC compilation has quite so many changes relative to the manual compilation.

chmltf commented 4 years ago

This helped, thanks. Although it is still having a problem finding a linked library for megan_bio_emis. Here is the pre.bas.e and the ldd megan_bio_emiss. pre.bash.e:

Name of source model =>NCEP GFS Analysis       GRID   4
 Name of source model =>NCEP GFS Analysis       GRID   4
 Name of source model =>NCEP GFS Analysis       GRID   4
 Name of source model =>NCEP GFS Analysis       GRID   4
 Name of source model =>NCEP GFS Analysis       GRID   4
 Name of source model =>NCEP GFS Analysis       GRID   4
 Name of source model =>NCEP GFS Analysis       GRID   4
 Name of source model =>NCEP GFS Analysis       GRID   4
 Name of source model =>NCEP GFS Analysis       GRID   4
 Name of source model =>NCEP GFS Analysis       GRID   4
 Name of source model =>NCEP GFS Analysis       GRID   4
 starting wrf task            0  of            1
./megan_bio_emiss: error while loading shared libraries: libnetcdf.so.11: cannot open shared object file: No such file or directory
mv: cannot stat ‘wrfbiochemi_d01’: No such file or directory
ncatted: ERROR file /nobackup/chmltf/HONO_13/run/wrfbiochemi_d01_WRFChem3.7.1 not found. It does not exist on the local filesystem, nor does it match remote filename patterns (e.g., http://foo or foo.bar.edu:file).
ncatted: HINT file-not-found errors usually arise from filename typos, incorrect paths, missing files, or capricious gods. Please verify spelling and location of requested file. If the file resides on a High Performance Storage System (HPSS) accessible via the 'hsi' command, then add the --hpss option and re-try command.
./wesely: error while loading shared libraries: libnetcdf.so.11: cannot open shared object file: No such file or directory
./exo_coldens: error while loading shared libraries: libnetcdf.so.11: cannot open shared object file: No such file or directory
/var/spool/sge_prod/d8s3b1/job_scripts/662520: line 149: ./anthro_emis: No such file or directory
ln: failed to create symbolic link ‘./anthro_emis.inp’: File exists
Traceback (most recent call last):
  File "sum_sector_emiss_wrfchemi.py", line 22, in <module>
    open_ds = xr.open_dataset(cwd + '/' + file)
  File "/nobackup/cemac/software/apps/wrfchemconda/2.7/1/default/lib/python2.7/site-packages/xarray/backends/api.py", line 321, in open_dataset
    filename_or_obj, group=group, lock=lock, **backend_kwargs)
  File "/nobackup/cemac/software/apps/wrfchemconda/2.7/1/default/lib/python2.7/site-packages/xarray/backends/netCDF4_.py", line 355, in open
    return cls(manager, lock=lock, autoclose=autoclose)
  File "/nobackup/cemac/software/apps/wrfchemconda/2.7/1/default/lib/python2.7/site-packages/xarray/backends/netCDF4_.py", line 314, in __init__
    self.format = self.ds.data_model
  File "/nobackup/cemac/software/apps/wrfchemconda/2.7/1/default/lib/python2.7/site-packages/xarray/backends/netCDF4_.py", line 359, in ds
    return self._manager.acquire().value
  File "/nobackup/cemac/software/apps/wrfchemconda/2.7/1/default/lib/python2.7/site-packages/xarray/backends/file_manager.py", line 172, in acquire
    file = self._opener(*self._args, **kwargs)
  File "/nobackup/cemac/software/apps/wrfchemconda/2.7/1/default/lib/python2.7/site-packages/xarray/backends/netCDF4_.py", line 247, in _open_netcdf4_group
    ds = nc4.Dataset(filename, mode=mode, **kwargs)
  File "netCDF4/_netCDF4.pyx", line 2135, in netCDF4._netCDF4.Dataset.__init__
  File "netCDF4/_netCDF4.pyx", line 1752, in netCDF4._netCDF4._ensure_nc_success
IOError: [Errno 2] No such file or directory: '/nobackup/chmltf/HONO_13/run/base/2014-01-02_12:00:00-2014-01-05_00:00:00/wrfchemi_00z_d01'
/var/spool/sge_prod/d8s3b1/job_scripts/662520: line 164: ./fire_emis: No such file or directory
 starting wrf task            0  of            1
/var/spool/sge_prod/d8s3b1/job_scripts/662520: line 217: ./mozbc: No such file or directory
cp: cannot stat ‘/nobackup/chmltf/HONO_13/restart/base/wrfrst_d01_2014-01-03_00:00:00’: No such file or directory
ldd megan_bio_emiss
        linux-vdso.so.1 =>  (0x00007ffee2196000)
        libnetcdf.so.11 => not found
        libnetcdff.so.6 => /apps/developers/libraries/netcdf/4.6.3/1/intel-19.0.4-intelmpi-2019.4.243/lib/libnetcdff.so.6 (0x00007f363ba74000)
        libm.so.6 => /lib64/libm.so.6 (0x00007f363b772000)
        libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f363b556000)
        libc.so.6 => /lib64/libc.so.6 (0x00007f363b189000)
        libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f363af73000)
        libdl.so.2 => /lib64/libdl.so.2 (0x00007f363ad6f000)
        libnetcdf.so.15 => /apps/developers/libraries/netcdf/4.6.3/1/intel-19.0.4-intelmpi-2019.4.243/lib/libnetcdf.so.15 (0x00007f363a850000)
        libhdf5_hl.so.10 => /apps/developers/libraries/hdf5/1.8.21/1/intel-19.0.4-intelmpi-2019.4.243/lib/libhdf5_hl.so.10 (0x00007f363a629000)
        libhdf5.so.10 => /apps/developers/libraries/hdf5/1.8.21/1/intel-19.0.4-intelmpi-2019.4.243/lib/libhdf5.so.10 (0x00007f363a059000)
        libz.so.1 => /lib64/libz.so.1 (0x00007f3639e43000)
        libcurl.so.4 => /lib64/libcurl.so.4 (0x00007f3639bda000)
        libifport.so.5 => /apps/developers/compilers/intel/19.0.4/1/default/compilers_and_libraries_2019.4.243/linux/compiler/lib/intel64_lin/libifport.so.5 (0x00007f36399ac000)
        libifcoremt.so.5 => /apps/developers/compilers/intel/19.0.4/1/default/compilers_and_libraries_2019.4.243/linux/compiler/lib/intel64_lin/libifcoremt.so.5 (0x00007f3639617000)
        libimf.so => /apps/developers/compilers/intel/19.0.4/1/default/compilers_and_libraries_2019.4.243/linux/compiler/lib/intel64_lin/libimf.so (0x00007f3639077000)
        libsvml.so => /apps/developers/compilers/intel/19.0.4/1/default/compilers_and_libraries_2019.4.243/linux/compiler/lib/intel64_lin/libsvml.so (0x00007f36376d3000)
        libintlc.so.5 => /apps/developers/compilers/intel/19.0.4/1/default/compilers_and_libraries_2019.4.243/linux/compiler/lib/intel64_lin/libintlc.so.5 (0x00007f3637461000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f363c031000)
        libmpifort.so.12 => /apps/developers/libraries/intelmpi/2019.4.243/1/intel-19.0.4/lib/libmpifort.so.12 (0x00007f36370a2000)
        libmpi.so.12 => /apps/developers/libraries/intelmpi/2019.4.243/1/intel-19.0.4/lib/release/libmpi.so.12 (0x00007f36361b0000)
        librt.so.1 => /lib64/librt.so.1 (0x00007f3635fa8000)
        libirng.so => /apps/developers/compilers/intel/19.0.4/1/default/compilers_and_libraries_2019.4.243/linux/compiler/lib/intel64_lin/libirng.so (0x00007f3635c36000)
        libidn.so.11 => /lib64/libidn.so.11 (0x00007f3635a03000)
        libssh2.so.1 => /lib64/libssh2.so.1 (0x00007f36357d9000)
        libssl3.so => /lib64/libssl3.so (0x00007f3635587000)
        libsmime3.so => /lib64/libsmime3.so (0x00007f3635360000)
        libnss3.so => /lib64/libnss3.so (0x00007f3635033000)
        libnssutil3.so => /lib64/libnssutil3.so (0x00007f3634e03000)
        libplds4.so => /lib64/libplds4.so (0x00007f3634bff000)
        libplc4.so => /lib64/libplc4.so (0x00007f36349fa000)
        libnspr4.so => /lib64/libnspr4.so (0x00007f36347bc000)
        libgssapi_krb5.so.2 => /lib64/libgssapi_krb5.so.2 (0x00007f363456f000)
        libkrb5.so.3 => /lib64/libkrb5.so.3 (0x00007f3634286000)
        libk5crypto.so.3 => /lib64/libk5crypto.so.3 (0x00007f3634053000)
        libcom_err.so.2 => /lib64/libcom_err.so.2 (0x00007f3633e4f000)
        liblber-2.4.so.2 => /lib64/liblber-2.4.so.2 (0x00007f3633c40000)
        libldap-2.4.so.2 => /lib64/libldap-2.4.so.2 (0x00007f36339eb000)
        libfabric.so.1 => /apps/developers/libraries/intelmpi/2019.4.243/1/intel-19.0.4/libfabric/lib/libfabric.so.1 (0x00007f36337b3000)
        libssl.so.10 => /lib64/libssl.so.10 (0x00007f3633541000)
        libcrypto.so.10 => /lib64/libcrypto.so.10 (0x00007f36330df000)
        libkrb5support.so.0 => /lib64/libkrb5support.so.0 (0x00007f3632ecf000)
        libkeyutils.so.1 => /lib64/libkeyutils.so.1 (0x00007f3632ccb000)
        libresolv.so.2 => /lib64/libresolv.so.2 (0x00007f3632ab2000)
        libsasl2.so.3 => /lib64/libsasl2.so.3 (0x00007f3632895000)
        libselinux.so.1 => /lib64/libselinux.so.1 (0x00007f363266e000)
        libcrypt.so.1 => /lib64/libcrypt.so.1 (0x00007f3632437000)
        libpcre.so.1 => /lib64/libpcre.so.1 (0x00007f36321d5000)
        libfreebl3.so => /lib64/libfreebl3.so (0x00007f3631fd2000)
lukeconibear commented 4 years ago

Okay, that's progress.

So pre.bash now first breaks at ./megan_bio_emiss: error while loading shared libraries: libnetcdf.so.11 and we can see that for megan_bio_emiss libnetcdf.so.11 => not found.

Comparing this with my manual version of megan_bio_emiss where I also use the modules intel/19.0.4, intelmpi/2019.4.243, and netcdf/4.6.3, mine is using libnetcdf.so.15 instead of libnetcdf.so.11:

# ldd on your megan_bio_emiss
libnetcdf.so.11 => not found

# ldd on my megan_bio_emiss
libnetcdf.so.15 => /apps/developers/libraries/netcdf/4.6.3/1/intel-19.0.4-intelmpi-2019.4.243/lib/libnetcdf.so.15 (0x00007f3c4523c000)

I don't think anything is wrong with the modules you have loaded in your environment.

I would try recompiling megan_bio_emiss in a separate clean shell, using:

conda deactivate
module purge
module load intel netcdf
export NETCDF=$(nc-config --prefix)
export NETCDF_DIR=$NETCDF
export FC=ifort

./make_util megan_bio_emiss

I've just tested this now and it worked.

I noticed I made a small mistake in the user guide beforehand regarding the environment variables here, but I've now corrected this.

chmltf commented 4 years ago

This worked! I had to do this for megan_bio_emis, and the analogous make_util for anthro_emis, fire_emis, wesley, exo_coldens, and mozbc.

But there was one other piece. In the pre.bash I had to add the following: Line 153 cp -r ${WRFanthrodir}/anthro_emis . Line 170 cp -r ${WRFfiredir}/fire_emis . Line 192 cp -r ${WRFMOZARTdir}/mozbc .

lukeconibear commented 4 years ago

Great, I'm glad that this solved the problem. Thanks for that additional information of what is needed, I've added it to the user guide and the blueprint files.

chmltf commented 4 years ago

Thank you for your help on this.