Error running case.setup on hera

[JulieSchramm]

Following the instructions in the Quickstart chapter:

git clone -b ufs-v1.0.0.alpha02 https://github.com/ufs-community/ufs-mrweather-app.git \ my_ufs_sandbox cd my_ufs_sandbox/ ./manage_externals/checkout_externals cd cime/scripts/ ./create_newcase --case /scratch1/BMC/gmtb/$USER/cases/ufs-mrweather-app-workflow.c96 -- \ compset GFSv15p2 --res C96 --workflow ufs-mrweather --project gmtb cd /scratch1/BMC/gmtb/$USER/cases/ufs-mrweather-app-workflow.c96 ./case.setup

produces the error message:

ERROR: Could not make directory '/scratch1/NCEPDEV/nems/Julie.Schramm/ufs-mrweather-app-workflow.c96/bld/atm/obj', error: [Errno 13] Permission denied: '/scratch1/NCEPDEV/nems/Julie.Schramm'

Is /scratch1/NCEPDEV/nems/$USER set correctly?

[arunchawla-NOAA]

@JulieSchramm do you have write access to /scratch1/NCEPDEV/nems ?

[rsdunlapiv]

@JulieSchramm You should please edit cime/config/ufs/machines/config_machines.xml and change CIME_OUTPUT_ROOT under the "Hera" entry to point to a folder that exists and where you have write access. Then please re-create the case and give it a try.

[rsdunlapiv]

@arunchawla-NOAA maybe there is a known scratch place on Hera that will work for ALL users? If so, we could change the default. Should it be under stmp?

[arunchawla-NOAA]

stmp space should work, but we need some people to try. Since we have access it is hard to know what is working and what is not

[rsdunlapiv]

@JulieSchramm can you please try setting it to stmp space?

[JulieSchramm]

@rsdunlapiv If I set

/scratch1/BMC/gmtb/$USER

the case.setup step works. From what I can tell on Hera, there is no common scratch or stmp space where all users have write permission. The disk space seems to be divided up by projects. With this mod,

./case.build works ./case.submit works

and the ufs-mrweather-app is running.

[jedwards4b]

Are you saying that not all hera users have a directory /scratch1/BMC/gmtb/$USER? Is there a systemwide $SCRATCH definition set to a users scratch space?

If there is no common scratch space then I would set

$ENV{SCRATCH}

which will require each user to define $SCRATCH in thier environment.

On Wed, Feb 26, 2020 at 8:31 AM JulieSchramm notifications@github.com wrote:

@rsdunlapiv https://github.com/rsdunlapiv If I set

/scratch1/BMC/gmtb/$USER

the case.setup step works. From what I can tell on Hera, there is no common scratch or stmp space where all users have write permission. The disk space seems to be divided up by projects. With this mod,

./case.build works ./case.submit works

and the ufs-mrweather-app is running.

— You are receiving this because you were assigned. Reply to this email directly, view it on GitHub https://github.com/ufs-community/ufs-mrweather-app/issues/101?email_source=notifications&email_token=ABOXUGEZZBJ6GGIQ25PAOLDRE2DLPA5CNFSM4K3VPVLKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOENAVY5Y#issuecomment-591486071, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABOXUGGF524OXMO2QJ5KT33RE2DLPANCNFSM4K3VPVLA .

-- Jim Edwards

CESM Software Engineer National Center for Atmospheric Research Boulder, CO

[rsdunlapiv]

@jedwards4b is there any error checking in place in case $SCRATCH is not set? Would the user get a message saying it needs to be set?

[JulieSchramm]

@jedwards4b Yes, BMC is a larger group that I am in, then gmtb is the project/account that I have access to. I am pretty sure there is no common $SCRATCH area on Hera.

@climbfuji Can you verify?

[climbfuji]

scratch space on hera is on a user-and-project basis. Users get scratch space underneath the projects they are members of. You cannot assume that any directory you are specifying will work for all. What you can do is have the user set an environment variable that defines $SCRATCH and use this in the CIME config.

[climbfuji]

Same on jet (and probably gaea, need to check), by the way.

[rsdunlapiv]

Makes sense, but just want to make sure that there is a nice error message if the user fails to do this. It is something that is easy to miss.

[climbfuji]

At least EMC and other NOAA users are used to this - we always need to set/correct the scratch directory and also the project name for the scheduler.

[jedwards4b]

Yes - if any environment variable is expected but cannot be resolved you do get a nice error message, for examples try running on the macos, linux, or stampede systems. Here is an example from stampede:

./create_newcase --case foo --compset GFSv15p2 --res C96 Compset longname is FCST_ufsatm%v15p2_SLND_SICE_SOCN_SROF_SGLC_SWAV Compset specification file is /home1/02503/edwardsj/ufs-mrweather-app/src/model/FV3/cime/cime_config/config_compsets.xml Automatically adding SESP to compset Compset forcing is ATM component is UFSATM Atmosphere with:CCPP physics version 15p2 LND component is Stub land component ICE component is Stub ice component OCN component is Stub ocn component ROF component is Stub river component GLC component is Stub glacier (land ice) component WAV component is Stub wave component ESP component is Stub external system processing (ESP) component Pes specification file is /home1/02503/edwardsj/ufs-mrweather-app/src/model/FV3/cime/cime_config/config_pes.xml Compset specific settings: name is RUN_STARTDATE and value is 2019-08-29 Compset specific settings: name is START_TOD and value is 0 Compset specific settings: name is COMP_CLASSES and value is ATM Compset specific settings: name is CHECK_TIMING and value is FALSE Machine is stampede2-skx Pes setting: grid match is a%C96 Pes setting: compset_match is ufsatm Pes setting: grid is a%C96_l%null_oi%null_r%null_g%null_w%null_z%null_m%null Pes setting: compset is FCST_ufsatm%v15p2_SLND_SICE_SOCN_SROF_SGLC_SWAV_SESP Pes setting: tasks is {'NTASKS_ATM': 108} Pes setting: threads is {'NTHRDS_ATM': 1} Pes setting: rootpe is {} Pes setting: pstrid is {} Pes other settings: {} Pes comments: none Compset is: FCST_ufsatm%v15p2_SLND_SICE_SOCN_SROF_SGLC_SWAV_SESP Grid is: a%C96_l%null_oi%null_r%null_g%null_w%null_z%null_m%null Components in compset are: ['ufsatm', 'slnd', 'sice', 'socn', 'srof', 'sglc', 'swav', 'sesp'] Using project from config_machines.xml: TG-ATM190009 No charge_account info available, using value from PROJECT

Directory /scratch/02503/edwardsj/foo/bld already exists, (r)eplace, (a)bort, or (u)se existing?r

Directory /scratch/02503/edwardsj/foo/run already exists, (r)eplace, (a)bort, or (u)se existing?r ufs model version found: 1e0a3a7 Batch_system_type is slurm job is case.run USER_REQUESTED_WALLTIME None USER_REQUESTED_QUEUE None job is case.st_archive USER_REQUESTED_WALLTIME None USER_REQUESTED_QUEUE None Creating Case directory /home1/02503/edwardsj/ufs-mrweather-app/cime/scripts/foo edwardsj@~/ufs-mrweather-app/cime/scripts((HEAD detached at 5e4b070))$ cd foo edwardsj@~/ufs-mrweather-app/cime/scripts/foo((HEAD detached at 5e4b070))$ ./case.setup ERROR: Undefined env var 'UFS_INPUT'

On Wed, Feb 26, 2020 at 8:44 AM Dom Heinzeller notifications@github.com wrote:

At least EMC and other NOAA users are used to this - we always need to set/correct the scratch directory and also the project name for the scheduler.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ufs-community/ufs-mrweather-app/issues/101?email_source=notifications&email_token=ABOXUGG6XQSPF343FAA2VHDRE2E7FA5CNFSM4K3VPVLKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOENAXSMQ#issuecomment-591493426, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABOXUGCG7V4YN52YE6C3AUTRE2E7FANCNFSM4K3VPVLA .

-- Jim Edwards

CESM Software Engineer National Center for Atmospheric Research Boulder, CO

[JulieSchramm]

Yes, your error message is nicer. Your variable was "undefined", mine was defined but erroneous :)

[rsdunlapiv]

Okay, that is great. In that case I really like the $SCRATCH option, and we need to add that to the quick start guide.

• Should it be $UFS_SCRATCH or just $SCRATCH?
• In the docs, how do we indicate for which platforms this setting is required? E.g., it's not needed on Cheyenne because of the global scratch space, but will be needed on NOAA machines.

[jedwards4b]

Since we called it UFS_SCRATCH on macos and linux definitions it makes sense to do the same here.

On Wed, Feb 26, 2020 at 8:56 AM Rocky Dunlap notifications@github.com wrote:

Okay, that is great. In that case I really like the $SCRATCH option, and we need to add that to the quick start guide.

• Should it be $UFS_SCRATCH or just $SCRATCH?
• In the docs, how do we indicate for which platforms this setting is required? E.g., it's not needed on Cheyenne because of the global scratch space, but will be needed on NOAA machines.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ufs-community/ufs-mrweather-app/issues/101?email_source=notifications&email_token=ABOXUGG3UGNHO2BKYKXGB4DRE2GKLA5CNFSM4K3VPVLKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOENAZFHY#issuecomment-591499935, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABOXUGFFX73I6D5K27RCFWTRE2GKLANCNFSM4K3VPVLA .

-- Jim Edwards

CESM Software Engineer National Center for Atmospheric Research Boulder, CO

[arunchawla-NOAA]

@JulieSchramm is this working now? Do you want to close this ticket?

[JulieSchramm]

So should a user on Hera set the UFS_SCRATCH variable? If so, how? In the documentation, this variable is only briefly mentioned in an introductory paragraph in the quickstart section:

On some systems users may also need to set environment variables to identify model input and scratch directories - these are $UFS_INPUT and $UFS_SCRATCH.

[jedwards4b]

UFS_SCRATCH should be set to a directory where the user has write permission and sufficient space to stage model output data.