NEXUS failed for both retro and NRT runs

JianpingHuang-NOAA commented 1 year ago

Description:

NEXUS failed for the NRT runs, then I checked out the latest online-cmaq workflow last night and still seeing failures of running NEXUS

1) retro example on Dogwood /lfs/h2/emc/ptmp/jianping.huang/emc.para/output/20221110 nexus_emission_2022111006_s00.id_1671021056.log nexus_emission_2022111006_s01.id_1671021056.log nexus_emission_2022111006_s02.id_1671021056.log

2) NRT runs /lfs/h2/emc/ptmp/jianping.huang/emc.para/output/20221213 nexus_emission_2022121312_s01.id_1670965539.log nexus_emission_2022121312_s00.id_1670965539.log

Please fix the issue as soon as possible. @bbakernoaa

Thanks !

chan-hoo commented 1 year ago

@JianpingHuang-NOAA, We can't access your 'data' directory. dlogin01:/lfs/h2/emc/physics/noscrub/jianping.huang> cd data/ -bash: cd: data/: Permission denied drwxr-sr-- 13 jianping.huang physics 4096 Dec 10 02:06 data. Please change its permission. It should be "drwxr-sr-x". Please run " chmod -R o+x data ". In addition, the latest hash was not tested completely due to the above data access issue. Please check out the hash 'e769b69'.

JianpingHuang-NOAA commented 1 year ago

@chan-hoo Thanks. The permission was done. What does the hash 'e769b69' represent?

bbakernoaa commented 1 year ago

@JianpingHuang-NOAA I ran the 2022-12-13 12:00 case on dogwood and didn't have an issue.

12/14/22 14:58:43 UTC :: FV3LAM_wflow.xml :: WARNING: <nodesize> support is deprecated, please use <nodes> to specify the requested resources
       CYCLE                    TASK                       JOBID               STATE         EXIT STATUS     TRIES      DURATION
================================================================================================================================
202212131200       nexus_emission_00                    33033172           SUCCEEDED                   0         1         979.0
202212131200       nexus_emission_01                    33033176           SUCCEEDED                   0         1         986.0
202212131200       nexus_emission_02                    33033173           SUCCEEDED                   0         1         973.0
202212131200        nexus_post_split    druby://dlogin02.dogwood.wcoss2.ncep.noaa.gov:35839          SUBMITTING                   -         0           0.0
202212131200           fire_emission                    33033174              QUEUED                   -         0           0.0
202212131200            point_source                    33033175           SUCCEEDED                   0         1         713.0
202212131200           get_extrn_ics                    33033177              QUEUED                   -         0           0.0
202212131200          get_extrn_lbcs                    33033178              QUEUED                   -         0           0.0
202212131200                make_ics                           -                   -                   -         -             -
202212131200               make_lbcs                           -                   -                   -         -             -
202212131200                aqm_lbcs                           -                   -                   -         -             -
202212131200                run_fcst                           -                   -                   -         -             -
202212131200           pre_post_stat                           -                   -                   -         -             -
202212131200            post_stat_o3                           -                   -                   -         -             -
202212131200          post_stat_pm25                           -                   -                   -         -             -

I haven't been able to use your emission directory due to permission issues and so I had copied over the emission data needed to run AQM yesterday here: /lfs/h2/emc/lam/noscrub/barry.baker/emissions/nexus

Again I didn't see any issues running this day pointing to my emission directory. You can see the output here: /lfs/h2/emc/ptmp/barry.baker/expt_dirs/aqm20221213

JianpingHuang-NOAA commented 1 year ago

@bbakernoaa I got a failure too when I pointed my script to your emission dir.

Here is the run log file (Dogwood)

/lfs/h2/emc/ptmp/jianping.huang/emc.para/output/20221109 > vim nexus_emission_2022110906_s02.id_1671030435.log

Can you take a look?

In addition, my nexus is located at /lfs/h2/emc/physics/noscrub/jianping.huang/nwdev/packages/aqm.v7.0.21/sorc/arl_nexus

"git log" shows commit 5c1fa371c76a72d3d263cd27aabead914f397cc0 Merge: 8b5ef7e fe5778a Author: Barry Baker bbakernoaa@users.noreply.github.com Date: Tue Nov 29 15:59:42 2022 -0500

Merge pull request #21 from noaa-oar-arl/feature/nexus_split

change from xarray to netcdf4-python for all machines

commit fe5778a0464bac8a60d140e72b9279b846e97621

Is this the right version?

Thanks,

JianpingHuang-NOAA commented 1 year ago

@chan-hoo @bbakernoaa NEXUS ran successfully with Barry's data for the large domain on 20221213 at 00z cycle but failed to run with same data for the grid_spec_RRFS_CONUS_13km or G775. I changed the HEMCO_sa_Grid.rc to us the same grid numbers that we ran the model for CONUS, but still failed. Is there any change that I need to make in order to run NEXUS for the CONUS domain?

JianpingHuang-NOAA commented 1 year ago

The revised HEMCO_sa_Grid.rc is located on Dogwood at

/lfs/h2/emc/physics/noscrub/jianping.huang/nwdev/packages/aqm.v7.0.24/sorc/arl_nexus/config/cmaq

JianpingHuang-NOAA commented 1 year ago

The config.yaml file can be found at /lfs/h2/emc/physics/noscrub/jianping.huang/nwdev/packages/aqm.v7.0.24/ush (Dogwood)

bbakernoaa commented 1 year ago

We shouldn’t need to change anything for the conus domain. The nexus grid is the intermediate grid and not the final one. So having a larger intermediate grid won’t matter as long as the domain is small and completely contained within.

bbakernoaa commented 1 year ago

I believe this has been solved. We should close it

JianpingHuang-NOAA commented 1 year ago

Hi Barry,

Can you sync your emission data to Cactus?

Thanks,

Jianping

On Thu, Dec 15, 2022 at 9:02 AM Barry Baker @.***> wrote:

I believe this has been solved. We should close it

— Reply to this email directly, view it on GitHub https://github.com/NOAA-EMC/AQM/issues/50#issuecomment-1353139698, or unsubscribe https://github.com/notifications/unsubscribe-auth/ANA2PI5ZTB5D4JQZL67HUG3WNMQIVANCNFSM6AAAAAAS6OUAJI . You are receiving this because you were mentioned.Message ID: @.***>

bbakernoaa commented 1 year ago

I am but it may take a while

Barry Baker

National Oceanic and Atmospheric Administration Air Resources Laboratory Physical Research Scientist Chemical Modeling and Emissions Group Leader NCWCP, R/ARL, Rm. 4204 5830 University Research Court College Park, Maryland 20740 Phone: ‪(301) 683-1395‬

On Thu, Dec 15, 2022 at 9:09 AM JianpingHuang-NOAA @.***> wrote:

Hi Barry,

Can you sync your emission data to Cactus?

Thanks,

Jianping

On Thu, Dec 15, 2022 at 9:02 AM Barry Baker @.***> wrote:

I believe this has been solved. We should close it

— Reply to this email directly, view it on GitHub https://github.com/NOAA-EMC/AQM/issues/50#issuecomment-1353139698, or unsubscribe < https://github.com/notifications/unsubscribe-auth/ANA2PI5ZTB5D4JQZL67HUG3WNMQIVANCNFSM6AAAAAAS6OUAJI

. You are receiving this because you were mentioned.Message ID: @.***>

— Reply to this email directly, view it on GitHub https://github.com/NOAA-EMC/AQM/issues/50#issuecomment-1353151934, or unsubscribe https://github.com/notifications/unsubscribe-auth/AFIUVNZZSKNND7SJHK3XW7DWNMRB7ANCNFSM6AAAAAAS6OUAJI . You are receiving this because you were mentioned.Message ID: @.***>

bbakernoaa commented 1 year ago

can we close this now?

NOAA-EMC / AQM

NEXUS failed for both retro and NRT runs #50

Description: