search

geoschem / GCHP

The "superproject" wrapper repository for GCHP, the high-performance instance of the GEOS-Chem chemical-transport model.
https://gchp.readthedocs.io
Other
22 stars 25 forks source link

[BUG/ISSUE] GCHP v13.3.2 crashing when using GFAS biomass burning inventory #284

Closed Twize closed 1 year ago

Twize commented 1 year ago

Hi, I am working with Dylan Jones at the University of Toronto and I am trying to configure a GCHP run which uses the GFASv1.2 biomass burning inventory instead of GFED4 to compare against a run that I have done using GEOS-Chem Classic.

Description of the problem

When I turn off GFED4, and turn on GFAS instead, I get the following error after the gchp executable is run: + mpirun -np 120 ./gchp At line 156 of file /scratch/d/dylan/tylerw/GCHP_v13.3.2_full/src/MAPL/base/FileMetadataUtilities.F90 Fortran runtime error: Bad integer for item 1 in list input

Going to that line in FileMetadataUtilities.F90 seems to refer to MAPL reading the seconds time info from the GFAS files. I have switched on debugging prompts for HEMCO and increased verbose to 3 in HEMCO_Config.rc. HEMCO.log doesn't seem to include any errors or relating to GFAS from what I can see. Looking at allPEs.log, it seems to find the GFAS file correctly, but then the file ends at "Retrieving formatter for: ./HcoDir/GFAS/v2018-09/2017/GFAS_201708.nc":

0000: CAP.EXTDATA: DEBUG: Updating bracket for GFAS_CO 0000: CAP.EXTDATA: DEBUG: UpdateBracketTime: Scanning template for side L 0000: CAP.EXTDATA: DEBUG: UpdateBracketTime: Target time : 2017-08-01 00:00:00 0000: CAP.EXTDATA: DEBUG: UpdateBracketTime: Reference time: 1985-01-01 00:00:00 0000: CAP.EXTDATA: DEBUG: UpdateBracketTime: untemplating ./HcoDir/GFAS/v2018-09/%y4/GFAS_%y4%m2.nc 0000: CAP.EXTDATA: DEBUG: ==> Target time : 2017-08-01 00:00:00 0000: CAP.EXTDATA: DEBUG: ==> Target time : 2017-08-01 00:00:00 0000: CAP.EXTDATA: DEBUG: ===> item%frequency: Reference time: 0000-01-00 00:00:00 0000: CAP.EXTDATA: DEBUG: ===> # iterations until found 391 0000: CAP.EXTDATA: DEBUG: Target file for found and is ./HcoDir/GFAS/v2018-09/2017/GFAS_201708.nc 0000: CAP.EXTDATA: DEBUG: UpdateBracketTime: Making metadata for ./HcoDir/GFAS/v2018-09/2017/GFAS_201708.nc 0000: CAP.EXTDATA: DEBUG: Retrieving formatter for: ./HcoDir/GFAS/v2018-09/2017/GFAS_201708.nc

I manually opened the GFAS_201708.nc file, and there doesn't seem to be anything weird with the timestamps in the file. I also compared it with the timestamps in a GFED4 daily file and they look to be formatted the same way. I copied the time formatter in Extdata.rc from the GFED4 daily field as the GFASv1.2 files are formatted very similarly (my Extdata.rc is attached below). An example line for GFAS from my Extdata.rc is included below:

GFAS_CO kg/m2/s N Y %y4-%m2-%d2T00:00:00 none none cofire ./HcoDir/GFAS/v2018-09/%y4/GFAS_%y4%m2.nc

I have seen from #183 that there are currently some issues with the GFAS injection heights (the mami values in the files) in GCHP, but this issue I am encountering does not appear to be the same one. At the moment I am just placing the emissions between the first level and the PBL height (xyL=1:PBL in HEMCO_Config.rc).

Any help would be appreciated!

GEOS-Chem version

GCHP v13.3.2

Log files

allPEs.log ExtData.rc.txt HEMCO_Config.rc.txt HEMCO.log.txt slurm_output_8772791.txt runConfig.sh.txt

Software versions

yantosca commented 1 year ago

Thanks for writing @Twize. This is a known issue in the MAPL library. We have a fix for this going into 14.1.0. See issue #251.

yantosca commented 1 year ago

Tagging @lizziel @SaptSinha @Jourdan-He

Twize commented 1 year ago

Hi @yantosca, thanks for the quick reply. I wasn't sure that this was the same issue, since the previous issue (#183) related specifically to the MAMI fields being read in which contained lots of fill values. In my case, I commented out the line to read the MAMI field in, so I'm not 100% convinced that it's the same issue with MAPL. The crash occurs for me before the GFAS file is fully read in to the model.

lizziel commented 1 year ago

Hi @twize, this is indeed a different issue than https://github.com/geoschem/GCHP/issues/251 since it is occurring in MAPL ExtData. The missing value issue would error out later in HEMCO due to incorrect values.

I checked the MAPL code in 13.2.1 and it looks like the error is occurring on the 3rd line of this code to read in the time dimension units:

             read (TimeUnits(hpos(1):hpos(2)), * ) hour
             read (TimeUnits(mpos(1):mpos(2)), * ) min
             read (TimeUnits(spos(1):spos(2)), * ) sec

I checked the time dimension units in the file GFAS_201708.nc and they are: time:units = "hours since 1970-01-01 00:00:0.0" ;

The problem is the 0.0 for seconds. It needs to be 00. I am not sure why my test runs with GFAS did not encounter this error during my development of the missing value fix. I will investigate that.

As an aside, I notice you are using gcc 9.2. We have had reports of issues with the MAPL error handling for gcc versions prior to v10. If you can update to 10+ then best do that.

Twize commented 1 year ago

Hi @lizziel, thanks for shining a bit more light on my issue! Its strange that I seem to be the first person to run into this particular issue when using GFAS. Is there any way for me to fix this by modifying the input format in Extdata.rc?

As for GCC 10+, I am unfortunately stuck with GCC 9.2.0/9.3.0 as they are the latest versions installed on Scinet Niagara. I may be able to reach out and ask if they would be able to install GCC 10+ for me, but not sure if they would be willing. I briefly mentioned this in my last comment on #270. I tried installing all of the GCHP pre-requisites from scratch, but in the end, the only thing that worked for me was using the pre-installed gcc 9.2.0 + OpenMPI 4.0.3 installations to compile the other requirements via Spack.

lizziel commented 1 year ago

Hi @Twize, I think not many people use GFAS with GCHP hence you are the first to report this. I figured out I did not have a problem in my testing because June and July 2019 already have the fix on our filesystem. I am not sure why that did not propagate to the rest of the files.

To fix this locally you can run the following in a bash script:

datadir=/path/to/your/ExtData/HEMCO/GFAS/v2018-09
for year in {2003..2022}; do
    for month in 01 02 03 04 05 06 07 08 09 10 11 12; do
    datafile=${datadir}/${year}/GFAS_${year}${month}.nc
        echo ${datafile}
    ncatted -O -h -a units,time,o,c,'hours since 1970-01-01 00:00:00' $datafile
    done
done
Twize commented 1 year ago

Hi @lizziel, thanks this is extremely helpful (I was just looking into ways that I could modify the reference time in the .nc files myself). I will give this a go and report back.

yantosca commented 1 year ago

@Twize, we also have some helpful info on the new GEOS-Chem ReadTheDocs pages. See:

  1. https://geos-chem.readthedocs.io/en/stable/geos-chem-shared-docs/supplemental-guides/netcdf-guide.html

  2. https://geos-chem.readthedocs.io/en/stable/geos-chem-shared-docs/supplemental-guides/coards-guide.html

yantosca commented 1 year ago

We also have a new repository with updated versions of isCoards and nc_chunk.pl. scripts:

https://github.com/geoschem/netcdf-scripts

You can use isCoards to check if any attributes in the file would render it unreadable by GCHP.

Twize commented 1 year ago

@yantosca thanks, that will be quite useful going forwards for me!

Twize commented 1 year ago

Hi @lizziel, your script worked for me and I was able to run a quick simulation using GFAS (with fixed injection heights). However, if I try to read in the MAMI values, I run into the same issue as in #183 (so at least that is consistent...).

I guess I may have to wait until the release of v14.1.0 if I want to use the GFAS injection heights 🙂 I am going to go ahead and close the issue since I think it has been solved.

lizziel commented 1 year ago

@Twize, yes I also ran into a problem when using GFAS with mami values, even with our dev version of 14.1.0. I am looking into a fix for this which will likely go into 14.1.1.

kilicomu commented 1 year ago

@Twize @lizziel I'm in the process of reworking the GFAS preprocessing - ECMWF have switched to a new data store, API, and Python package 😞 - so I'll take this into account when I release a new version of the GFAS data!

Twize commented 1 year ago

@lizziel @kilicomu thanks to you both for the updates and for keeping me in the loop! A lot of my recent work has been looking at the 2017 Canadian wildfires, and GFAS seems to do a much better job at the timing and scale of the emissions versus GFED4 when compared against IASI observations, so I am hoping to stick with GFAS going forwards.

I assume v14.1.0 will likely be released quite soon, will v14.1.1 will be deployed at the same time?

lizziel commented 1 year ago

@kilicomu, @Twize: I completely forgot we have this issue open in HEMCO: https://github.com/geoschem/HEMCO/issues/173. GFAS_pFe is failing for a yet-to-be-identified reason. Turning off that field gets around the issue.

lizziel commented 1 year ago

See also the fix at https://github.com/geoschem/HEMCO/issues/173 which allows you to still use GFAS_pFe.

Twize commented 1 year ago

@lizziel wait, so the GFAS_pFe field is causing the issue that is described in #183? Or is that an additional issue on top of the fill values in the GFAS mami fields ?

Edit: Had a closer look, guess that is an additional issue (although I don't really use the GFAS_pFe fields, so the fix would work fine for me)