[BUG/ISSUE] Known bug in 12.7-13.0: Incorrect timestamps in OFFLINE_BIOVOC/v2019-10 crash simulations

[LiamBindle]

Description of the problem

The OFFLINE_BIOVOC/v2019-10 inputs have incorrect timestamps. The affected files are

• biovoc_05.YYYY0101.nc
• biovoc_05.YYYY1229.nc
• biovoc_05.YYYY1230.nc
• biovoc_05.YYYY1231.nc

and this affects all years up to (and including) 2019-01-01. This causes GCHP to crash because the filename's timestamp does not match the "time" variable's value.

Affected GEOS-Chem versions

This affects GCHP versions 12.7–13.0.

The fix

The fix is trivial: the "time" variable's "unit" attribute should be "hours since YYYY-MM-DD 00:00:00 GMT", where YYYY-MM-DD is corresponding date. You can do this manually with ncatted (NCO utility) or run this script on your incorrect files, e.g.:

$ ./fix-biovoc_5 biovoc_05.20151229.nc
+ ncatted -O -h -a 'units,time,o,c,hours since 2015-12-29 00:00:00' biovoc_05.20151229.nc

Root issue

• https://github.com/geoschem/geos-chem/issues/271

Notes

Thanks to @Haihui-Zhu for re-identifying the issue and helping test the fix.

[LiamBindle]

I think this issue should stay open until 13.2 release for visibility. I'll also add a new page to the GCHP docs for "Known bugs".

Edit: Here is the known bugs page: https://gchp.readthedocs.io/en/latest/reference/known-bugs.html

[yantosca]

One idea is to have a Python script parse the output from Xarray to make sure that each file is COARDS-compliant and GCHP-compliant. This should be easy to set up. (We already have isCoards but that is in Perl and would need to be modified, it might be simpler to just start over in Python). This could be a useful tool that we can use to check the format validity of input datasets.

[LiamBindle]

In the GCST call today we briefly talked about how to handle this. The plan is we recommend users run the script shared above, and we will not regenerate OFFLINE_BIOVOC/v2019-10 with corrected timestamps.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. If there are no updates within 7 days it will be closed. You can add the "never stale" tag to prevent the Stale bot from closing this issue.

[sdeastham]

@LiamBindle do we have a long-term fix planned? And is the short term fix listed as part of the documentation for running GCHP? I somehow keep tripping over this on new systems.

[lizziel]

@sdeastham, do you mean you are running into the problem use fresh data downloads from Compute Canada?

[sdeastham]

Indeed - for example, the 2014-12-31 file on Compute Canada still has a time stamp for 2014-12-28 (just downloaded to double check), so GCHP will crash when simulating that year.

[lizziel]

@LiamBindle, what is the status of this? The files here at Harvard were corrected last spring but it looks like Compute Canada still hasn't synced with them. The earlier post here says we will leave it the users to update the files. What was the rationale for this?

[LiamBindle]

I was under the impression there was going to be a new version of OFFLINE_BIOVOC very soon, so it wasn't worth duplicating the data to fix the timestamps.

Yeah, it looks like the inplace update of the files at Harvard last spring didn't make it onto Compute Canada. Is it easy to push those updates to Compute Canada?

If it isn't super easy, I could log in an run that ./fix-biovoc_5 script on the files on CCVM. I'm working from home for the next few days, and I can't access CCVM from my home computer, so it will be a few days before I can do this.

Sorry this keeps falling through the cracks.

[lizziel]

The new offline files will fix the issue but the old inventory will still be default in GCHP run directories for some versions. @YanshunLi-washu, could you sync the files? The inventory is listed in the new or updated data directories in 12.8.1 table.

[LiamBindle]

@lizziel I'm in the office today, so I can log into CCVM and run the ./fix-biovoc_5 script on all those files.

[LiamBindle]

The attributes in OFFLINE_BIOVOC/v2019-10 on ComputeCanada for 0.25deg and 0.5deg are fixed now. The Harvard FTP has a subset of the data that is on Compute Canada, so editted the files that are currently on ComputeCanada.

For anyone that wants to patch their own local copy, here's the command I used for the 0.5deg files (requires NCO installation)

find -type f \( -name 'biovoc_05.[0-9][0-9][0-9][0-9]0101.nc' -o -name 'biovoc_05.[0-9][0-9][0-9][0-9]1229.nc' -o -name 'biovoc_05.[0-9][0-9][0-9][0-9]1230.nc' -o -name 'biovoc_05.[0-9][0-9][0-9][0-9]1231.nc' \) -exec /path/to/fix-biovoc_05 {} \;

[YanshunLi-washu]

@lizziel Hi Lizziel, I am still in the process of applying my own CCVM account, the administrator haven't respond me yet. I'll work on this sort of issue next time. Apology if this caused any delay. @LiamBindle Thanks for the help!

[fritzt]

After dealing with the same issue as described on this thread when running GCHP, I noticed that biovoc_05.20161228.nc in OFFLINE_BIOVOC/v2019-10 has a suspicious file size and seems to be filled with just zeros. Does anyone know what caused this?

[msulprizio]

I can confirm that this seems to be the case in the original file on Compute Canada at http://geoschemdata.computecanada.ca/ExtData/HEMCO/OFFLINE_BIOVOC/v2019-10/0.5x0.625/2016/12/.

@hongjianweng Could you please check the original file on your end? If needed, could you reprocess and upload the corrected file to Compute Canada?

[sdeastham]

Is it worth reopening this issue/opening a new one?

[hongjianweng]

Hi @msulprizio , The 2016/12 files have been updated. The others will be updated continuously, but it will take time because the network connection with globus is not good recently. Hongjian

[msulprizio]

Thanks @hongjianweng. The corrected file for 2016/12/28 has been copied to Harvard's site as well.

Note: It now looks like the file for 2016/12/25 on Compute Canada is zero bytes. We still have the old file at Harvard, but I wanted to point this out. I'm not sure if a sync is still in progress or if this is a mistake.

[msulprizio]

@sdeastham pointed out the files for 2012 and 2013 on Harvard's ftp site still had the timestamp issue. I have now fixed those files (Jan 1, Dec 29-31) with Liam's fix-biovoc_05 script. The files for other years were already corrected.