Closed Row-Bean closed 10 months ago
@SaptSinha Could you please reprocess this file when you have a chance?
@Row-Bean I uploaded the reprocessed "MERRA2.19830927.A3dyn.05x0625.NA.nc4" file. You can access it from WashU's 1983/09 MERRA2 collection.
Excellent, thank you! I downloaded the new file, it appears not to have any problems.
Thanks @SaptSinha and @Row-Bean! I will close out this issue now.
I found another one: GEOS_0.5x0.625_NA/MERRA2/1983/11/MERRA2.19831102.A3mstC.05x0625.NA.nc4
The GEOS-Chem error is similar:
In Ncop_Rd, cannot open: /home/rstevens/projects/rrg-hayesp/rstevens/GC/ExtData//GEOS_0.5x0.625_NA/MERRA2/1983/11/MERRA2.19831102.A3mstC.05x0625.NA.nc4
And I cannot open this file with ncview at all:
ncview: can't recognize format of input file MERRA2.19831102.A3mstC.05x0625.NA.nc4
Again, I tried re-downloading via Globus from the WashU collection, but I still cannot open the file with ncview.
Thank you for letting us know about this earlier. I have reprocessed and uploaded the MERRA2.19831102.A3mstC.05x0625.NA.nc4 file. You can find it from our 1983/11 MERRA2 collection.
Thank you very much! I have downloaded the new file, ncview can read it without any problems.
Excellent! I am closing this issue for now but please do not hesitate to open a new one in case you encounter any further data corruptions.
I also wanted to let you know that I will post an update regarding your other issue #1544 sometime this month as we have finally resolved some of the problems that we were previously encountering with inconsistent global dust strength values.
I have found another one:
In Ncop_Rd, cannot open: /home/rstevens/projects/rrg-hayesp/rstevens/GC/ExtData//GEOS_0.5x0.625_NA/MERRA2/1983/11/MERRA2.19831105.I3.05x0625.NA.nc4
ncview: can't recognize format of input file ~/GC/ExtData//GEOS_0.5x0.625_NA/MERRA2/1983/11/MERRA2.19831105.I3.05x0625.NA.nc4
Would you prefer for me to open a new issue when I find more corrupt files, or would it be best for me to continue to post them here?
@Row-Bean I reopened this issue and renamed it to reflect that there are multiple problematic files within GEOS_0.5x0.625_NA/MERRA2. Feel free to continue reporting them here and tag @SaptSinha, as he is the one that will reprocess them for you.
Thank you! I have one more file for @SaptSinha :
I looked ahead and found one more file that doesn't appear to open with ncview: GEOS_0.5x0.625_NA/MERRA2/1983/11/MERRA2.19831106.A3mstE.05x0625.NA.nc4
I did a quick check of the rest of 1983 and 1984, but I didn't find any more corrupt files yet.
Thanks for reporting about these corrupted files, @Row-Bean. I have reprocessed and uploaded the files "MERRA2.19831105.I3.05x0625.NA.nc4" and "MERRA2.19831106.A3mstE.05x0625.NA.nc4". You can find them at our 1983/11 MERRA2 collection.
Thank you again! I downloaded the new files, and ncview can read them without problems.
Great! We can keep this issue open in case you encounter any more corrupted files.
Hi @SaptSinha, I found another one: GEOS_0.5x0.625_NA/MERRA2/1983/11/MERRA2.19831105.A3mstE.05x0625.NA.nc4
No worries, I finished reprocessing and just uploaded the MERRA2.19831105.A3mstE.05x0625.NA.nc4 file in the same location.
Excellent, thanks! I checked the new file, it appears to be corruption-free.
Thanks @SaptSinha and @Row-Bean. We can close out this issue now.
Hi @SaptSinha, I found another one: GEOS_0.5x0.625_NA/MERRA2/1983/11/MERRA2.19831106.A3dyn.05x0625.NA.nc4
I'm juggling this with a couple of other priorities at the moment, but when I get some free time I'll find a way to look through the files more efficiently to find the corrupt ones. It shouldn't take long to put together a Python script to find them.
@laestrada Wrote a script to find corrupted met files. You can give this a try to see if it identifies any files ahead of time for you. See https://github.com/geoschem/integrated_methane_inversion/blob/main/src/utilities/find_corrupt_files.sh. You can remove the lines about AWS. Essentially, you just want the loop over the files.
@SaptSinha It may be worth bringing this script into the met-processing workflow to identify corrupted files at WashU before users download them.
Thanks to @msulprizio for sending that script! I applied it to the rest of the time period I plan to run (up to 1985), and the only corrupt files the script found were GEOS_0.5x0.625_NA/MERRA2/1983/11/MERRA2.19831106.A3dyn.05x0625.NA.nc4 (as listed above) and GEOS_0.5x0.625_NA/MERRA2/1983/11/MERRA2.19831107.A3cld.05x0625.NA.nc4
So once @SaptSinha uploads those two files, that should be all that I need.
Hello @Row-Bean, the files below have now been reprocessed and uploaded to http://geoschemdata.wustl.edu/ExtData/GEOS_0.5x0.625_NA/MERRA2/1983/11/. Please feel free to reach out if you need any further assistance.
GEOS_0.5x0.625_NA/MERRA2/1983/11/MERRA2.19831106.A3dyn.05x0625.NA.nc4 GEOS_0.5x0.625_NA/MERRA2/1983/11/MERRA2.19831107.A3cld.05x0625.NA.nc4
Excellent, thank you very much! I have downloaded and checked the new files. Fingers crossed, this issue should be closed for the last time this time. I appreciate your assistance with this.
Name and Institution (Required)
Name: Robin Stevens Institution: Université de Montréal
Description of your issue or question
Hi, I'm running GEOS-Chem 13.3.3, and I ran into the following error:
I used ncview to take a look at MERRA2.19830927.A3dyn.05x0625.NA.nc4, and when I try to view the RH, ncview crashes with the following error:
I re-downloaded the file from the WashU collection using Globus, but the file still seems to be corrupt in the same way.