Issue with index.html when years available not equal to years requested in ocean

[vanroekel]

When trying to run a-prime on edison I was encountering a number of "file not found" on numerous ocean images (climatologyMap) even though all diagnostics completed successfully and images are evident in the directory. It appears that when MPAS-Analysis checks available data and trims the requested length if data is not available, the file name is not updated in the a-prime index.html file. For example, in a recent run (/project/projectdirs/acme/www/lvroekel/cori-knl.2017-master-dec12.hmod402bt32x4.mdec12.st04.at2.si.nhf.s32.ne120np4_oRRS18to6v3_ICG_years1-3_vs_obs) I can see a figure with the following name sstHADOI_cori-knl.2017-master-dec12.hmod402bt32x4.mdec12.st04.at2.si.nhf.s32.ne120np4_oRRS18to6v3_ICG_JFM_years0001-0002.png, but the generated link in index.html is sstHADOI_cori-knl.2017-master-dec12.hmod402bt32x4.mdec12.st04.at2.si.nhf.s32.ne120np4_oRRS18to6v3_ICG_JFM_years0001-0003.png, which was what I requested in run_aprime.bash.

[xylar]

@milenaveneziani, correct me if I'm wrong but I think the intention in a-prime was simply not to support the feature of setting the start and end years out of bounds. The atmosphere diagnostics don't (or at least didn't) support clipping those bounds to the available data. I thought at least for awhile we had been checking in a-prime and quitting with an error if the start and/or end years were out of range. Is that no longer the case?

[vanroekel]

@xylar and @milenaveneziani A few extra notes 1) This may be a slightly special case where there was 6 months of data available in year 3 and the atmospheric diagnostics finished fine, as did the ocean, but with a different range and hence missing plots on the webpage 2) If I specify a range outside the available data (e.g. years 1 - 5 in this case), no warnings are thrown to top level output, in fact the atmosphere diagnostics signal that they completed successfully, but there are no plots generated and the logs are full of errors. The ocean again completed fine in this test, but with years 1-2 again and no plots available through the webpage. 3) All of my edison tests are with a-prime master (a-prime v1.1)

[xylar]

I see. I can only speak for the MPAS-Analysis side. Since ncclimo doesn't support incomplete years, we make sure the data is there to give complete years as part of correcting the start and end year. It sounds like this check is not the same as what happens internally in A-Prime.

The fact that you are not seeing error messages even when there are error in individual jobs on the atmosphere side may indicate that #11 is not fully resolved.

Anyway, I will need to leave this to @salilmahajan and @milenaveneziani to resolve as they see fit. I'm happy to help where I can.

[milenaveneziani]

hmm, how disappointing, I thought we had solved the error message issue on the atm side in the previous PR..

I think fixing the figure name issue in the index.html file should be easy though. I will look into it in the next update. Thanks @vanroekel for reporting this.

[xylar]

@milenaveneziani, the easiest thing to do in the next update, to be honest, might just to turn on html on MPAS-Analysis and link to the generated webpage, don't you think?

[vanroekel]

@xylar I hope I did not imply that MPAS-Analysis was doing anything incorrectly or should do as the atmosphere does. It certainly was not my intention. I was just noting the difference, for seasonal averages it seems to me that we should require complete years.

[milenaveneziani]

yes, that's true. I actually found the problem: it's a bug in generate_html, and makes me wonder how come we didn't catch it before..

[milenaveneziani]

So, to summarize, to-do list for next PR (in addition to updating the MPAS-Analysis submodule):

• check atm error message when years are out-of-bounds
• turn on html page generation in mpas-analysis and change generate_html to point to ocn-ice page (Note: I will probably leave links to most common plots, such as the global SST or OHC trends, as are now, so that people can easily access them).

[xylar]

No worries, @vanroekel. I agree that MPAS-Analysis is doing what it should so it's more a question of the easiest way to make sure a-prime agrees. It sounds like @milenaveneziani already may have figured it out.

[milenaveneziani]

Addressed by #35.