Open rswamina opened 3 weeks ago
@valeriupredoi - can you please look into what I might be doing wrong when you get a chance? Thanks!
@rswamina could you please post the entire trace or the debug file, please? Not much I can gauge from that bit of output :grin:
Yes, of course..will try to attache the whole main_log_debug.txt file attached above. Let me know if you cannot access it. main_log_debug.txt
If all preprocessor functions are lazy, the save
step is where the variable (in this case pr
) data is loaded from disk, computations happen, and the result is written to the output file. Are you sure nothing is happening? How long did you wait for?
The timestamps in the attached debug log do not match those in the top post.
I reran it to generate a fresh debug file. Please consider only the timestamps in the file. I waited around 30 minutes and then killed the job. How long should I wait for?
I should add that I tried this several times wondering if there was an issue with JASMIN. The longest I waited (not sure if it was this particular run) was 30 minutes.
I tried running the recipe on my laptop with just the first dataset
- {dataset: CESM2-WACCM, project: CMIP6, exp: historical, ensemble: r1i1p1f1, start_year: 1995, end_year: 2014, grid: gn}
It completes in about 6 seconds and uses 2GB of RAM with ESMValCore v2.11.0 and 9 seconds / 4GB of RAM with v2.10.0. Maybe it is something specific to Jasmin indeed. Can you access the files that are listed in the top post, e.g. with ncdump
?
I had a similar problem with other data that originated from using dask distributed with the mask_landsea
data. This will be fixed soon (https://github.com/ESMValGroup/ESMValCore/pull/2515).
From the debug log it looks like @rswamina is using the default scheduler (so no distributed).
I am using the default scheduler. On JASMIN, SSP245 has just the r1 ensemble member's data. It seems to hang for both historical and ssp245 experiments. I tried an ncdump
on the files and am able to see the file content under the /badc/
archive path. This was not an issue for other models. I cannot think of what else could be an issue unless someone else can reproduce this on JASMIN. I will also add that I have successfully been able to process historical data for this model a few months ago.
Am travelling back to the UK today, will have a look tommz or on Thu, before I fo that though - Ranjini, could you pls confirm there is enough room on the disk you writing to, and what iris and esmvalcore you using? Cheers 🍺
Thanks @valeriupredoi. I have enough space on disk. I am using the esmvaltool
module installed on JASMIN for this, here are the version details:
>esmvaltool version
ERROR 1: PROJ: proj_create_from_database: Open of /apps/jasmin/community/esmvaltool/miniconda3_py311_23.11.0-2/envs/esmvaltool/share/proj failed
ESMValCore: 2.10.0
ESMValTool: 2.10.0
I am not sure how to check the iris version though.
Thanks Ranjini! I know how to do that 😁 Speaking of, I should install 2.11 rather sooner than later
ESMValTool seems to have trouble writing a preprocessed cube after extracting IPCC regions for the CEMS2-WACCM dataset. I don't know if the program hangs or simply is taking too long but the log message stops at a point where the cube is being saved under the preproc directory. I am using the
esmvaltool
module installed on JASMIN. As far as I can tell there are no typos!Here is the recipe:
Here are the final lines of the error message. After the last line shown, the program just waits and I don't know why it hangs there. This recipe works for other models.: