Land-sea masking takes 3 times longer to run

[hahsan1]

When running a recipe with land-sea masking, I find that it can take 3 (and sometimes even up to 5) times longer to run compared to without a mask. Is this normal?

I'm running with ESMValTool v2.1.1.

[valeriupredoi]

@hahsan1 could you please post the recipe you using? Masking is a rather time-consuming operation so am not surprised it takes longer, probably not 3-5x more though; I'll go straight and try answer the other issue you opened, my apologies, I was away :+1:

[hahsan1]

I have attached two recipes. The _global_diffshp-10p-red has no mask applied to it and it took 0:17:53 to run. The other recipe is _land_diffshp-10p-red and it masks out the sea. The time it took to run was 0:56:12.

These recipes are analyzing the same models and variables. The only difference is the masking.

global_diff_shp-10p-red.txt land_diff_shp-10p-red.txt

[valeriupredoi]

cheers! I'll have a look today :beer:

[valeriupredoi]

hi @hahsan1 I tried running your recipes but that data can not be found neither locally on the ESGF node on CEDA/JASMIN nor on any other remote ESGF node to be downloaded - is that your locally produced model? If you could supply me with a used case that shows the massive cliff in performance due to masking, using readily-available ESGF data, that would be great! Also note that the difference in timing might be due to the diagnostic taking longer for one case; also if the data is very high resolution then yes, you'd expect a significant drop in performace when doing masking (especially with shapefiles) since the number of operations per dataset increases by quite a bit when masking

[hahsan1]

I was able to reproduce it with data that can be found on ESGF, but on a smaller scale. So the global_test.yml recipe has no masking and it took 0:02:41, and the land_test.yml has sea masking and took 0:07:04, so about 2.5 times. land_test.txt global_test.txt

I installed the files using the wget command, as attached here: test_files.txt

It may be, as you said, just the increased number of operations with masking. In that case, this is just an observation and no course of action is needed to be taken to remedy it.

[hahsan1]

We managed to work around the longer processing time by running recipes in parallel. This seems to work for us.

[valeriupredoi]

@hahsan1 excellent, cheers for the heads up! What are you guys working on with ESMValTool as a fork? it'd be nice if you dropped a brief description of the project either as an issue or in the Discussion section - that way we'd know more of your use of ESMValTool and we could offer assistance/pointers etc. Also we could add a link to the documentation for other users to be aware of your project (if it's not private) :beer:

[hahsan1]

We are working on a project called Emissions-MIP which focusses on evaluating the sensitivity of models to various emissions characteristics. We are still in the stage of verifying our experimental protocol, and we've essentially been using ESMValTool for all of our analysis. You and your team have been extremely helpful and we thank you for that!

We presented preliminary results at the AeroCom meeting last month! (Our abstract is the second one down if you're interested)

[valeriupredoi]

awesome, excellent work and very happy we could (and will, in the future) help! Cheers, mate, would you be OK if I quoted your project summary (in the comment above) in our communications? I am thinking we will gather some basic info on projects/teams using ESMValTool for pride&joy on either the website https://www.esmvaltool.org/ or in the intro in the documentation :beer:

[hahsan1]

Yes that would be great, thanks!