[i-first] Support kernels with "operates_on = domain"

[arporter]

• [x] Add parsing and validation of metadata for kernels with operates_on = domain - PR #944
• [x] Add PSy-layer code-generation for domain kernels - PR #1082
• [ ] Add stub-generator support for domain kernels

[arporter]

When thinking about halo exchanges, am I correct in thinking that where previously we thought about a "loop that iterates over owned cells", we are now thinking about any kernel that operates on either cells or the whole domain? (Since the latter is still cell based.)

[arporter]

Another question, really for @christophermaynard - presumably a kernel that has operates_on = DOMAIN cannot be a CMA kernel?

[arporter]

And also not a multi-grid kernel?

[arporter]

We discussed this in the PSyclone telco today and agreed that, at least in the first instance, a kernel with operates_on = DOMAIN will only accept field arguments.

[arporter]

Next question, what are the rules going to be for allowed accesses for these new kernels? Presumably, as soon as we start thinking about phase 2 of this work (which I think is putting the loop over segments into the PSy layer?) then it's clear we need to keep the rules the same as for cell columns?

[TeranIvy]

Not quite sure, as the cell-column access rules have the implicit assumption of colouring (gh_inc) for continuous function spaces and not colouring (gh_write, gh_readwrite) for discontinuous FS and built-ins. My guess is that updating a field on the whole domain would be more similar to how built-ins work.

[arporter]

Yes, that's what I was thinking originally but then I remembered that the PSy layer will ultimately be taking control of the looping over segments and then we have the boundaries between those segments to worry about.

[rupertford]

Physics routines are independent in the horizontal so read, write, and readwrite should be all we need if we are limiting ourselves to full columns. People talk about eventually being able to support stencils in the horizontal but there is no code that does that at the moment as far as I know. I guess the simplest thing to do in the short term would be to specify fields with a discontinuous function space, which I think is what they do now - using Iva's discontinuous_any_space - but perhaps they should just all use "w3", and in the longer term we can look to restructure/simplify the metadata for this type of kernel?.

[arporter]

The next stage for this issue is to generate the correct PSy-layer code for kernels with operates_on=DOMAIN. I'll do this on 925_code_gen.

[arporter]

In giving 'domain' kernels control of their own iteration space, we no longer have information on whether or not they compute into the halo or read from the halo. This will therefore have to be specified as a rule (or potentially a meta-data extension?). Note that these kernels may only operate on fields on discontinuous spaces that do not have stencil accesses.

[arporter]

As Rupert pointed out, it is the DynLoop.gen_code() method that currently creates the set_dirty()/set_clean() calls. If we have no Loop, we don't get those calls.

[arporter]

After chatting with Rupert we're thinking that a 'null Loop' might be the best way to avoid a massive refactoring. Such a Loop would exist in the PSyIR but would not actually generate any Fortran. Since 'domain' kernels only read and write fields on discontinuous spaces there are no annexed dofs to worry about. Since stencil accesses are also forbidden then these kernels will never require halo exchanges before them. Any fields written to will have their full halos marked as dirty after the kernel execution.

[TeranIvy]

@arporter, I recently updated the LFRic version of argument_mod.F90 (there were many unclear and unused things there) and I included it in my PR #1052 so we have up-to-date version in PSyclone. The DOMAIN iterator ID is there. The argument_mod.F90 on my branch from PR #1052 is quite different from what you have (as it was updated with the recent cleaned-up and updated LFRic trunk version) so it will clash with this branch if it goes to master before it.