Migrating changes from experimental to core

[diptorupd]

Capturing the design sketch of how we will migrate the new features from experimental to core.

• [x] Rename numba_dpex.experimental.kernel_iface to numba_dpex.kernel_api. The kernel_api module will be a pure Python module to simulate kernel programming.

• [x] Merge features from DpexExpKernelTarget to DpexKernelTarget

• [ ] Convert DpexKernelTarget into an interface that will provide the template for implementing a kernel back end.

  • [ ] Maybe drop the Dpex suffix altogether and call it AbstractKernelTarget.

• [x] Rename the DpexKernelTarget to SpirvKernelTarget

• [ ] Define a sub-module organization structure for a back end. It will be the guideline for developing future kernel code generation back ends.

  • [ ] Each back end should provide a specialization of DpexKernelTarget
  • [ ] Overloads
  • [ ] Math registry

• [ ] Move types.py, dpcpp_types and literal_intenum_type into core.types.

• [ ] Merge experimental.models.py into core.datamodel.models.models.py

• [ ] Merge typeof.py

• [ ] Convert numba_dpex.experimental.flag_enum.py into numba_dpex.flag_enum.py

• [ ] Move dispatcher, launcher to numba_dpex.core.kernel_interface (remove the old JitKernel based dispatcher)

• [ ] Remove numba_dpex.core.kernel_interface.dispatcher

• [ ] Remove numba_dpex.core.kernel_interface.func

• [ ] Remove numba_dpex.core.kernel_interface.arg_pack_unpacker

[diptorupd]

I audited all the modules in numba_dpex and have drawn up a first draft of how things should be reorganized. Not all of the noting are applicable to the experimental to core migration.

numba_dpex
    |-> _kernel_api_impl : <<NEW>>
          |-> spirv
          |-> descriptor
          |-> target (typing and target contexts)
              |-> target-specific overlaod for kernel_api
              |-> codegen/codegen library
              |-> dispatcher
              |-> spirv_generator # Moved from top-level module
    |-> core
      |-> datamodel : <<NOCHANGE>>
      |-> decorators : <<NEW>> renamed from numba_dpex.decorators
      |-> kernel_interface : <<REMOVE>> Make sure parfor work with new API
          |-> indexers : <<MODIFY_AND_MOVE>> (Python classes to move to kernel_api and overloads to core)
      |-> parfors : <<NOCHANGE>>|<<MOVE>> move to experimental or leave as is?
      |-> passes : <<NOCHANGE>>
      |-> pipelines : <<NOCHANGE>>
      |-> runtime : <<NOCHANGE>>
      |-> targets
             |-> kernel_target : <<MOVE>> To numba_dpex._kernel_api_impl.spirv.target
      |-> typeconv : <<REMOVE> to be removed along with core.kernel_interface
      |-> types : <<NOCHANGE>>
      |-> typing : <<NOCHANGE>>
      |-> utils:
            |-> caching_utils : <<REMOVE>> (?)
            |-> suai_helper : <<REMOVE>>
      |-> caching : <<REMOVE>> (?)
      |-> codegen : <<MOVE>> To _kernel_api_impl.spirv.codegen
      |-> compiler : <<REMOVE>> (?)
      |-> descriptor : <<MODIFY>> DpexTargetOptions move to numba_dpex._kernel_api_impl.spirv.descriptor
      |-> dpjit_dispatcher : <<NOCHANGE>>
      |-> exceptions : <<RENAME>> To errors to match numba (?)
      |-> itanium_mangler : <<NOCHANGE>>
    |-> dpctl_iface : <<NOCHANGE>>
    |-> dpnp_iface : <<NOCHANGE>>
    |-> examples : <<NOCHANGE>>
    |-> experimental
            |-> _kernel_dpcpp_spirv_overloads : <<MOVE>> numba_dpex._kernel_api_impl.spirv.overloads
            |-> kernel_iface : <<MOVE>> numba_dpex.kernel_api
            |-> decorators : <<MOVE>> numba_dpex.core.decorators
            |-> dpcpp_types : <<MOVE>> numba_dpex.core.types
            |-> flag_enum : <<MOVE>> numba_dpex.kernel_api (?)
            |-> kernel_dispatcher : <<MOVE>> numba_dpex._kernel_api_impl.spirv.dispatcher
            |-> launcher : <<MOVE>> To numba_dpex.core
            |-> literal_enum_type : <<MOVE>> To numba_dpex.core.types
            |-> models : <<MOVE_AND_MERGE>> To numba_dpex.core.datamodel.models
            |-> target : <<MOVE_AND_MERGE>> To numba_dpex.core.targets.kernel_target and then move to 
                        numba_dpex._kernel_api_impl.spirv.target
            |-> testing : <<RENAME_AND_MOVE>> To numba_dpex.core.utils (?)
            |-> typeof : <<MOVE_AND_MERGE>> To numba_dpex.core.typing.typeof
            |-> types : <<MOVE>> To numba_dpex.core.types
    |-> kernel_api : <<NEW>> top-level module with working Python API simulating kernel programming.
                     Renamed from numba_dpex.experimental.kernel_iface
            |-> math : <<PROPOSAL>> Python library with wrappers of scalar math functions usable in a kernel
                                    compendium of math, cmath, numpy
    |-> ocl : <<REMOVE>> To be removed and replaced by target-specific overloads in backend
         |-> stubs : <<MOVE>> To kernel_api with working Python implementation added
         |-> mathdecl : (?)
         |-> Mathimpl : (?)      
         |-> ocldecl : <<REMOVE>> Replaced by kernel_api with working Python implementation added for all ocl stubs
         |-> oclimpl : <<REMOVE>> Replaced by target-specific overloads in numba_dpex._kernel_api_impl 
    |-> tests : Changes to be determined
    |-> utils 
          |-> array_utils : <<REMOVE>> (?)
          |-> constants : <<REMOVE>> NOTE: the address_space class should be moved to kernel_api
          |-> llvm_codegen_helpers : To be evaluated
    |-> debuginfo : <<MOVE>> to numba_dpex.core
    |-> decorators : <<MOVE>> To numba_dpex.core
                       : (2) replace existing decorators with experimental.decorators for kernel and device_func
    |-> parfor_diagnostics : <<MOVE>> To numba_dpex.core.parfors
    |-> printimpl : <<MOVE>> Should be moved into the numba_dpex._kernel_api_impl.spirv
    |-> spirv_generator : <<MOVE>> To _kernel_api_impl.spirv
    |-> vectorizers : Note that the functionality is required but the implementation is broken and should be removed??

[diptorupd]

I am marking this as "Done" as experimental module no longer exists and most changes are migrated. There are still few leftover cleanups namely: getting rid of core.kernel_interface, moving ocl.mathimpl and ocl.mathdecl to kernel_api_impl. But those can be done later.