Unable to run tasks in remote projects

[acrosman]

Trying to use the feature to run tasks and flows from another projects, and getting an error message that cci cannot find the module containing the task.

Reproduction instructions:

1. Create new cci project in empty directory: git init; cci project init (used all default settings, including skipping project extensions).
2. Add the data generation tool draft from SFDO Philadelphia sprint to the end of the cumulusci.yml file:

   sources:
   data_import:
       github: https://github.com/SFDO-Sprint-2019-Philadelphia/DataGenerationTool

3. Run the task: cci task run data_import:generate_data.

The result is a python error that ends: ModuleNotFoundError: No module named 'tasks'

If you add a local tasks directory with just an __init__.py file the error will switch to: ModuleNotFoundError: No module named 'tasks.GenerateRecordPermutations'

[acrosman]

I should also include that the project does successfully check out from the remote source, and a copy is located in the local .cci directory.

[davisagli]

@acrosman thanks for the report -- I don't think we tested including a project that defines its own custom tasks, and it doesn't surprise me that that will need some work to make sure the tasks get imported from the right path.

[edmondop]

@davisagli can you help me with that?

[TamiTakamiya]

@davisagli Hello, we also want to have a solution of this issue. I created sample projects for easier problem recreation.

• sdfx-proj1
• sdfx-proj2

sfdx-proj2 defines two tasks.

1. dxtask - Issues an sfdx command using CCI's dx task
2. mytask - Invokes a Python script, which is stored in the /scripts/tasks directory of sfdx-proj2

sdfx-proj1 defines sfdx-proj2 as a source. and cci task run sfdx-proj2:dxtask completes successfully, but cci task run sfdx-proj2:mytask fails with No module named 'scripts.tasks' because it could not find the script.

I think when an external project is loaded, Python class_path needs to be adjusted accordingly. Thanks.

[davisagli]

The challenge is that if two different projects are included as sources and they both have custom tasks in folders with the same name, then they can't both be imported in Python. There will be a name collision in sys.modules.

[TamiTakamiya]

IMO, if the rule is documented, just allowing one from multiple tasks with a same name would be fine.

For example, dependencies in Maven is resolved by the first dependency found even if different versions of the same artifact are found in the dependency tree. Is it possible to implement a similar mechanism for searching custom tasks with a same name?

[prescod]

The issue is not so much at the filename level, though that is a bit of an issue.

The issue is that every project tends to register custom tasks into the exact same namespace called tasks:

NPSP:

    is_rd2_enabled:
        description: This preflight check ensures that Enhanced Recurring Donations is enabled
        class_path: tasks.is_rd2_enabled
        group: NPSP

EDA

    has_einstein_licenses:
        group: "EDA: Einstein Templates"
        description: Queries the target org for the existence of required permsets and returns true if found.
        class_path: tasks.check_einstein.CheckPermSetLicenses
        options:
            permission_sets:
                - EinsteinAnalyticsUser
                - EinsteinAnalyticsAdmin
                - EinsteinAnalyticsPlusAdmin
                - EinsteinAnalyticsPlusUser

So we either need to change this practice or figure out how to hack around it.

[prescod]

The Snowfakery task runs Snowfakery out-of-process for parallelization reasons. (GIL!!!)

A similar work-around could allow custom tasks to be run out-of-process and have their inputs and outputs transferred through files or pipes. Then custom tasks would never conflict with each other. But they'd also be quite a bit slower to start up.