Invalid Lookup Error in Snowfakery Recipes

[apretorius]

Describe the bug

As of release 2.84.2 we are seeing this error when trying to load Snowfakery recipes "The lookup Account is not a valid lookup for ep01Organizationc in sf_object: npspGeneral_Accounting_Unitc. Exception in task snowfakery"

This is happening in all the lookup fields.

Here's an error gist from a load that was importing from a csv: https://gist.github.com/apretorius/e1e1cddc0937dcbbb934adbecaf683bd

Reproduction steps

1. Select a snowfakery recipe that is populating a lookup field with a namespace token
2. Try to load it into an org
3. Get the lookup error.

Your CumulusCI and Python versions

Cumulus CI version: 2.84.2 Python version: 3.12

Operating System

macOS Monterey Version 12.6.2

Windows environment

No response

CumulusCI installation method

brew

Error Gist

https://gist.github.com/apretorius/e1e1cddc0937dcbbb934adbecaf683bd

Additional information

This is blocking our development work because we can't load in test data, so it's a pretty urgent item.

[aditya-balachander]

Hey @apretorius, the error is happening at the load_dataset step of the flow. There is a validation that ensures that the lookup tables specified are in the refereceTo of the lookup field. This is where it is failing in your case (returning an error like The lookup <lookup_sobject> is not a valid lookup for <lookup_field> in sf_object: <sobject>)

To reproduce the error, I tried creating a custom namespaced sobject (since it looks like it is failing in custom objects only), adding a lookup field and running extract and load. Unfortunately I was not able to reproduce the error you encountered.

I would need your recipe.yml, the mapping and sql file generated and the org setup details to see why this error is occurring

[mikechayes]

We are seeing the same errors loading data in v3.84.3, v3.84.1 is working with no error.

Here is the CCI load_data command: cci task run load_dataset --mapping datasets/mapping.yml --sql_path datasets/data.sql --drop_missing_schema True

Here is a sample of the errors: (This is happening on all Lookup fields)

The lookup Contact is not a valid lookup for Signatorc in sf_object: AQBTransactionc The lookup Account is not a valid lookup for MatchedAccount__c in sf_object: AQBMatchLinkc The lookup AQBQuidSpecificationc is not a valid lookup for QuidSpecificationNumberc in sf_object: AQBQuidProQuoc

[frankeetou]

We are seeing the same behavior! with version 3.84.3 Upvoting this one !

[RBLopez-BTH]

@apretorius and I work together, and chiming in here to say we have a blanket recommendation out to our entire organization to refrain from upgrading past 3.84.1 until this is fixed.

We've had issues with namespaces in our data in prior versions and were finally on track to eliminate the need for maintaining two different datasets (one namespaced, one not) after https://github.com/SFDO-Tooling/CumulusCI/pull/3713 . Is it possible that namespaces aren't being correctly identified and dealt with in this new lookup validation behavior?

[mikechayes]

We have developers on 3.84.1 who can load the dataset successfully, while only those on 3.84.3 are facing the issue. Is it possible to revert to 3.84.1?

[apretorius]

@mikechayes

YES, you can fortunately downgrade pretty simply:

pipx uninstall cumulusci pipx install cumulusci==3.84.1

[apretorius]

@aditya-balachander Here is the sql file we're loading along with the mapping file. blanked-data.zip

The task is this:

[mikechayes]

Thanks! After the downgrade to 3.84.1, we can load data again.

[RobertoMartinez-districtm]

We had the same problem here. load_dataset would fail on lookups with 3.84.3. Rolled back to 3.84.1 for the win!

[n-parupudi]

issues with v3.84.3 : load_dataset failing when :

1. Lookup field to itself on a custom object without namespace fails to parse mapping with invalid reference error.
   Workaround - specify complete namespace and double underscore __ before the custom lookup field name.
2. lookup to another custom object with after: insert in the mapping is failing with cumulusci.core.exceptions.BulkDataException: Error on record with id 1: MISSING_ARGUMENT: Id not specified in an update call. This is a major issue. Cant find any workaround Moving back to v3.84.1 loads the dataset just fine.

[apretorius]

@aditya-balachander requesting a follow up here since more folks are reporting this same error with the only work around being to not upgrade cumulusci

[prescod]

Let me know if someone from the CCI team wants to pair-program on this.

[aditya-balachander]

Hey @apretorius thank you providing the mapping and sql file. The issue was due to the way that our validation was handling namespaces. I have raised a PR #3759. This should fix the issue