[QUESTION] - Use new Record Ids in transformation

[Libertha]

Goal: migrate records from one production to another production and transform a key that contain some recordIds combined.

Issue: We need the newly created recordIds to be used in the transformation to build the key. However the transform will only use old recordids and not the recordIds of the newly created records.

Example: OLD ORG Unique_Key__c = {oldrecordId1}{oldrecordId2}{oldrecordId3}

NEW ORG Unique_Key__c = {newrecordId1}{newrecordId2}{newrecordId3}

export.json: { "objects": [ { "operation": "Upsert", "query": "SELECT Old_Quote_Idc FROM Quote WHERE Old_Quote_Idc = '0Q0W5000004lFkTKAU'", "externalId": "Old_Quote_Idc", "afterAddons": [ { "module": "core:ExportFiles" } ] }, { "operation": "Readonly", "query": "SELECT all FROM Survey_Question__c", "externalId": "Lift_Surveyr.Name;Name" }, { "operation": "Readonly", "query": "SELECT all FROM Survey_Question_Optionc", "externalId": "Survey_Question__r.Lift_Surveyr.Name;Survey_Questionr.Name;Name" }, { "operation": "Upsert", "query": "SELECT all FROM Survey_Responsec WHERE Quoter.Old_Quote_Id__c = '0Q0W5000004lFkTKAU'", "externalId": "Name", "afterAddons": [ { "module": "core:ExportFiles" } ] }, { "operation": "Upsert", "query": "SELECT all FROM Survey_Answerc WHERE Survey_Responser.Quoter.Old_Quote_Idc = '0Q0W5000004lFkTKAU'", "externalId": "Name" } ], "dataRetrievedAddons": [ { "module": "core:RecordsTransform", "args": { "fields": [ { "alias": "responseId", "sourceObject": "Survey_Answer__c", "sourceField": "Survey_Responsec" }, { "alias": "questionId", "sourceObject": "Survey_Answerc", "sourceField": "Survey_Question__c" }, { "alias": "optionId", "sourceObject": "Survey_Answerc", "sourceField": "Answer_Optionc" } ], "transformations": [ { "targetObject": "Survey_Answer__c", "targetField": "Unique_Keyc", "formula": "formula.responseId + formula.questionId + formula.optionId " } ] } } ] }

[hknokh]

Hello, @Libertha

Thank you for reaching out. I will do my best to assist you as quickly as possible and will keep you updated on my progress.

Cheers

[hknokh]

Hello,

The purpose of the core:RecordsTransform module is to build compound source values to update target records, rather than to modify target records based on other target records. Therefore, your goal cannot be accomplished using this module.

Regards

[hknokh]

Arranged as a feature request:

https://github.com/forcedotcom/SFDX-Data-Move-Utility/issues/792

However, I'm not sure if this feature can be implemented because you need to ensure that all the target records of all objects used to build compound values for the transformed record are already populated in the target org. Since SFDMU uses dynamic object processing ordering, you can't be certain that your target values are ready before running the transformation.

[Libertha]

@hknokh thank you for investigating.

I thought it might work the same way as it works to do the lookup updates which happens after an insert for example. So it would be more of a "post processing" where you can transform the newly created records and then be taken into the update.

Do you have any suggestions in how i could accomplish this? or it is just not possible with sfdmu?

[hknokh2]

It technically can't be post-processing because the transformation is injected into the normal object update cycle. You are using multiple objects as a transformation source, which have to be ready before the transformation begins. Using an add-on to modify records after the target records are ready would require an extra update to the target with the transformation result, which is not possible. Lookup updates are synchronized with the object processing and have a dedicated mechanism to keep the records updated in any case. Add-ons are not synchronized and apply additional modifications to the existing records. I hope you understand the technical obstacle preventing me from adapting the module to your use case. We need to consider another approach using different SFDMU features or a set of features. A possible approach is to run two jobs: first, update the target prod with oldrecord1, oldrecord2, and oldrecord3; then use a second job with a single-direction migration and the recordstransform add-on module to populate the field with transformed values taken from the target org to update the same org.

[hknokh2]

Again, I will see if I can do something with the add-on itself, but it can take a while, I would not advise you to wait for this to be completed in a nearest time frame.

[hknokh]

Hello.

This case is now considered resolved, and I am closing it for now. If you have any additional questions or concerns, please don't hesitate to reach out.

Regards.