[QUESTION] - We are facing "COLUMN DEFINED IN THE SCRIPT IS MISSING IN THE CSV FILE" After updating SFDMU Version

[vijay8432]

We are running with some weird issues after updating SFDMU plugging. We are not querying "OwnerId, CreatedById, LastModifiedById" these fields in our export.json file but still, we are getting below issues.

1. We are facing the "COLUMN DEFINED IN THE SCRIPT IS MISSING IN THE CSV FILE" issue when we removed "UserAndGroup.csv" from the list of CSV files those we are retrieving from Source Org.

2. We are facing the "OwnerId" child missing error in "MissingParentRecordsReport.csv" even we don't use "OwnerId" in our SOQL queries in the export.json file.

We are not able to understand why plugging adds 'ownerId' field on retrieved CSV files from source org but the strange thing is that we are not facing that error with all objects.

NOTE:- Facing that issue with some of the objects. https://prnt.sc/uqkhye https://prnt.sc/uqkio5 Thanks

[hknokh]

Hello, @vijay8432

First, ensure that you have not implicitly included these lookup fields (such as ownerid) in the query using multiselect keyword like all. If the case, the option is exclude all unnecessary fields with excludedFields property.

But basically, the COLUMN DEFINED IN THE SCRIPT IS MISSING IN THE CSV FILE is just in terms of a warning and it's not a critical for the migration process if you don't want to populate those lookups in the Target org.

---

BTW. I would ask you when opening cases in future, to kindly use one of our issue tracker templates + to provide the export.json & .log file.
Cause very mostly, these details are critical to analyse the issue and to provide a correct solution. Thanks you!

[vijay8432]

Hello @hknokh We used "excludedFields" property and added these fields but still facing the same issue.

One more concern If we are not using "ownerId" field in our SOQL query then why it's populating in CSV file for some objects, not for all objects and its standard field and we don't have control over these fields.

Previously its working fine without any issue but after the version update, we are facing these issues with some of the objects, not for all objects Thanks

[hknokh]

Can you provide the export.json + .log file please ?

[vijay8432]

Hello @hknokh Please check the below link and let me know If you need any other information. https://drive.google.com/drive/folders/1wn1xUDkZVOjNnnI_STzBKOyYEns2Rg6-?usp=sharing

Thanks

[hknokh]

Hello @vijay8432 First, you put Id for upsert operation, note that it will not work for org2org and will always create new records since the Id is different in both orgs. Regarding the subject. This row in the log: [19:31:48.336] The parent sobject User for the lookup field Contact.Customer_Portal_Userc has been included into the process. The externalId is set to 'Name'. ... says that you are including lookup field Contact.Customer_Portal_Userc, that is referencing the User sobject, that is why the Plugin is automatically adding User sobject as readonly into the migration schema and it creating the csv file on Org2Csv and requiring the csv file and on Csv2Org.

[hknokh]

But regarding the OwnerId I don't see this field referenced neither in log file nor in export.json. Please recheck, that the provided CSVIssuesReport.csv is matching the latest log/export.json files.

I made several test with all keyword on my and - the OwnerId never included in the query.

[vijay8432]

Hello @hknokh

Yes, The CSVIssuesReport.csv and Export.JSON are the latest ones. The Issue occurs after re-installing the SFDMU plugin.

I added new folder "MenuFormObjectOwnerIdIssue" on same link https://drive.google.com/drive/folders/1wn1xUDkZVOjNnnI_STzBKOyYEns2Rg6-.

We are using the "Id" field as an "externalId" because of duplicate records that we are having for testing purposes. If we are using 'Name' or any other field as "externalId"**** then might be there is a chance of record missing or records are not populated in a proper way.

It will be good for us If we will connect and discuss the same for further information. Please let me know your thoughts.

Thanks

[hknokh]

Hello, @vijay8432 Yes, I see, that it includes PagesApiMenuc.OwnerId into the schema and have no clue why. This should be checked and debugged on your end on live org. Unfortunatelly I can't help you in this from my end...

[vijay8432]

Hello @hknokh How can we debug the same? Can you please help us? Can you please tell us who will help us in this situation? Thanks

[hknokh]

For self-debugging, please follow this article: https://help.sfdmu.com/plugin-basics/basic-usage/debugging the file that you need to check is script.ts, primaraly review this method processObjectsMetadataAsync()

Or you can provide me the credentials to your org where you have the issue + relevant version of the export.json and I can debug it myself. Normally, I am in holiday and can't provide you support now only next week, but you have the option for the self-debugging.

[vijay8432]

Hello @hknokh Thanks for the quick response.

I will try the same by myself and will let you know and share the credentials and all details If I am not able to debug the same issue

Thanks

[vijay8432]

Hello @hknokh I found the issue in "sfdx.ts" async describeSObjectAsync function here you are getting the complete info of the target object and I guess you are not filtering the described Result of the target object with export Json SOQL query for that respected object.

Able to resolve the issue but not 100% sure whether its a valid workaround or not.

I just added that condition for now. if(!field.referenceTo.includes('Group') && !field.referenceTo.includes('User') && !f.label.includes('Tradestyle')){ sObjectDescribe.fieldsMap.set(f.name, f); }

Old Code is at line number 433

sObjectDescribe.fieldsMap.set(f.name, f);

async describeSObjectAsync function you are returning the array of sObject as per queries in export.json.

Please have a look once you will be back.

Thanks

[hknokh]

Hello, I am still in vacation). But I just had a quick look on your soultion and It's not the solution)) Because it will omit any lookup field to the User object and so will block the Plugin from handling all user lookups. Think, I need to take a look on it on your end. If you could provide me access I will be able to investigate.

[vijay8432]

Hello @hknokh Yes, I know that was not a good approach. I debugged the code and I was found the Issue but unfortunately, my VS code stop debugging, and now I am not able to debug the code for typescript on VS code.

I guess when we try to get a distinct field list from the described object Array and our SOQL query field array then there might be some code issue that adds ownerId fields for some of the objects.

We are facing issue with Account object as well. In our Source org column "TradeStyle" is not available but In our target Org that column available in Target org for Account Object.

Thanks

[hknokh]

Hello @vijay8432 Thank you for the update. I removed the provided credentials from the public access but saved this for me privately. I will look on it ASAP.

[vijay8432]

Hello @hknokh Thanks for the update. Sure, Please let me know if you need any further information. Thanks

[hknokh]

Hello

• Can't access the source org because of OAuth Error (maybe SSO is enabled or/and I am not a member of your organizartion).
• Can't also access the target org (sandbox) because of verification number is required. Please create a dedicated user for haim.knokh@gmail.com with the same profile as yours and check for the access to the source org.

[vijay8432]

Hello @hknokh Thanks for the quick response. I created a new user for target & Source org and removed the IP Address range specified for Source org. You will get an email for a set password for your user. Please check your email address and let me know If you haven't got an email.

Please check and let me know If you need further information.

Target Org Url: https://app-momentum-9119-dev-ed.lightning.force.com/ Source Org Url: https://gcouuk6ys.lightning.force.com/

Thanks

[hknokh]

Hello @vijay8432 I did a quick check and was able to reproduce the issue. I will update you further. Can I delete / modify data from the target org ?

[vijay8432]

Hello @hknokh Yes, You can do whatever you want. Please check the Account import as well for target org. Thanks

[vijay8432]

Hello @hknokh I hope you are doing well. Is there any update regarding the issue? If you need any help feel free to let me know. Thanks

[hknokh]

Hello @vijay8432 I'm sorry, have no time to tackle this for now due to very busy with my primary job. I will do it soon.

[hknokh]

@vijay8432 The issue should be solved. Please update and retry

[vijay8432]

Hello @hknokh Sure, I will check and let you know. It will be good to know If you will let me know about the issue. Thanks

[hknokh]

The problem was with the resoultion of the misspelled SOQL query string. When there is no such field or sobject in the org the plugin is trying to match the closest field and use it instead of the original one, it was applied OwnerId instead of one of the missing fields in the target org.

[vijay8432]

Thanks for the quick response and clear the thing. I checked the merged request changes as well. I tested after updating SFDMU and It's working fine.

Why it's throwing an error when we tried to import "Account" into target org for 'TradStyle' Column? Thanks