[FEATURE REQUEST]-Advanced targetRecordsFilter

[nvuillam]

Is your feature request related to a problem? Please describe.

It would be nice to be able to use advanced filters with targetRecordsFilter configuration property. For example, a badword detector for GDPR requirements

Describe the solution you'd like

The capability to configure export.json with such advanced filter

Example:

{
  "sfdxHardisLabel": "Detect badwords in leads",
  "sfdxHardisDescription": "",
  "objects": [
    {
      "query": "SELECT Id,Name,ResidenceRecherche__c,Description,CreatedById,LastModifiedById,OwnerId,Status,DatePremierContact__c FROM Lead",
      "operation": "Readonly",
      "targetRecordsFilter": "core:DetectBadwords",
      "targetRecordsFilterParams": { 
        "badwordsFile": "../badwords.json",
        "detectFields": [
          "Description"
        ],
        "highlightWords": true
      }
    }
  ]
}

[hknokh]

Hello.

Thanks a lot for the contribution and it's a great idea having such filters! And indeed I can say, that i was pleasantly surprised at how you understand my code ... Awesome job done! However, can I ask you to rearrange your code to implement the standard sfdmu Addon Api which is already released? I would like to get this feature in a different way, as an external api core module, like it's done for the core:ExportFiles. Refer here: https://help.sfdmu.com/full-documentation/add-on-api/export-file-core-add-on-module And please see the module's source, how it was implemented. You have the detailed documentation about the addon api. The reason behind this is, that I've released the addon api engine to standardize the addition of new features by third-party developers, as well as to add any advanced features which for some reasons shoud not be a part of the main sfdmu code. I would not like to inject any third party code directly into the plugin source or to change anything. See that even me was using my own api for the export files feature... Your module should be fired using runtime event. If you want to have additional event besides the already existing, let me know, i will add it for you and you will be able to run your addon for instance before the target is being updated. First see the existing events, maybe it is already enough for your goal. Please don't change any existing plugin code, only use the addon api to build the desired filter. Thx.

[nvuillam]

Thanks for your reply :)

I had to put a lot of breakpoints and click a lot on "step into" to succeed to make this evolution 😄

I understand your desire to use the addOn architecture you defined, and i think it's a great idea, but I'm sorry , as it is today, the documentation (that i've read a lot) is not enough simple for me to really understand how to build one... it probably needs a complete example, tutorials, explanations about how to package the addOn (not run it from C:/ like in the example), etc ...

I have to use badwords filter in the production of a client, and I plan to reuse it on other clients, so ... as you know well your architecture, and as all the source code is in my PR, would you consider refactoring it the way you want ? 👼 (or accept the PR because i'm really nice ? ^^ )

Note: I plan to write an article about badwords detection related to GDPR and SFDMU will obviously be the main hero of it :)

[hknokh]

Hey, ok. Sorry, can't accept you Pr even it is nice). As I said, we have strong conventions how to extend the core code, we can't get around this . See, that this is not cosmetic modification, which can be accepted.

In summary I can create this plugin based on your pr. Now I am working on some evolutions in the code, so can touch yours as by the way. Will look onto your piece of code deeply and let you know if i have questions.

[hknokh]

If you want you can put here the descriptions about badwords.json, input parameters , how to make the transformations. Even i can understand this from your code it will save the time and avoid misunderstanding of details even the overall concept is clean. Thx

[nvuillam]

Thanks a lot :)

My code is quite simple, everything is within class https://github.com/nvuillam/SFDX-Data-Move-Utility/blob/9668c7553f4f7a9a315652daacfa4967169aa493/src/addons/filters/detectBadwords.ts , and it just needs a MigrationJobTask and a list of records as input data

It has to be called at some point, before writing target CSV file ( __filterRecords is a good place for that ^^ )

The rest of the PR is:

• Management of parameters for targetRecordsFilter (I imagine it won't be necessary and addOn can have their own parameters)
• error messages description

I also added what I think is a core bug fix (I needed it, else CSV columns were empty because their values retrieved from SF API were replaced by undefined), maybe I can put this one in another separated PR ?

Please let me know if you have any question :) I can help testing if necessary

badwords.json is a simple array of strings in a json file: ex:

[
   "whatever",
   "words",
   "or expressions",
   "here",
   "même accentués"
]

[hknokh]

Don't understand what is the issue with the undefined in csv file, can you explain more? May be any concrete example of rows coming from the sf? And an example of produced csv which gets messed up?

[nvuillam]

My SFDMU config file is this one (don't bother the hardis stuff, it's for UI integration within VsCode SFDX Hardis)

{
  "sfdxHardisLabel": "Detect badwords in leads",
  "sfdxHardisDescription": "",
  "objects": [
    {
      "query": "SELECT Id,Name,ResidenceRecherche__c,Description,CreatedById,LastModifiedById,OwnerId,Status,DatePremierContact__c FROM Lead",
      "operation": "Readonly",
      "targetRecordsFilter": "core:DetectBadwords",
      "targetRecordsFilterParams": { 
        "badwordsFile": "../badwords.json",
        "detectFields": [
          "Description"
        ],
        "highlightWords": true
      }
    }
  ]
}

The use case was to also extract Names from related lookup fields (CreatedBy,LastModifierBy,Owner,ResidenceRecherche__c -lookup to Account- )

• I first directly put CreatedBy.Name, ResidenceRecherche__c.Name, etc... but it was returning empty columns.
• Then I saw in the doc that __r stuff was not supported by SFDMU
• So I decided to extract Ids, and replaced by ResidenceRecherche__c,CreatedById,LastModifiedById,OwnerId in the query
• Then I saw that it also created __r fields columns in the output CSV, but with empty values
• After some debugging, I saw that the values were returned by the API, then replaced by undefined here, that's why I added this debug, because it's sad to get rid of values and extract empty columns whereas values are available :)

[hknokh]

Hey @nvuillam

• I've released your feature under the new RecordsFilter core Add-On: https://help.sfdmu.com/full-documentation/add-on-api/records-filter-core-add-on-module

• Also I've applied your suggested change regarding the undefined.

• The format of badwords.json should not be an array as you mentioned above: [ "whatever", "words", "or expressions", "here", "même accentués" ] => since it's incorrect json format and can't be parsed by JSON.parse() So it has been modified with: { "badwords" : [ "bad", "word" ] } (see the help)

• The rest of your parameters are left intact.

• I've reused your code which processes the transformations but arranged it to use the Core API, so the modifications of the main Sfdmu code were minor. Also I've mentioned your contribution in the relvant section of the Sfdmu Help )

---

Please update your forked version to use the latest Sfdmu. In common I highly suggest you to keep the plugin clone everytime updated because each release main contain fixes for major bugs.

[nvuillam]

That's great, thanks a lot ! I'll test that ASAP and I'll confirm you my use case works the same :)