Sync Job connector snapshot JSON incorrect

[anghelnicolae]

Bug Description

The "filtering" field of the "connector" snapshot object should be an array, but when creating a new sync job from Kibana, the field is created as an object.

To Reproduce

Steps to reproduce the behavior:

1. Create index for any connector
2. Click on 'Sync'
3. Select 'Full' or 'Incremental'
4. Check the new document created in the '.elastic-connectors-sync-jobs' index

Expected behavior

The connector snapshot object should have a property "filtering" of type array, but it is an object.

Environment

8.11

[seanstory]

Hi @anghelnicolae - thanks for reporting an issue!

Can I get some clarifying information? What do you mean by "snapshot"? As far as I know, the only place we use "snapshot" in the connectors codebase is for versioning, and it shouldn't end up in the documents in .elastic-connectors-sync-jobs.

Because you mention filtering, it sounds like you might be talking about the sync rules portion of the document like:

{
  ...
  "connector": {
    ...
    "filtering": {
      "advanced_snippet": {
        "created_at": "2024-01-31T20:50:14.695Z",
        "updated_at": "2024-01-31T20:50:14.695Z",
        "value": {}
       },
       "rules": [
         {
           "created_at": "2024-01-31T20:50:14.695Z",
           "field": "_",
           "id": "DEFAULT",
           "order": 0,
           "policy": "include",
           "rule": "regex",
           "updated_at": "2024-01-31T20:50:14.695Z",
           "value": ".*"
         }
       ],
       "validation": {
         "errors": [],
           "state": "valid"
         }
       },
       ...
     },
     ...
 }

Is this what you're talking about? What is it that's indicating to you that the filtering value should be an array, and not an object? Are you getting an error when you run your connectors, or is there some documentation that says this should be an array, or are you just disagreeing with the design in general?

If you can share any snippets, logs, or screenshots to help us understand the issue you're having, that would be helpful.

[anghelnicolae]

Sorry for not being clear enough. In the documentation (https://github.com/elastic/connectors/blob/main/docs/CONNECTOR_PROTOCOL.md) the connector job index is supposed to have the following structure:

connector: { -> Connector snapshot filtering: [ -> Array of filtering rules, connectors use the first entry by default

When Kibana creates an entry in that index, it doesn't respect this structure. The "filtering" property is an object, not an array. My issue is that I'm using the .NET client and when I try to deserialize the job document I get an error because of this mismatch.

[seanstory]

Ah gotcha! Thanks for clarifying. I'm tempted to say this is just a documentation error, but let me check with the team to double-check that this is intentional.

[anghelnicolae]

Please note that the ".elastic-connectors" index entries do respect this schema.