Include/Exclude Filtering Not excluding field

[thenickot2]

Submitting what I believe is a bug with the latest version 1.0.0beta3. Below is my torodb-stampede.yml file:

---
logging: {}
metricsEnabled: false
replication:
  replSetName: "XXX"
  syncSource: "XXX"
  ssl:
    enabled: false
    allowInvalidHostnames: false
    fipsMode: false
  auth:
    mode: "negotiate"
    user: "XXX"
    source: "admin"
  mongopassFile: "/etc/torodb-stampede/.mongopass"
  include:
    database_name:
      firstCollection: "*"
      secondCollection: "*"
  exclude:
    database_name:
      firstCollection:
        - name: "_field"
      secondCollection:
        - name: "_field"
backend:
  pool:
    connectionPoolTimeout: 10000
    connectionPoolSize: 30
  postgres:
    host: "XXX"
    port: 5432
    database: "XXX"
    user: "XXX"
    toropassFile: "/etc/torodb-stampede/.toropass"
    applicationName: "toro"
    ssl: false

For two of my collections, I'm trying to include every field, except for "_field". This field begins with an underscore and is an object that I don't want ToroDB to read/parse.

Following the last example in https://www.torodb.com/stampede/docs/1.0.0-beta3/configuration/filtered-replication/ it appears this is the correct configuration?

From the logs: Nov 03 01:09:51 XXX torodb-stampede[XXX]: 2017-11-03T01:09:51.883 INFO BACKEND - Created internal index did_pid_seq_idx for table firstcollection__field_role_xxx And in postgres, I can see the data being inserted.

Any help is appreciated!

[teoincontatto]

Hello @thenickot2,

Unfortunately ToroDB Stampede is not able to filter out a single field, it can exclude an entire database, an entire collection or an index.

The configuration you are using filter out indexes with name "_field" from the included collections.

Also the log you posted is about the creation of an internal index. Internal indexes are always created by ToroDB since they are required for read operations to retrieve the document.

[thenickot2]

Hi @teoincontatto That makes complete sense, thank you for clarifying