search

dremio-professional-services / dremio-cloner

27 stars 20 forks source link

space.folder.filter.paths config is not being respected while performing reads #30

Closed nshetty15 closed 1 year ago

nshetty15 commented 1 year ago

Below config filter path is not being applied when performing a get operation using config_read_dir.json {"space.folder.filter.paths": [""]},

For example, Let's say I have a space named 'my_space' and within that space I have a folder called 'trades'. Using below config, dremio-cloner should only pull the objects located within this folder.

    {"space.filter": "*"},
    {"space.filter.names": ["my_space"]},
    {"space.exclude.filter": ""},
    {"space.folder.filter": "*"},
    {"space.folder.filter.paths": ["trades"]},
    {"space.folder.exclude.filter":""},

However right now it is pulling all the objects located in the space.

nshetty15 commented 1 year ago

Looks like it's because of dependencies b/w the VDS's. I was able to get the expected result after turning off dependencies.process_mode. (Changed get toignore) {"vds.dependencies.process_mode":"ignore"},