dat-labs / verified-sources

Repository to hold the supported verified sources (data warehouses, databases, cloud file system, local file system, FTP, SFTP, etc).
Apache License 2.0
1 stars 4 forks source link

Wrong Catalog parameters getting passed #34

Open pandeyganesh-dc opened 1 month ago

pandeyganesh-dc commented 1 month ago

Is there an existing issue for this?

Description

For some connection, If I choose SplitByHtmlHeaderSettings and later decide to edit configuration and choose SplitByCharacterSettings, other options remain in the catalog like headers_to_split_on and doesn't get options like seperator

Sample catalog dict :

"catalog": {
        "document_streams": [
            {
                "name": "html",
                "json_schema": null,
                "namespace": "idk4",
                "read_sync_mode": "FULL_REFRESH",
                "write_sync_mode": "APPEND",
                "cursor_field": "",
                "advanced": {
                    "splitter_settings": {
                        "splitter_settings": "SPLIT_BY_CHARACTER",
                        "headers_to_split_on": [
                            "h2",
                            "h3"
                        ]
                    }
                },
                "usr_file_path": "dummy",
                "obj_file_path": "kaushiki.html"
            }
        ]
    },

So, I have to make a new connection.

Steps to Reproduce

It would reproduce the error.

Screenshots

image

Environment

Additional Context

No response