Update hub.json

[VigneshwaranJeyakumar]

Added data profiler package

Description

Tell us about your new package!

Checklist

This checklist is a cut down version of the best practices that we have identified as the package hub has grown. Although meeting these checklist items is not a prerequisite to being added to the Hub, we have found that packages which don't conform provide a worse user experience.

First run experience

• [x] The package includes a README which explains how to get started with the package and customise its behaviour
• [x] The README indicates which data warehouses/platforms are expected to work with this package

Customisability

• [x] The package uses ref or source, instead of hard-coding table references.

  Packages for data transformation (delete if not relevant):

• [x] provide a mechanism (such as variables) to customise the location of source tables.
• [x] do not assume database/schema names in sources.

Dependencies

Dependencies on dbt Core

• [x] The package has set a supported require-dbt-version range in dbt_project.yml. Example: A package which depends on functionality added in dbt Core 1.2 should set its require-dbt-version property to [">=1.2.0", "<2.0.0"].

  Dependencies on other packages defined in packages.yml:

• [x] Dependencies are imported from the dbt Package Hub when available, as opposed to a git installation.
• [x] Dependencies contain the widest possible range of supported versions, to minimise issues in dependency resolution.
• [x] In particular, dependencies are not pinned to a patch version unless there is a known incompatibility.

  Interoperability

• [x] The package does not override dbt Core behaviour in such a way as to impact other dbt resources (models, tests, etc) not provided by the package.
• [x] The package uses the cross-database macros built into dbt Core where available, such as {{ dbt.except() }} and {{ dbt.type_string() }}.
• [x] The package disambiguates its resource names to avoid clashes with nodes that are likely to already exist in a project. For example, packages should not provide a model simply called users.

Versioning

• [x] (Required): The package's git tags validates against the regex defined in version.py
• [x] The package's version follows the guidance of Semantic Versioning 2.0.0. (Note in particular the recommendation for production-ready packages to be version 1.0.0 or above)

[joellabes]

Hey @vigneshj23, thanks for opening this!

I see in the README that you say it's based on the dbt_profiler project but with improved performance, did you consider submitting your changes as a PR to improve the original project? That would reduce the confusion for users trying to choose one package over another.

Let me know if I've misunderstood the approach you've taken

[VigneshwaranJeyakumar]

Hey @joellabes, thanks for the response

Actually, we were just inspired by the existing dbt-profiler package, but we enhanced it. These features and strategies have an entirely different underlying logic from the one that is currently available.

Features:

• We applied the Chunk methodology as well as the Incremental method (Create and Insert operations). As a result, it will not allow more than 10K lines of code. On some platforms, more than 10k line single query will lead to memory problems.
• It will produce a single data profile table for various tables, such as whole schema tables and multiple schema tables.

If we change the core logic of the present one, the entire package will be affected. As a result, we believe that if the user wants to profile many tables, each with more than 100 columns, can utilize our package.

[joellabes]

OK! Thanks for clarifying 🙏