search

GoogleCloudPlatform / datacatalog-connectors-bi

Sample code with integration between Data Catalog and BI data sources.
Apache License 2.0
32 stars 16 forks source link

Ingest JAQL formula metadata #96

Closed ricardolsmendes closed 3 years ago

ricardolsmendes commented 3 years ago

- What I did Added features that enable the Sisense connector to ingest JAQL formula metadata. The formula field is available for JAQL queries with formulas.

- How I did it

  1. Added the DataCatalogEntryFactory.__make_column_schema_for_jaql_formula() method.
  2. Added the DataCatalogTagFactory.__make_tags_for_jaql_formula() method.
  3. Added unit tests to fully cover the new methods and required changes.

- How to verify it Run the unit tests and, if possible, the connector in an integrated environment to check the results.

- Description for the changelog Added features that enable the Sisense connector to ingest JAQL formula metadata.

PS: This PR is part of the effort to deliver #70.

ricardolsmendes commented 3 years ago

LGTM, just one comment... and overall I feel that the tags are becoming complex enough, should we have on the connector documentation a list of the tags, their description and what a user might expect to get from running this connector?

Thanks, @mesmacosta! I don't see the tags themselves becoming more complex. On the other hand, this connector creates much more Tags than the others, at least in the BI scope, to enable the column-level lineage mechanism. This PR addresses formulas, which are special fields of the JAQL objects, and maybe this is why you feel the tags are becoming complex. Although it adds more code to process JAQL metadata, the end result remains simple: tags with ~4 fields.

I like your suggestion on improving the documentation, so I've added the new 5. Templates, Tags, and Data Lineage section to README.md. Please let me know your thoughts.

Spoiler: the next PR will be very similar, adding the filter.by JAQL field. The last special field at least for now.