Looker parsing improvements for liquid templating and view/model aliasing

sam-mccarty-mavenclinic commented 1 month ago

I would like to propose two improvements to resolve issues with Looker ingestion, one with liquid template variables in sql_table_name and another with aliased views and explores

Describe your changes:

Fixes #17911

I added python-liquid as a requirement
I added a call to try to render the sql table name as a liquid template before passing it to build the lineage request so that it may render the template as the base condition
I added a check for a non-empty from_ attribute for views to replace the name while building the lineage to correctly map to the actual view being joined
I replaced the existing conditions with a check for a non-empty view_name on the model so that the underlying view it is based on is included as part of the lineage, regardless of whether there are other views joined

#

Type of change:

[ ] Bug fix
[ ] Improvement
[ ] New feature
[ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
[ ] Documentation

#

Checklist:

[x] I have read the CONTRIBUTING document.
[ ] My PR title is Fixes <issue-number>: <short explanation>
[ ] I have commented on my code, particularly in hard-to-understand areas.
[ ] For JSON Schema changes: I updated the migration scripts or explained why it is not needed.

github-actions[bot] commented 1 month ago

Hi there 👋 Thanks for your contribution!

The OpenMetadata team will review the PR shortly! Once it has been labeled as safe to test, the CI workflows will start executing and we'll be able to make sure everything is working as expected.

Let us know if you need any help!

harshach commented 1 month ago

@sam-mccarty-mavenclinic thanks for the contribution

sam-mccarty-mavenclinic commented 1 month ago

@harshach no problem, i was making some tweaks to see if i could get the lineage fixed for our specific looker implementation, but i am open to more generic approaches to these issues if this initial solution is a bit too opinionated

sushi30 commented 1 month ago

Hi @sam-mccarty-mavenclinic, thank you for the PR.

You didn't submit the change to setup.py therefore the errors in the CI:

 E   ModuleNotFoundError: No module named 'liquid'

Can you give an example of this scenario? Maybe an example liquid template that we can test?

sam-mccarty-mavenclinic commented 1 month ago

One sec, I will push a fix to the setup file. Happy to add a test with sample file with the templating included, can you point me to the right location to add my test?

sushi30 commented 1 month ago

Take a look here.

sushi30 commented 1 month ago

@sam-mccarty-mavenclinic I merged the branch with main bc we had some failing tests that were fixed. If the CI is green we can merge this.

sam-mccarty-mavenclinic commented 1 month ago

@sam-mccarty-mavenclinic I merged the branch with main bc we had some failing tests that were fixed. If the CI is green we can merge this.

Thank you, I've been on a few other things but will check on any new failures today now that the unrelated failures have been cleared up

sonarcloud[bot] commented 1 month ago

Quality Gate passed for 'open-metadata-ingestion'

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
50.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarCloud

open-metadata / OpenMetadata