search

kedro-org / kedro-starters

Templates for your Kedro projects.
Apache License 2.0
64 stars 59 forks source link

Use console logging by default in the `databricks-iris` starter #135

Closed jmholzer closed 1 year ago

jmholzer commented 1 year ago

Motivation and Context

Solves the problem that rich logging does not work when deploying projects as wheel files, but console logging does.

How has this been tested?

Manually on Databricks.

Checklist

jmholzer commented 1 year ago

Is there a particular issue link to this? I know there is a problem with click when deploying on Databricks, I am not familiar with problem of rich or maybe I just forgot.

Yes, unfortunately rich logging produces the following when a packaged Kedro project is run as a job:

Screenshot 2023-06-06 at 16 54 06

This is the output with console logging:

Screenshot 2023-06-06 at 17 06 00

jmholzer commented 1 year ago

I did a quick test and got weird results: on one hand, I already had this logging configuration in my project and I'm not sure why (I was using the starter I cloned before this PR was up), but also I'm still seeing HTML in the logs after launching a test workflow.

image

I ran out of time to debug this before the end of the week but I don't want to block the PR @jmholzer , hope you can find someone else to review this.

I think you potentially used the pyspark-iris starter for this. Did you also remember to copy the conf/ folder from local to DBFS before running this project? It is interesting that the rich output renders correctly for you when run as a project, which Databricks runtime version are you using? I never saw rich rendered correctly with < 13, but haven't tried >= 13.

astrojuanlu commented 1 year ago

It is interesting that the rich output renders correctly for you when run as a project, which Databricks runtime version are you using? I never saw rich rendered correctly with < 13, but haven't tried >= 13.

I'm still using DBR 12.2 LTS but the New Navigation UI for Databricks. Maybe that played a role?

jmholzer commented 1 year ago

Don't forget to update the release notes ✍️

@merelcht Did you mean to add this comment on the Kedro docs PR? I can't see a RELEASE.md for starters.

merelcht commented 1 year ago

Don't forget to update the release notes ✍️

@merelcht Did you mean to add this comment on the Kedro docs PR? I can't see a RELEASE.md for starters.

Whoops you're right! I mean to add it here: https://github.com/kedro-org/kedro-starters/releases Just so we know when this was updated 🙂