Logs page freeze / lag when dbt plugin generates a lot of models results

kestra-io / kestra

Infinitely scalable, event-driven, language-agnostic orchestration and scheduling platform to manage millions of workflows declaratively in code.

https://kestra.io

Apache License 2.0

7.63k stars 463 forks source link

Logs page freeze / lag when dbt plugin generates a lot of models results #1885

Closed aballiet closed 11 months ago

aballiet commented 1 year ago

Expected Behavior

We expect the logs section of dbt cloud trigger run to load in a reasonable amount of time.

Actual Behaviour

A lot of timeout and laggy page / blocked / crashing.

Steps To Reproduce

Run large project dbt cloud trigger run from Kestra

Environment Information

Kestra Version: 0.11.0-SNAPSHOT
Operating System (OS / Docker / Kubernetes): Kubernetes
Java Version (If not docker):

Example flow

No response

anna-geller commented 1 year ago

trying to understand the issue better: when you say logs, do you mean the parsing of the manifest to display model names is what generates some extra lag, correct @aballiet?

Is this a feature that is required in your use case? I wonder whether we should provide a property to allow skipping model parsing. WDYT?

We also have that open issue to allow rendering those in the topology view, which could be friendlier than displaying those at the end of the logs https://github.com/kestra-io/plugin-dbt/issues/38

aballiet commented 1 year ago

Did a quick Loom video to show the bug : https://www.loom.com/share/0556984bd9564b328ea4cf413954a514

A fix is required because currently clicking on "logs" freezes the web browser :(

Yes I think displaying just the logs like we have in the Gantt is fine. No need to display one block per model. Might be nice though to list all warning models / tests and in case of error, highlight the model responsible of error.

They have this system on dbt cloud :

Skraye commented 1 year ago

Hi @aballiet ! Could you show us your browser console? That could leverage if the issue is coming from the front or the back

edit: You may need to open the console before the crash occurred

aballiet commented 1 year ago

Nothing in console :/

anna-geller commented 1 year ago

Note from the team: It seems to be a frontend issue -- we create so many dynamic tasks for DBT that it can be challenging to handle that in the UI with thousands of models.

brian-mulier-p commented 11 months ago

Hello @aballiet can you try again with the latest SNAPSHOT version which includes this ? I think it will fix the issue :+1:

brian-mulier-p commented 11 months ago

Also maybe it's because you have the "expand all" settings for logs which make the page display everything at once, might help with FAILED only or collapse all at first. Still you can try it as-is with the new version I made a lot of improvement on the page