search

dbt-labs / dbt-core

dbt enables data analysts and engineers to transform their data using the same practices that software engineers use to build applications.
https://getdbt.com
Apache License 2.0
9.96k stars 1.63k forks source link

Detailed log data #3029

Closed rpammeraal closed 3 years ago

rpammeraal commented 3 years ago

It would be great to have access to the logs that DBT in the cloud creates. Currently, there is none.

jtcohen6 commented 3 years ago

Hey @rpammeraal, it should be possible to access the debug-level logs that dbt Cloud creates when running a specific job, You can view the last 1000 lines in browser, or click the "Debug Logs" button to download the full logs:

Screen Shot 2021-01-26 at 11 25 02 AM

If you're instead referring to debug-level logs when developing in the dbt Cloud IDE, those should available (as they are on the CLI) in the logs/ folder, or whatever your log-path is.

In any event, I'm going to close this issue because it relates to dbt Cloud, not dbt Core (the code in this repository). If you have questions about dbt Cloud, your best bet is to contact support from within the application.

rpammeraal commented 3 years ago

Hi Jeremy,

Thanks for your answer. I am looking for a way to automate this, i.e. download this data automatically. The goal is to parse this data and put this into one or more database tables, create reports and alerts on it, so we can proactively take action if the population of a DBT model fails or take too long. Is there a way to do this?

Thanks, Roy

On Tue, Jan 26, 2021 at 2:28 AM Jeremy Cohen notifications@github.com wrote:

Closed #3029 https://github.com/fishtown-analytics/dbt/issues/3029.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/fishtown-analytics/dbt/issues/3029#event-4249978428, or unsubscribe https://github.com/notifications/unsubscribe-auth/AGE35HGGYUSNI7IAMGMNNELS32KOHANCNFSM4WSETBIA .

jtcohen6 commented 3 years ago

@rpammeraal Interesting! I don't believe there's a way to do this using dbt Cloud today, but it touches on a few long-term, pie-in-the-sky conversations we've had about real-time log streaming or alerting via integrations with tools like DataDog, PagerDuty, etc.

In the meantime:

rpammeraal commented 3 years ago

Yes! At Peerstreet, we continue to be amazed by the rich feature set that DBT provides. I personally just found out about the test feature in DBT. In the past I created a mechanism that does exactly the same. But yes, if we were able to get real time data on runtimes, it'll provides us a way to proactively adjust models (add indexes et al) if they start to become slow in population.

Thanks for the feedback Jeremy,

Roy

On Thu, Jan 28, 2021 at 3:52 AM Jeremy Cohen notifications@github.com wrote:

@rpammeraal https://github.com/rpammeraal Interesting! I don't believe there's a way to do this using dbt Cloud today, but it touches on a few long-term, pie-in-the-sky conversations we've had about real-time log streaming or alerting via integrations with tools like DataDog, PagerDuty, etc.

In the meantime:

  • dbt Cloud can alert you if a job fails because a model fails to build
  • All dbt Cloud jobs support a "run timeout" parameter
  • For longitudinal analysis of model build times, dbt artifacts can play a role

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/fishtown-analytics/dbt/issues/3029#issuecomment-769002177, or unsubscribe https://github.com/notifications/unsubscribe-auth/AGE35HB4J3DTM5LSDXFEVHLS4FFXJANCNFSM4WSETBIA .

pdr commented 2 years ago

@jtcohen6 Just stumbled upon this thread looking for the exact same thing that @rpammeraal is asking for. We're starting to use DBT for more and more critical things and not being able to centralize the logs somewhere is a growing concern. Is there any update on this? I know it said it was "pie in the sky" but this doesn't feel too pie in the sky to me!!!

Shipping logs to a tool like DataDog is actually quite simple. All you would need is a place for people to put in their log endpoint and DBT Cloud would send POST requests to that endpoint.

pdr commented 2 years ago

In any event, I'm going to close this issue because it relates to dbt Cloud, not dbt Core (the code in this repository). If you have questions about dbt Cloud, your best bet is to contact support from within the application.

Ah I just saw this ... apologies. We will reach out to support as well. Thanks.