databrickslabs / overwatch

Capture deep metrics on one or all assets within a Databricks workspace
Other
221 stars 60 forks source link

Show `DataFrame` records in logs #1224

Open neilbest-db opened 1 month ago

neilbest-db commented 1 month ago

See doc string in DataFrameSyntax.scala:

  /**
    * Implements extension methods for Spark `Dataset[T]`s (including
    * `DataFrame`s) that mimic Spark's built-in `Dataset.show()`
    * family of overloaded methods.
    *
    * Instead of producing output on the console, they return an
    * `Iterator[String]` suitable for redirecting output to a logger,
    * for example, like this:
    *
    * {{{
    * debugDF
    *   .showLines( debugDF.count.toInt, 0, true)
    *   .foreach( logger.log( Level.INFO, _))
    * }}}
    *
    * Produces log entries according to your logger configuration:
    *
    * {{{
    * 24/05/04 00:18:16 INFO Silver: -RECORD 0---------------------------------
    * 24/05/04 00:18:16 INFO Silver:  organization_id       | 2753962522174656
    * 24/05/04 00:18:16 INFO Silver:  jobId                 | 903015066329560
    * 24/05/04 00:18:16 INFO Silver:  fromMS                | 1709419775381
    * . . .
    * 24/05/04 00:18:16 INFO Silver: -RECORD 1---------------------------------
    * . . .
    * }}}
    */
sonarcloud[bot] commented 1 month ago

Quality Gate Passed Quality Gate passed

Issues
1 New issue
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
0.0% Duplication on New Code

See analysis details on SonarCloud

neilbest-db commented 4 weeks ago

Pasted docstring into initial comment above for reference.

neilbest-db commented 3 weeks ago

@sriram251-code, @souravbaner-da, do you need more info before we move ahead with this? I don't think it's right for me to mark the conversations resolved . . . are you satisfied or still have doubts?

sriram251-code commented 3 weeks ago

I am fine with the comments