SparkEventLogs table not populating

[Mpicca]

Overwatch Version 0.7.2.1 Using Azure.

Describe the bug Bronze_SparkEventLogs not populating

Screenshots

Full Validation Report:

[mohanbaabu1996]

Hi @Mpicca,

• How many workspaces have you configured ?
• Does all the cluster have logging path enabled and have necessary read, write permission ?
• Can you able to display the one of the logs from the overwatch master workspace ? display(dbutils.fs.head("cluster_logging_path"))

Please have an look into the documentation

[Mpicca]

@mohanbaabu1996 1 workspace configured All cluster have have logging path enabled. Unable to display 3 bullet point i receive the following: IllegalArgumentException: Cannot head a directory: dbfs:/mnt/storageaccountname/monitoring/overwatch

[mohanbaabu1996]

Thanks! Are you sure this path is the cluster logging path ?

dbfs:/mnt/storageaccountname/monitoring/overwatch

If yes, can you drill down the sub folders and check do you have .gz files

[mohanbaabu1996]

Hi @Mpicca

Did you get a chance to check the above test case ? Thank you!

[Mpicca]

@mohanbaabu1996 yes, the spark tables look to be populating now after multiple runs of the job. A new issue has arisen where the job looks to be completing but the gold tables are not processing / generating through the job, only bronze and silver: Is there any way we can get on a quick call?

[mohanbaabu1996]

Hi @Mpicca,

I'm from IST and let me know yours. Before getting into a call, can you please share the pipReport to get the answer ready.

select * from ETL_DB.pipeline_report order by Pipeline_SnapTs desc

Please share me to my ID: mohan.baabu@databricks.com

[Mpicca]

@mohanbaabu1996 i'm unable to share due to company policy. There are no failures in any modules. Will be happy to share on call. I'm free anytime, please let me know soonest time you are available.

[mohanbaabu1996]

Google Meet joining info Video call link: https://meet.google.com/ufy-psmm-kxk Or dial: ‪(US) +1 252-843-1062‬ PIN: ‪584 907 704‬# More phone numbers: https://tel.meet/ufy-psmm-kxk?pin=3437854930262

Can you please join here now ?

[mohanbaabu1996]

Let me know if this works for you

[joaoaugusto05]

What was the solution in this case? I'm facing the same problem :(

[mohanbaabu1996]

Hi @joaoaugusto05 What is the status message for 1006 moduleID in the pipeline_report?

select * from ETL_DB.pipeline_report order by Pipeline_SnapTs desc

[joaoaugusto05]

Hi @mohanbaabu1996, it was actually quite strange. The error message where, actually, that there was no new data.

I tried to run again but I got the same message. At the end, I solved rolling it back. It was a little expensive but worked :))