Open Manjesh80 opened 4 years ago
Added JSON support by method QNL.getStats_json. Usage shown below
**** SET-UP ** QNL = sc._jvm.com.qubole.sparklens.QuboleNotebookListener.registerAndGet(sc._jsc.sc()) import time if (QNL.estimateSize() > QNL.getMaxDataSize()): QNL.purgeJobsAndStages() startTime = int(round(time.time() * 1000)) **** SET-UP **
SPARK-CODE
**** END ** endTime = int(round(time.time() * 1000)) time.sleep(QNL.getWaiTimeInSeconds()) stats_json = QNL.getStats_json(startTime, endTime) print(stats_json)
**** END **
Added JSON support by method QNL.getStats_json. Usage shown below
**** SET-UP ** QNL = sc._jvm.com.qubole.sparklens.QuboleNotebookListener.registerAndGet(sc._jsc.sc()) import time if (QNL.estimateSize() > QNL.getMaxDataSize()): QNL.purgeJobsAndStages() startTime = int(round(time.time() * 1000)) **** SET-UP **
SPARK-CODE
**** END ** endTime = int(round(time.time() * 1000)) time.sleep(QNL.getWaiTimeInSeconds()) stats_json = QNL.getStats_json(startTime, endTime) print(stats_json)
Save to FIle, Print will truncate the data
**** END **