Open girishkumarbk opened 6 years ago
wdecay
commented
6 years ago The root cause is: ValueError: Only one class present in y_true. ROC AUC score is not defined in that case. It's likely that you don't have enough data to train the model on. Did you generate the data via DataGeneration.ipynb or use DataIngestion.ipynb?
girishkumarbk
commented
6 years ago Hi, I have used DataIngestion.ipnb for this exercise after data generator had pushed some level of data into IoThub.
1.0) How do we know that data is sufficiently ingested to start the training process ? 2.0) Also, could you please indicate on how to configure DataGeneration.ipynb generate training data directly on ABS ? How to configure DataGenerator to operate in these two modes 1.0) Push data to IoT hub (2) Generate data on ABS directly ?
Regards, /Girish BK
wdecay
commented
6 years ago DataGeneration.ipynb doesn't not push data directly to ABS. It writes it to the local hard drive (or DBFS if Databricks is used). Nothing prevents you from pushing this data into ABS with a few lines of code, of course, but that would be outside the scope of the solution.
Re: DataIngestion.ipynb, it's provided mainly for reference to enable a production scenario. If you use the solution AS-IS, the answer to your question (How do we know that data is sufficiently ingested to start the training process ?) is: It's never going to be enough, strictly speaking, unless you run the solution for at least several weeks and generate data for a reasonably large number of devices; in a more loose sense, however, you need to have both positive and negative data points, which means you need to have at least one device that had failed and at least one healthy device. The more the better, of course.
Hi,
I get this error in the model training playbook.
Area Under the Curve (AUC) AUC is the area under the receiver operating characteristic curve (ROC curve), which is 1.0 for ideal classifiers and 0.5 for those that do no better than random guessing. Let's compare the AUC score of the trained model with that of the dummy classifier.
roc_auc_score expects binarized labels
binarizer = LabelBinarizer() binarizer.fit(Y_train_res) Y_test_binarized = binarizer.transform(Y_test) def auc_score(y_true, y_pred): return roc_auc_score(binarizer.transform(y_true), binarizer.transform(y_pred), average='macro') print('ROC AUC scores') print('Trained model: {0}\nDummy classifier: {1}'.format(auc_score(Y_test, Y_predictions), auc_score(Y_test, Y_dummy))) ROC AUC scores
ValueError Traceback (most recent call last)