Open exalate-issue-sync[bot] opened 1 year ago
Wendy Wong commented: Duplication here: https://h2oai.atlassian.net/browse/PUBDEV-8979
Wendy Wong commented: This basically means that we will run one test in one hardware setup with activeProcessorCount set to a number. Then, we will run another test in another hardware setup with activeProcessorCount set to the same number. Here are some python code you can use to do the run:
Okay, I will propose to run this test to check for reproducibility across different hardware setups:
{noformat}from future import division from builtins import range import sys import h2o import tempfile from h2o.estimators.gbm import H2OGradientBoostingEstimator
def extract_from_twoDimTable(metricOfInterest, fieldOfInterest, takeFirst=False): """ Given a fieldOfInterest that are found in the model scoring history, this function will extract the list of field values for you from the model.
:param aModel: H2O model where you want to extract a list of fields from the scoring history
:param fieldOfInterest: string representing a field of interest.
:return: List of field values or None if it cannot be found
"""
allFields = metricOfInterest._col_header
return extract_field_from_twoDimTable(allFields, metricOfInterest.cell_values, fieldOfInterest, takeFirst=False)
def extract_field_from_twoDimTable(allFields, cell_values, fieldOfInterest, takeFirst=False): if fieldOfInterest in allFields: cellValues = [] fieldIndex = allFields.index(fieldOfInterest) for eachCell in cell_values: cellValues.append(eachCell[fieldIndex]) if takeFirst: # only grab the result from the first iteration. break return cellValues else: return None
def compare_frame_one_column(f1, f2, tol=1e-6): temp1 = f1.as_data_frame(use_pandas=False) temp2 = f2.as_data_frame(use_pandas=False)
for rowInd in range(1, f1.nrow):
v1 = float(temp1[rowInd][0])
v2 = float(temp2[rowInd][0])
diff = abs(v1 - v2) / max(1.0, abs(v1), abs(v2))
assert diff <= tol, "Failed frame values check at row {2} and column {3}! frame1 value: {0}, column name: {4}." \
" frame2 value: {1}, column name:{5}".format(temp1[rowInd][0], temp2[rowInd][0],
rowInd, 0, f1.names[0], f2.names[0])
fr = h2o.import_file("https://h2o-public-test-data.s3.amazonaws.com/bigdata/laptop/covtype/covtype.full.csv")
m = H2OGradientBoostingEstimator(seed=1234, score_tree_interval=2) m.train(x=list(range(0, 12)), y="Cover_Type", training_frame=fr) pred = m.predict(fr)
h2o.download_csv(pred, "/some/dictory/pred.csv") relative_importance = extract_from_twoDimTable(m._model_json["output"]["variable_importances"], "relative_importance", takeFirst=False)
print(relative_importance)
pred = h2o.import_file("/path/to/pred.csv") pred2 = h2o.import_file("/path/to/pred2.csv") for index in range(1, pred.ncols): compare_frame_one_column(pred[index], pred2[index]){noformat}
JIRA Issue Details
Jira Issue: PUBDEV-8988 Assignee: Arun Aryasomayajula Reporter: Wendy Wong State: Reopened Fix Version: N/A Attachments: N/A Development PRs: N/A
Wendy Wong commented: For new H2O-3 versions 3.35.0.5 or newer, Michalk has completed this JIRA: [https://h2oai.atlassian.net/browse/PUBDEV-8425|https://h2oai.atlassian.net/browse/PUBDEV-8425|smart-link] to ensure reproducibility.
Need to show how this works and add documentation on it [https://h2oai.atlassian.net/browse/PUBDEV-8425|https://h2oai.atlassian.net/browse/PUBDEV-8425|smart-link] to ensure reproducibility?