hamham223
commented
1 year ago Update on error 2, array not equal
The 2 predicted array are not equal because the input dataframes are not the same.
rdd_map = rdd.map(lambda x: (DenseVector(np.random.randn(1, ).astype(np.float32)),
int(np.random.randint(0, 2, size=())),
os.getpid()))
df = rdd_map.toDF(["feature", "label", "pid"]) # this is lazy by defaultExperiment 1
With the config
conf = {"spark.python.worker.reuse": "false"}
sc = init_orca_context(conf=conf)
Every time a new process is created, with the same random seed, df.collect() always returns the same value, and therefore the predict can give the same result.
before_res = trainer.predict(df, feature_cols=["feature"]).collect()
expect_res = np.concatenate([part["prediction"] for part in before_res])
trainer.load(os.path.join(temp_dir, "cifar10_savemodel"))
# continous predicting
after_res = trainer.predict(df, feature_cols=["feature"]).collect()
pred_res = np.concatenate([part["prediction"] for part in after_res])
assert np.array_equal(expect_res, pred_res)Experiement 2
Without the config
sc = init_orca_context()
Without the config, it will use a fixed number of Python workers (i.e. always produce random number in the same processes), so the dataframe will be different in two predicts and thus the results are not the same.
Reference:
- https://spark.apache.org/docs/latest/configuration.html#runtime-environment
- crtl+f search for
spark.python.worker.reuse
Problem
Some tensorflow training unit tests require the following extra config in
conftest.py, especially for spark backend. Some ray backend uts also use this config.Typical Errors
Removing the above config may cause the following kinds of errors:
How-to reproduce the problem
remove the above config in:
and run orca pytests
Current Solutions