intel-analytics / BigDL-trainings

Training materials for BigDL
Apache License 2.0
13 stars 16 forks source link

Lambda function is not working in py36+ #18

Closed AjayJadhavDS closed 4 years ago

AjayJadhavDS commented 4 years ago

Working function

def array2rdd(ds):

build Sample from ndarrays

def build_sample(balance,sex,education,marriage,age,default):
    feature = np.array([balance,sex,education,marriage,age]).flatten()
    label = np.array(default)
    return Sample.from_ndarray(feature, label)

rdd = ds.map(lambda X: build_sample(X[0],X[1],X[2],X[3],X[4],X[5]))
return rdd

cc_rdd_train = array2rdd(trainingData.rdd.map(list)) cc_rdd_train.cache() cc_rdd_train.count()

cc_rdd_test = array2rdd(validData.rdd.map(list)) cc_rdd_test.cache() cc_rdd_test.count()