Fix segment issue in shifu init - Githubissues

ShifuML / shifu

An end-to-end machine learning and data mining framework on Hadoop

https://github.com/ShifuML/shifu/wiki

Apache License 2.0

249 stars 109 forks source link

Fix segment issue in shifu init #741

Closed Liu-Delin closed 3 years ago

Liu-Delin commented 3 years ago

Description

This fix is a following change of https://github.com/ShifuML/shifu/pull/739.

We cannot calculate non-segment column amount by below expression. Because we don't have any segment column in config when we do shifu init.

nonSegmentColumnAmount = columnAmountInConfig / (1+segmentSize)

Therefore, I add a new field in column config to mark the column as segment or not.

Tests

I manually tested init, stats, norm and train.