Transform+Split in Runner

neeleshdodda44 commented 5 years ago

Major changes:

Runner

Transform and training-test split now takes places as a new type of job: transform_and_split in the method _transform_and_split_data.
Corresponding enums have been created
Tests have been updated to test transform/split functionality, as well as test training/testing off of transformed and split data.

DMLJob

New attributes added for raw_filepath (previously rfp), session_filepath, and transform_function.
serialize_job and deserialize_job have been updated accordingly.

Testing utils

Created a new utils for transforming and splitting data.
For tests that facilitate training + validation outside of Runner tests, I just had added the session filepath manually to the initialize job. Although transform and splitting for all of these tests is not necessary, we need to implement transforming and splitting data in the optimizer logic. This should be a separate PR, maybe assigned to @kiddyboots216?

Dataset Manager

Refactored code, tests and documentation so that it is clear the Dataset Manager does not handle transforms any more.

neeleshdodda44 commented 5 years ago

@georgymh Can you review this PR soon so I don't have to keep fixing merge conflicts from recently merged PRs 😢

neeleshdodda44 commented 5 years ago

Addressed all of Georgy's comment, so I'm merging now.

georgymh / decentralized-ml