refactor loading in features

LSSTDESC / RESSPECT

The RESSPECT project is a result from an inter-collaboration agreement established between the Cosmostatistics Initiative (COIN) and the LSST Dark Energy Science Collaboration (DESC) with the goal of developing a recommendation system for telescope resource allocation able to optimize photometric supernova cosmology anaylsis.

MIT License

1 stars 0 forks source link

Change Description

Finishes up the refactor of reading and writing feature files by moving some of the file loading in database.py to the feature handling module. Resolves #78 . Everything should be ready now to start implementing loading features to and from MongoDB.

[x] My PR includes a link to the issue that I am addressing

Solution Description

Takes the loading code that we had in the load_features_from_file method and moves it feature_handling_utils.py. Maintains all the options for fileloading (even though we've moved to a more unified version of writing out features) to ensure backwards compatibility.

Code Quality

[x] I have read the Contribution Guide
[x] My code follows the code style of this project
[x] My code builds (or compiles) cleanly without any errors or warnings
[x] My code contains relevant comments and necessary documentation

Before [93de9077]	After [75694c8a]	Ratio	Benchmark (Parameter)
149±0.6ms	153±2ms	1.03	benchmarks.time_learn_loop('KNN', 'UncSampling')
195M	197M	1.01	benchmarks.peakmem_learn_loop('KNN')
186M	188M	1.01	benchmarks.peakmem_learn_loop('RandomForest')
138±2ms	140±2ms	1.01	benchmarks.time_feature_creation
2.58±0.03s	2.60±0.02s	1.01	benchmarks.time_learn_loop('RandomForest', 'UncSampling')
151±0.8ms	151±0.8ms	1	benchmarks.time_learn_loop('KNN', 'RandomSampling')
2.58±0.01s	2.58±0.02s	1	benchmarks.time_learn_loop('RandomForest', 'RandomSampling')

Before [93de9077]

After [75694c8a]

Ratio

Benchmark (Parameter)

149±0.6ms

153±2ms

1.03

benchmarks.time_learn_loop('KNN', 'UncSampling')

195M

197M

1.01

benchmarks.peakmem_learn_loop('KNN')

186M

188M

1.01

benchmarks.peakmem_learn_loop('RandomForest')

138±2ms

140±2ms

1.01

benchmarks.time_feature_creation

2.58±0.03s

2.60±0.02s

1.01

benchmarks.time_learn_loop('RandomForest', 'UncSampling')

151±0.8ms

benchmarks.time_learn_loop('KNN', 'RandomSampling')

2.58±0.01s

2.58±0.02s

benchmarks.time_learn_loop('RandomForest', 'RandomSampling')

LSSTDESC / RESSPECT