VowpalWabbit / coba

Contextual bandit benchmarking
https://coba-docs.readthedocs.io/
BSD 3-Clause "New" or "Revised" License
48 stars 19 forks source link

wip: save 'online' log to test 'offline' scenarios #23

Closed lalo closed 1 year ago

lalo commented 1 year ago

Hey Mark, I wanted to keep the human readable name even after flattening the categorical type features. I make a big assumption rows.py, left it as a comment.

Could it be implemented as a Filter for the Environment? although I would still need to modify the Finalize call in process.py. Although it is a combination of Environment + first predict call to Learner, since we need access to label.

environments = cb.Environments.cache_dir('.coba_cache').from_openml(180,take=1500).shuffle(n=3).save_copy()
learners     = [cb.VowpalLearner(args="--cb_explore_adf --epsilon 0.05")]
experiment   = cb.Experiment(environments, learners)

experiment.run(quiet=False).plot_learners(err='se')

Sample log from openml(180): {"context": {"x": [["elevation", 3257.0], ["aspect", 83.0], ["slope", 20.0], ["horizontal_distance_to_hydrology", 0.0], ["Vertical_Distance_To_Hydrology", 0.0], ["Horizontal_Distance_To_Roadways", 1528.0], ["Hillshade_9am", 243.0], ["Hillshade_Noon", 201.0], ["Hillshade_3pm", 78.0], ["Horizontal_Distance_To_Fire_Points", 1794.0], ["wilderness_area1", 1.0], ["wilderness_area2", 0.0], ["wilderness_area3", 0.0], ["wilderness_area4", 0.0], ["soil_type_1_0", 1], ["soil_type_1_1", 0], ["soil_type_2_0", 1], ["soil_type_2_1", 0], ["soil_type_3_0", 1], ["soil_type_3_1", 0], ["soil_type_4_0", 1], ["soil_type_4_1", 0], ["soil_type_5_0", 1], ["soil_type_5_1", 0], ["soil_type_6_0", 1], ["soil_type_6_1", 0], ["soil_type_7_0", 1], ["soil_type_7_1", 0], ["soil_type_8_0", 1], ["soil_type_8_1", 0], ["soil_type_9_0", 1], ["soil_type_9_1", 0], ["soil_type_10_0", 1], ["soil_type_10_1", 0], ["soil_type_11_0", 1], ["soil_type_11_1", 0], ["soil_type_12_0", 1], ["soil_type_12_1", 0], ["soil_type_13_0", 1], ["soil_type_13_1", 0], ["soil_type_14_0", 1], ["soil_type_14_1", 0], ["soil_type_15_0", 1], ["soil_type_15_1", 0], ["soil_type_16_0", 1], ["soil_type_16_1", 0], ["soil_type_17_0", 1], ["soil_type_17_1", 0], ["soil_type_18_0", 1], ["soil_type_18_1", 0], ["soil_type_19_0", 1], ["soil_type_19_1", 0], ["soil_type_20_0", 1], ["soil_type_20_1", 0], ["soil_type_21_0", 1], ["soil_type_21_1", 0], ["soil_type_22_0", 1], ["soil_type_22_1", 0], ["soil_type_23_0", 1], ["soil_type_23_1", 0], ["soil_type_24_0", 1], ["soil_type_24_1", 0], ["soil_type_25_0", 1], ["soil_type_25_1", 0], ["soil_type_26_0", 1], ["soil_type_26_1", 0], ["soil_type_27_0", 1], ["soil_type_27_1", 0], ["soil_type_28_0", 1], ["soil_type_28_1", 0], ["soil_type_29_0", 1], ["soil_type_29_1", 0], ["soil_type_30_0", 1], ["soil_type_30_1", 0], ["soil_type_31_0", 1], ["soil_type_31_1", 0], ["soil_type_32_0", 1], ["soil_type_32_1", 0], ["soil_type_33_0", 1], ["soil_type_33_1", 0], ["soil_type_34_0", 1], ["soil_type_34_1", 0], ["soil_type_35_0", 1], ["soil_type_35_1", 0], ["soil_type_36_0", 1], ["soil_type_36_1", 0], ["soil_type_37_0", 1], ["soil_type_37_1", 0], ["soil_type_38_0", 0], ["soil_type_38_1", 1], ["soil_type_39_0", 1], ["soil_type_39_1", 0], ["soil_type_40_0", 1], ["soil_type_40_1", 0]]}, "adfs": [{"a": [1, 0, 0, 0, 0, 0, 0]}, {"a": [0, 1, 0, 0, 0, 0, 0]}, {"a": [0, 0, 1, 0, 0, 0, 0]}, {"a": [0, 0, 0, 1, 0, 0, 0]}, {"a": [0, 0, 0, 0, 1, 0, 0]}, {"a": [0, 0, 0, 0, 0, 1, 0]}, {"a": [0, 0, 0, 0, 0, 0, 1]}], "labels": ["1:-1:0.14286", null, null, null, null, null, null], "label": "1:-1:0.14286"}

lalo commented 1 year ago

https://github.com/VowpalWabbit/coba/blob/4c9cbd3ef5daceeae5be7c9b26a0a3f901279814/coba/tests/test_environments_core.py#L659