search

ULTR-Community / ULTRA_pytorch

Unbiased Learning To Rank Algorithms (ULTRA)
https://ultr-community.github.io/ULTRA_pytorch/
Apache License 2.0
93 stars 8 forks source link

Confusion for example in main.py #4

Closed rowedenny closed 2 years ago

rowedenny commented 2 years ago

Hi,

Thanks for open source this amazing work.

I have a question for the current example in main.py. Specifcially, for the following code,

https://github.com/ULTR-Community/ULTRA_pytorch/blob/ec4fe329e4239b588a940cb4bcdd6a321aade679/main.py#L92

If I understand correctly, is this should be ultra.utils.find_class(exp_settings['valid_input_feed']).preprocess_data(valid_set, exp_settings['valid_input_hparams'], exp_settings)

since for the valid set, we would like to evaluate based on the human annotation for LTR, thus the key should be valid_input_feed instead of train_input_feed, and for the same reason, the key for the second parameter shall be altered as well.

In addition, should be the test_set need to altered correspondingly.

Thanks for your time and consideration.

QingyaoAi commented 2 years ago

Thanks for the question. Actually the "preprocess_data" is mean to be a function for preprocessing the input data before creating the input_feed, so all train/valid/test data should be preprocessed with the same "preprocess_data" function to make the models work. Here we use the "preprocess_data" function of train_input_feed to preprocess all data.

In fact, we haven't implemented any "preprocess_data" function yet (as can be seen in base_input_feed.py). In the future, we may refactor the code to create a separate data preprocessing class to avoid confusion.

rowedenny commented 2 years ago

I see and thanks for the clarification.