search

ULTR-Community / ULTRA_pytorch

Unbiased Learning To Rank Algorithms (ULTRA)
https://ultr-community.github.io/ULTRA_pytorch/
Apache License 2.0
93 stars 8 forks source link

KeyError: 'selection_bias_cutoff' in example/toy/offline_exp_pipeline.sh #2

Closed hideojoho closed 2 years ago

hideojoho commented 3 years ago

Hi. Thanks for sharing the great project. I'm new to this and tried the first example script. Training seems fine but testing fails as follows. Any idea?

Thanks

(venv) me@DESKTOP-FCVGD1U:~/workspace/github/ULTRA_pytorch$ bash example/toy/offline_exp_pipeline.sh
Reading data in ./example/toy/data/
Read data from ./example/toy/data//train in libsvm format.
Reading finish: 0 lines
Remove 7 invalid queries.
Data reading finish!
Finished reading 20 queries with lists.
Read data from ./example/toy/data//valid in libsvm format.
Reading finish: 0 lines
Remove 3 invalid queries.
Data reading finish!
Finished reading 6 queries with lists.
Train Rank list size 9
Valid Rank list size 9
Users can only see the top 9 documents for each query in training.
Creating model...
Build DLA

Loss Function is softmax_loss
Created model with fresh parameters.
Create simluated clicks feed

Create direct label feed with list size 9 with feature size 136
max_train_iter:  10
 Loss 4.346205 at Global Step 0: 
 Loss 6.528031 at Global Step 1: 
 Loss 8.008065 at Global Step 2: 
 Loss 4.675643 at Global Step 3: 
 Loss 8.523232 at Global Step 4: 
 Loss 4.580126 at Global Step 5: 
 Loss 5.745630 at Global Step 6: 
 Loss 4.260286 at Global Step 7: 
 Loss 4.781512 at Global Step 8: 
 Loss 4.347456 at Global Step 9: 
 Loss 4.397479 at Global Step 10: 
 Loss 4.239964 at Global Step 11: 
 Loss 4.159795 at Global Step 12: 
 Loss 5.694969 at Global Step 13: 
 Loss 4.179561 at Global Step 14: 
 Loss 4.300393 at Global Step 15: 
 Loss 4.171987 at Global Step 16: 
 Loss 4.124520 at Global Step 17: 
 Loss 5.438332 at Global Step 18: 
 Loss 4.089553 at Global Step 19: 
 Loss 5.418541 at Global Step 20: 
 Loss 4.175595 at Global Step 21: 
 Loss 4.735269 at Global Step 22: 
 Loss 4.107638 at Global Step 23: 
 Loss 4.677073 at Global Step 24: 
 Loss 4.106503 at Global Step 25: 
 Loss 4.410158 at Global Step 26: 
 Loss 4.117153 at Global Step 27: 
 Loss 4.603172 at Global Step 28: 
 Loss 4.095118 at Global Step 29: 
 Loss 4.604589 at Global Step 30: 
 Loss 4.101539 at Global Step 31: 
 Loss 4.362749 at Global Step 32: 
 Loss 4.077655 at Global Step 33: 
 Loss 4.241923 at Global Step 34: 
 Loss 4.122931 at Global Step 35: 
 Loss 4.558557 at Global Step 36: 
 Loss 4.125799 at Global Step 37: 
 Loss 4.322569 at Global Step 38: 
 Loss 4.135751 at Global Step 39: 
 Loss 4.521419 at Global Step 40: 
 Loss 4.149001 at Global Step 41: 
 Loss 4.357186 at Global Step 42: 
 Loss 4.099861 at Global Step 43: 
 Loss 4.441446 at Global Step 44: 
 Loss 4.093635 at Global Step 45: 
 Loss 4.306568 at Global Step 46: 
 Loss 4.094622 at Global Step 47: 
 Loss 4.529568 at Global Step 48: 
 Loss 4.184571 at Global Step 49: 
global step 50 learning rate 0.0500 step-time 0.13 loss 4.6094
mrr_3 tensor(0.5972, device='cuda:0')
mrr_5 tensor(0.5972, device='cuda:0')
mrr_10 tensor(0.5972, device='cuda:0')
ndcg_3 tensor(0.4604, device='cuda:0')
ndcg_5 tensor(0.5386, device='cuda:0')
ndcg_10 tensor(0.6191, device='cuda:0')
Save model, valid ndcg_10:0.619
current_step:  50
Reading data in ./example/toy/data/
Read data from ./example/toy/data//test in libsvm format.
Reading finish: 0 lines
Remove 3 invalid queries.
Data reading finish!
Finished reading 6 queries with lists.
Build DLA

Traceback (most recent call last):
  File "main.py", line 291, in <module>
    main(argv)
  File "main.py", line 284, in main
    test(exp_settings)
  File "main.py", line 236, in test
    model = create_model(exp_settings, test_set)
  File "main.py", line 70, in create_model
    model = ultra.utils.find_class(exp_settings['learning_algorithm'])(data_set, exp_settings)
  File "/home/me/workspace/github/ULTRA_pytorch/ultra/learning_algorithm/dla.py", line 97, in __init__
    self.rank_list_size = exp_settings['selection_bias_cutoff']
KeyError: 'selection_bias_cutoff'
QingyaoAi commented 2 years ago

Thanks for raising the issue! It should be fixed in the code right now. Please let us know if it still happens.

hideojoho commented 2 years ago

@QingyaoAi , @anhtran1010 Thank you for your support. The original error seems solved but produced another one as follows.

(venv) me@DESKTOP-FCVGD1U:~/workspace/github/ULTRA_pytorch$ python main.py --max_train_iteration=10
Reading data in ./tests/data/
Read data from ./tests/data//train in ULTRA format.
Feature reading finish.
List reading finish.
Label reading finish.
Remove 7 invalid queries.
Data reading finish!
Finished reading 20 queries with lists.
Read data from ./tests/data//valid in ULTRA format.
Feature reading finish.
List reading finish.
Label reading finish.
Remove 3 invalid queries.
Data reading finish!
Finished reading 6 queries with lists.
Train Rank list size 9
Valid Rank list size 9
Users can only see the top 9 documents for each query in training.
Creating model...
Build DLA

Loss Function is softmax_loss
Created model with fresh parameters.
Create simluated clicks feed

Create direct label feed with list size 9 with feature size 136
max_train_iter:  10
 Loss 4.440492 at Global Step 0: 
 Loss 6.424829 at Global Step 1: 
 Loss 4.830202 at Global Step 2: 
 Loss 10.693382 at Global Step 3: 
 Loss 5.427406 at Global Step 4: 
 Loss 5.200952 at Global Step 5: 
 Loss 4.584223 at Global Step 6: 
 Loss 4.241126 at Global Step 7: 
 Loss 4.160506 at Global Step 8: 
 Loss 4.166048 at Global Step 9: 
 Loss 4.358397 at Global Step 10: 
 Loss 4.206917 at Global Step 11: 
 Loss 4.137994 at Global Step 12: 
 Loss 4.561807 at Global Step 13: 
 Loss 4.151238 at Global Step 14: 
 Loss 4.385555 at Global Step 15: 
 Loss 4.136350 at Global Step 16: 
 Loss 4.306522 at Global Step 17: 
 Loss 4.081818 at Global Step 18: 
 Loss 4.461146 at Global Step 19: 
 Loss 4.122521 at Global Step 20: 
 Loss 4.049473 at Global Step 21: 
 Loss 4.759341 at Global Step 22: 
 Loss 4.263655 at Global Step 23: 
 Loss 4.331538 at Global Step 24: 
 Loss 3.980463 at Global Step 25: 
 Loss 4.498557 at Global Step 26: 
 Loss 4.095665 at Global Step 27: 
 Loss 4.836687 at Global Step 28: 
 Loss 4.038417 at Global Step 29: 
 Loss 4.069120 at Global Step 30: 
 Loss 4.188335 at Global Step 31: 
 Loss 4.101030 at Global Step 32: 
 Loss 3.871989 at Global Step 33: 
 Loss 4.743153 at Global Step 34: 
 Loss 4.104199 at Global Step 35: 
 Loss 4.333329 at Global Step 36: 
 Loss 4.276215 at Global Step 37: 
 Loss 4.049036 at Global Step 38: 
 Loss 4.262655 at Global Step 39: 
 Loss 4.079478 at Global Step 40: 
 Loss 4.245237 at Global Step 41: 
 Loss 4.023198 at Global Step 42: 
 Loss 4.118076 at Global Step 43: 
 Loss 4.011585 at Global Step 44: 
 Loss 4.106015 at Global Step 45: 
 Loss 4.046475 at Global Step 46: 
 Loss 4.240688 at Global Step 47: 
 Loss 3.965695 at Global Step 48: 
 Loss 4.201066 at Global Step 49: 
global step 50 learning rate 0.0500 step-time 0.12 loss 4.4594
Traceback (most recent call last):
  File "main.py", line 301, in <module>
    main(argv)
  File "main.py", line 296, in main
    train(exp_settings)
  File "main.py", line 184, in train
    valid_summary = validate_model(valid_set, valid_input_feed)
  File "main.py", line 176, in validate_model
    _, _, summary = model.validation(input_feed)
  File "/home/me/workspace/github/ULTRA_pytorch/ultra/learning_algorithm/dla.py", line 280, in validation
    metric_values = ultra.utils.make_ranking_metric_fn(
  File "/home/me/workspace/github/ULTRA_pytorch/ultra/utils/metrics.py", line 91, in _mean_reciprocal_rank_fn
    return mean_reciprocal_rank(
  File "/home/me/workspace/github/ULTRA_pytorch/ultra/utils/metrics.py", line 284, in mean_reciprocal_rank
    labels, predictions, weights, topn = _prepare_and_validate_params(
  File "/home/me/workspace/github/ULTRA_pytorch/ultra/utils/metrics.py", line 249, in _prepare_and_validate_params
    topn = [min(n, list_size) for n in topn]
TypeError: 'int' object is not iterable

As far as I examined, def prepare_and_validate_params in ultra/utils/metrics.py is defined at Line 224, where topn is expected to receive an array or similar type.

def _prepare_and_validate_params(labels, predictions, weights=None, topn=None):
    """Prepares and validates the parameters.

    Args:
      labels: A `Tensor` of the same shape as `predictions`. A value >= 1 means a
        relevant example.
      predictions: A `Tensor` with shape [batch_size, list_size]. Each value is
        the ranking score of the corresponding example.
      weights: A `Tensor` of the same shape of predictions or [batch_size, 1]. The
        former case is per-example and the latter case is per-list.
      topn: A list of cutoff for how many examples to consider for this metric.

    Returns:
      (labels, predictions, weights, topn) ready to be used for metric
      calculation.
    """

However, the function is called at Line 283 where topn seems an integer.

    list_size = predictions.size()[-1]
    labels, predictions, weights, topn = _prepare_and_validate_params(
        labels, predictions, weights, list_size)

Hope this helps.

anhtran1010 commented 2 years ago

Hi,

First, thank you so much for bringing this to our attention. We have patched the bug and updated our Github repository. As you may already know, we are currently updating ULTRA_pytorch by adding more learning algorithms, ranking models as well as improving the execution time of the toolbox. As a result, bugs like this may occur even with our testing. I sincerely apologize for any inconvenience.

Feel free to reach out to us with any questions, or possible issues that need to be addressed.

Thank you so much

Anh Tran

On Thu, Oct 28, 2021 at 3:15 AM Hideo Joho @.***> wrote:

@QingyaoAi https://github.com/QingyaoAi , @anhtran1010 https://github.com/anhtran1010 Thank you for your support. The original error seems solved but produced another one as follows.

(venv) @.***:~/workspace/github/ULTRA_pytorch$ python main.py --max_train_iteration=10 Reading data in ./tests/data/ Read data from ./tests/data//train in ULTRA format. Feature reading finish. List reading finish. Label reading finish. Remove 7 invalid queries. Data reading finish! Finished reading 20 queries with lists. Read data from ./tests/data//valid in ULTRA format. Feature reading finish. List reading finish. Label reading finish. Remove 3 invalid queries. Data reading finish! Finished reading 6 queries with lists. Train Rank list size 9 Valid Rank list size 9 Users can only see the top 9 documents for each query in training. Creating model... Build DLA

Loss Function is softmax_loss Created model with fresh parameters. Create simluated clicks feed

Create direct label feed with list size 9 with feature size 136 max_train_iter: 10 Loss 4.440492 at Global Step 0: Loss 6.424829 at Global Step 1: Loss 4.830202 at Global Step 2: Loss 10.693382 at Global Step 3: Loss 5.427406 at Global Step 4: Loss 5.200952 at Global Step 5: Loss 4.584223 at Global Step 6: Loss 4.241126 at Global Step 7: Loss 4.160506 at Global Step 8: Loss 4.166048 at Global Step 9: Loss 4.358397 at Global Step 10: Loss 4.206917 at Global Step 11: Loss 4.137994 at Global Step 12: Loss 4.561807 at Global Step 13: Loss 4.151238 at Global Step 14: Loss 4.385555 at Global Step 15: Loss 4.136350 at Global Step 16: Loss 4.306522 at Global Step 17: Loss 4.081818 at Global Step 18: Loss 4.461146 at Global Step 19: Loss 4.122521 at Global Step 20: Loss 4.049473 at Global Step 21: Loss 4.759341 at Global Step 22: Loss 4.263655 at Global Step 23: Loss 4.331538 at Global Step 24: Loss 3.980463 at Global Step 25: Loss 4.498557 at Global Step 26: Loss 4.095665 at Global Step 27: Loss 4.836687 at Global Step 28: Loss 4.038417 at Global Step 29: Loss 4.069120 at Global Step 30: Loss 4.188335 at Global Step 31: Loss 4.101030 at Global Step 32: Loss 3.871989 at Global Step 33: Loss 4.743153 at Global Step 34: Loss 4.104199 at Global Step 35: Loss 4.333329 at Global Step 36: Loss 4.276215 at Global Step 37: Loss 4.049036 at Global Step 38: Loss 4.262655 at Global Step 39: Loss 4.079478 at Global Step 40: Loss 4.245237 at Global Step 41: Loss 4.023198 at Global Step 42: Loss 4.118076 at Global Step 43: Loss 4.011585 at Global Step 44: Loss 4.106015 at Global Step 45: Loss 4.046475 at Global Step 46: Loss 4.240688 at Global Step 47: Loss 3.965695 at Global Step 48: Loss 4.201066 at Global Step 49: global step 50 learning rate 0.0500 step-time 0.12 loss 4.4594 Traceback (most recent call last): File "main.py", line 301, in main(argv) File "main.py", line 296, in main train(exp_settings) File "main.py", line 184, in train valid_summary = validate_model(valid_set, valid_input_feed) File "main.py", line 176, in validatemodel , _, summary = model.validation(input_feed) File "/home/me/workspace/github/ULTRA_pytorch/ultra/learning_algorithm/dla.py", line 280, in validation metric_values = ultra.utils.make_ranking_metric_fn( File "/home/me/workspace/github/ULTRA_pytorch/ultra/utils/metrics.py", line 91, in _mean_reciprocal_rank_fn return mean_reciprocal_rank( File "/home/me/workspace/github/ULTRA_pytorch/ultra/utils/metrics.py", line 284, in mean_reciprocal_rank labels, predictions, weights, topn = _prepare_and_validate_params( File "/home/me/workspace/github/ULTRA_pytorch/ultra/utils/metrics.py", line 249, in _prepare_and_validate_params topn = [min(n, list_size) for n in topn] TypeError: 'int' object is not iterable

As far as I examined, def prepare_and_validate_params in ultra/utils/metrics.py is defined at Line 224, where topn is expected to receive an array or similar type.

def _prepare_and_validate_params(labels, predictions, weights=None, topn=None): """Prepares and validates the parameters. Args: labels: A Tensor of the same shape as predictions. A value >= 1 means a relevant example. predictions: A Tensor with shape [batch_size, list_size]. Each value is the ranking score of the corresponding example. weights: A Tensor of the same shape of predictions or [batch_size, 1]. The former case is per-example and the latter case is per-list. topn: A list of cutoff for how many examples to consider for this metric. Returns: (labels, predictions, weights, topn) ready to be used for metric calculation. """

However, the function is called at Line 283 where topn seems an integer.

list_size = predictions.size()[-1]
labels, predictions, weights, topn = _prepare_and_validate_params(
    labels, predictions, weights, list_size)

Hope this helps.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ULTR-Community/ULTRA_pytorch/issues/2#issuecomment-953662438, or unsubscribe https://github.com/notifications/unsubscribe-auth/AGMWVLWNXUNACHKYJWRRUSLUJEICBANCNFSM5GLNTLEQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

hideojoho commented 2 years ago

Thanks! Both training and testing went fine.