-
Firstly, thanks for this cool work combining Paligemma and Colbert!
Given a PDF file
When I try the same PDF using the HF demo app at: https://huggingface.co/spaces/manu/ColPali-demo
And I try t…
-
Hi,
Thanks for the repo! One thing that is unclear to me is how you evaluated your models on the Amazon-C4 dataset. In your repo, you mentioned that _"sampled_item_metadata_1M.jsonl contains ~1M it…
-
Hola!
Al principio del enunciado se dice que debemos evaluar para cada modelo ndcg@10, ndcg@20, ndcg@30, map@10, map@20, map@30. Sin embargo la actividad 4 pide un análisis de sensibilidad sólo para …
-
I tried to evaluate both unicamp-dl/mt5-base-en-msmarco and unicamp-dl/mt5-13b-mmarco-100k, but the performance of 13b is lower than base model. Here is a simple comparison of reranking results of BM2…
-
![image](https://user-images.githubusercontent.com/52364987/205821622-3811d8bb-0951-4439-838d-adfd58e374e7.png)
ndcg_cut.10 and -M 10 ndcg are different when multiple level of relevance is given.
H…
-
I tried `evaluation_threads = 56`, but the time spent for each epoch is pretty the same:
`evaluation_threads = 1`
Iteration 0 [112.8 s]: HR = 0.4967, NDCG = 0.2792, loss = 0.3568 [2.6 s]
Iterat…
-
Excuse me, after training by the default arguments, I get the recall and NDCG score, but the result isn't as good as the report in the paper.
Here is my result after 500 epochs:
```
recall@50, ndcg…
-
Hi,thanks for sharing the code.
With your source code unmodified (dropout: 0.5, neg-weight: 0.5), I have tried on ml-1m and get the following results:
First col: Recall
Second col: NDCG
> loss,…
-
As reported by @karlhigley,
> I now take issue with the implementation of NDCG in sklearn, which seems like it could use `argpartition` and be much faster for long lists of items with small top-K r…
-
Hi Kevin,
I am looking to add the **Normalized Discounted Cumulative Gain** ([NDCG](https://en.wikipedia.org/wiki/Discounted_cumulative_gain)) metric to the `AccuracyCalculator` while also using th…