search

CoIR-team / coir

A Comprehensive Benchmark for Code Information Retrieval.
https://archersama.github.io/coir/
Apache License 2.0
63 stars 11 forks source link

New model submission to CoIR Leaderboard #9

Open yeliu918 opened 5 days ago

yeliu918 commented 5 days ago

Hi @archersama,

We have a family of new embedding models, CodeXEmbed, and would like to submit its results to our leaderboard. Could you please assist us with this?

{"Model": "CodeXEmbed-7B", "Model Size (Million Parameters)": 7B, "URL": "”, "Apps": 85.22, "CosQA": 33.27, "Synthetic Text2sql":64.57, "CodeSearchNet": 78.84, "CodeSearchNet-CCR": 86.77, "CodeTrans-Contest": 90.64, "CodeTrans-DL": 32.31, "StackOverFlow QA": 94.25, "CodeFeedBack-ST": 80.93, "CodeFeedBack-MT": 57.83, "Avg": 70.46}

{"Model": "CodeXEmbed-2B", "Model Size (Million Parameters)": 2B, "URL": "”, "Apps": 74.99, "CosQA": 36.31, "Synthetic Text2sql":59.00, "CodeSearchNet": 73.50, "CodeSearchNet-CCR": 85.77, "CodeTrans-Contest": 86.63, "CodeTrans-DL": 33.17, "StackOverFlow QA": 90.54, "CodeFeedBack-ST": 81.15, "CodeFeedBack-MT": 53.08, "Avg": 67.41}

{"Model": "CodeXEmbed-400M", "Model Size (Million Parameters)": 2B, "URL": "”, "Apps": 48.57, "CosQA": 34.05, "Synthetic Text2sql":58.96, "CodeSearchNet": 72.53, "CodeSearchNet-CCR": 80.15, "CodeTrans-Contest": 75.67, "CodeTrans-DL": 34.85, "StackOverFlow QA": 89.51, "CodeFeedBack-ST": 78.87, "CodeFeedBack-MT": 45.75, "Avg": 61.89}

archersama commented 5 days ago

Great Work!!! But could you provide some URLs or API endpoints? This would help prevent individuals from submitting unverifiable models, which could potentially compromise the integrity of the leaderboard.