search

amazon-science / cceval

CrossCodeEval: A Diverse and Multilingual Benchmark for Cross-File Code Completion (NeurIPS 2023)
https://crosscodeeval.github.io/
Apache License 2.0
115 stars 17 forks source link

Retriever performance evaluation #14

Closed bsobolev closed 10 months ago

bsobolev commented 10 months ago

Hello @wasiahmad @zijwang, The paper suggests that the framework can be leveraged for retrieval performance evaluation; However, the repository, as it currently stands, seems to include only pre-generated line completions. For our purposes, we are specifically interested in assessing the performance of various similarity calculation methods within the retrieval process.

Is it possible to get access to the original data and the code used to generate line completion?
Thanks

zijwang commented 10 months ago

Thanks for your interest, @bsobolev . For raw data, can you please email us? (This is now stated in https://github.com/amazon-science/cceval#requirements) For retrieval code, we are working on a code release and will post once it is ready.

zijwang commented 10 months ago

@bsobolev can you please check #16 to see whether that answers your question?

bsobolev commented 10 months ago

@zijwang Thank you so much for getting back to me on this question with the dataset and the scripts!