search

nuprl / MultiPL-E

A multi-programming language benchmark for LLMs
https://nuprl.github.io/MultiPL-E/
Other
201 stars 38 forks source link

Dev merging #96

Closed arjunguha closed 1 year ago

arjunguha commented 1 year ago

@cassanof and @JohnGouwar are we using this file:

https://github.com/nuprl/MultiPL-E/pull/87/commits/4d48a4884a61b375adb710c38702d47fc9ff157c

If so, I haven't yet merged it into this PR.

arjunguha commented 1 year ago

@cassanof should we merge this commit: https://github.com/nuprl/MultiPL-E/pull/87/commits/6d3b30a7a66f16889295cf86ef7760ca355a30db

Just react with :-1: if there is no need to merge. Otherwise, reply with a comment.

arjunguha commented 1 year ago

@cassanof should we merge this: https://github.com/nuprl/MultiPL-E/pull/87/commits/f82258be3bc25a0307cd134b5fd7616e1bb3cb8f

Just react with :-1: if there is no need to merge. Otherwise, reply with a comment.

arjunguha commented 1 year ago

@cassanof should we merge this: https://github.com/nuprl/MultiPL-E/pull/87/commits/d75a644ec1d7d10961c0c894ceade0b9e53eb675

Just react with :-1: if there is no need to merge. Otherwise, reply with a comment.

cassanof commented 1 year ago

@cassanof should we merge this commit: 6d3b30a

Just react with 👎 if there is no need to merge. Otherwise, reply with a comment.

This is to make the canonical solution work. Maybe I can try to find another way that also works, but I'm not sure what the regex was there for in the beginning.

@cassanof should we merge this: f82258b

Just react with 👎 if there is no need to merge. Otherwise, reply with a comment.

This is pretty handy, no? On the H100 I can run multiple 1b models on the same GPU this way

@cassanof should we merge this: d75a644

Just react with 👎 if there is no need to merge. Otherwise, reply with a comment.

Yeah this is pretty useful. Otherwise you can only see pass@1 at temp 0.2 or pass@10 + pass@100 at temp 0.8

arjunguha commented 1 year ago

Confused:

@cassanof should we merge this: f82258b

This is pretty handy, no? On the H100 I can run multiple 1b models on the same GPU this way

Why do you need bfloat16 for that? Can't you use float16 too?

cassanof commented 1 year ago

Confused:

@cassanof should we merge this: f82258b

This is pretty handy, no? On the H100 I can run multiple 1b models on the same GPU this way

Why do you need bfloat16 for that? Can't you use float16 too?

float16 is way too slow compared to bfloat16. if the hardware supports bf16, why not use it?