thunlp / DebugBench

The repository for paper "DebugBench: "Evaluating Debugging Capability of Large Language Models".
Apache License 2.0
55 stars 4 forks source link

leetcode.rest.ApiException: (429) Reason: Too Many Requests #2

Closed shuaiwang2022 closed 8 months ago

shuaiwang2022 commented 8 months ago

![Uploading image.png…]() However, it suffers from the above problems

Rachum-thu commented 8 months ago

A client will undoubtedly have a query rate limit to prevent attacks.

There are two approaches to solve the problem.

1. Prolong cooldown time for the tester at evaluation/leetcode_oj/leetcode_tester.py

...
class LeetCodeTester(object):

    def __init__(self):
        os.environ['LEETCODE_SESSION'] = LEETCODE_SESSION_COOKIE
        self.env = LeetCodeEnv(cooldown=15)
        self.lang_dict = {
            "python": ProgrammingLanguage.PYTHON3,
            "python3": ProgrammingLanguage.PYTHON3,
            "java": ProgrammingLanguage.JAVA,
            "c": ProgrammingLanguage.C,
            "cpp": ProgrammingLanguage.CPP,
        }
   ...

For my experience, set cooldown to 15 or more will solve the problem. You can try a larger cooldown value.

2. Utilize Multiple Cookies for Increased Speed:

If the speed limitation of a single account is unsatisfactory, consider using multiple cookies to create a multi-threaded tester.

ywq2001 commented 4 months ago

I tried the first method, I set cooldown to 25 or 35, there will still be too many requests, it seems difficult to solve, does this need to connect to the vpn test

Rachum-thu commented 3 months ago

I tried the first method by setting the cooldown to 25 or 35, but there are still too many requests. It seems difficult to solve. Do I need to connect to the VPN to test this?

@ywq2001

ywq2001 commented 3 months ago

leetcode.rest.ApiException: (403) Reason: Forbidden

Have you ever had such a situation, as if a cookie used too many times will be forbidden, how to solve it?

Rachum-thu commented 3 months ago

leetcode.rest.ApiException: (403) Reason: Forbidden

Have you ever had such a situation, as if a cookie used too many times will be forbidden, how to solve it?

@ywq2001 Yes, I have.

Reason for the Problem: This problem occurs because LeetCode imposes a limit on the number of online judge submissions each client can make within a specific time period. When this limit is exceeded, you receive a “403 Forbidden” error.

How to Overcome the Limitation: To avoid this issue, you can use cookies from multiple accounts to run your code. I strongly recommend this approach as it addresses both the submission limit and improves the speed of your evaluations.

Implementing the use of multiple cookies for synchronous queries will be essential for more efficient and convenient benchmarking. Thank you for your feedback. I will update the relevant code this weekend.

Feel free to ask if you need any further adjustments :)