Mattie / cataclysm

Cataclysm - Code generation library for the end game
MIT License
405 stars 44 forks source link

doom.first_prime_with_3_digits() returns None #2

Open tpsjr7 opened 1 year ago

tpsjr7 commented 1 year ago

I installed cataclysm with pip today. I have a simple main.py

from cataclysm import doom
uhoh = doom.first_prime_with_3_digits()
print(uhoh)

I can see that it called open api (3.5) and it generated the code and put it here datafiles/cataclysm/code/function_first_prime_with_3_digits.yml

"signatures":
  "first_prime_with_3_digits-0-0": |-
    def is_prime(n):
        if n < 2:
            return False
        for i in range(2, int(n ** 0.5) + 1):
            if n % i == 0:
                return False
        return True

    def first_prime_with_3_digits():
        for i in range(100, 1000):
            if is_prime(i):
                return i

When I step in the debugger, I can see that up to the exec(code, ldict) that the "code" variable looks like this

def is_prime(n):
    if n < 2:
        return False
    for i in range(2, int(n ** 0.5) + 1):
        if n % i == 0:
            return False
    return True

def first_prime_with_3_digits():
    for i in range(100, 1000):
        if is_prime(i):
            return i

and the ldict has{'_exec_return_values': None, 'args_in': (), 'kwargs_in': {}} https://github.com/Mattie/cataclysm/blob/master/cataclysm/doomed.py#L83

but after the exec, the ldict['_exec_return_values'] evaluates to None instead of the expected value of 101.

This happens with other functions I've tried.

I'm running Windows and python 3.11.2

Mattie commented 1 year ago

I'm assuming this is an issue with GPT 3.5 that I don't see with GPT4. The current prompt may not be enough to convince GPT 3.5 to have the proper return values. I'll add some tests for GPT 3.5 and see if I can get it to work better! Thanks for mentioning this.