search

GreyDGL / PentestGPT

A GPT-empowered penetration testing tool
MIT License
7.29k stars 886 forks source link

API alternates between GPT 3.5 Turbo and 4 regardless of selected reasoning model #96

Closed Af7eR9l0W closed 1 year ago

Af7eR9l0W commented 1 year ago

I have encountered some anomalies with the API that might be related to OpenAI’s implementation. It seems that the API does not always respect the model selection and arbitrarily switches between different models. This compromises the quality and consistency of the outputs, especially when using GPT 4, which appears to be unreliable. With GPT 3.5 Turbo selected, it seems the variability is lower and more stable than GPT 4 does with itself. The initial prompt is almost always generated using GPT 3.5. I suspect this is a result of OpenAIs inefficient and poor management of requests under high demand.

Is this an issue on my end? Since the RPM is much lower for 4 image

GPT 4 Selected ptbug

GPT 3.5 turbo selected ptbug2

GreyDGL commented 1 year ago

This is an interesting one. Let me do some investigation. Ideally even if you set the model to GPT-4, there are still GPT-3.5 usage because PentestGPT integrates some parsing functions that by default use GPT3.5.