Ability to send no limit to opeanai completion

polacekpavel commented 1 year ago

First check

[X] I added a descriptive title to this issue.
[X] I used the GitHub search to look for a similar issue and didn't find it.
[X] I searched the Marvin documentation for this feature.

Describe the current behavior

The current default for max_tokens of completion call is set to 1500 which is problematic for big context completion. It could be lowered, but I think it is better to let Openai compute the remaining tokens simply by not passing this property while calling chat completion https://platform.openai.com/docs/api-reference/chat/create#chat/create-max_tokens.

I've drafted PR which is adding this behavior. Let me know what you think or if I miss how to configure this properly.

Describe the proposed behavior

If -1 is passed to max_tokens property then no limit is applied for openai models

Example Use

marvin.settings.llm_max_tokens = -1

Additional context

No response

aaazzam commented 1 year ago

Great suggestion @polacekpavel.

I think -1 makes sense, the other option is to make llm_max_tokens: Optional[int] and interpret None as no max.

I'm okay with either, but will check to see if there's a standard on this.

Either way, thanks for the PR - I'll review it soon and it should make our next release!

maxbaluev commented 1 year ago

But this is not an option if you need to work with a really large context. What are your ideas on how to solve this problem?

PrefectHQ / marvin