The maximum number of input tokens.

[yohohohoho]

May I ask what the token limit for input is? I've noticed a significant decrease in field localization accuracy when dealing with large tables.

[buddyroo30]

I'd also like to know this. I tried it out on some smaller schemas and it worked well, using the inference.py script. But when I try it on a larger schema I get this error:

torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 11.33 GiB (GPU 0; 22.20 GiB total capacity; 20.22 GiB already allocated; 997.12 MiB free; 20.24 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

However, to test if it is the length of the schema that is the issue I cut out some of the tables from the larger schema and eventually cut out enough so that it worked and generated SQL, so clearly it was the length of the schema that was the issue. Note I ran these tests on an AWS g5.12xlarge, so that should be large enough (it was large enough to load the sqlcoder LLM without error, it was just that I got the out of memory error when trying to do inference on the large schema). Is there any way to get it to work for larger schemas (just run on larger GPU server with more GPU RAM?)

[durgesh-rathod-sf]

@yohohohoho @buddyroo30 , Please Check this link if it answers your question related to token length

[smashfan]

I'd also like to know this. I tried it out on some smaller schemas and it worked well, using the inference.py script. But when I try it on a larger schema I get this error:

torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 11.33 GiB (GPU 0; 22.20 GiB total capacity; 20.22 GiB already allocated; 997.12 MiB free; 20.24 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

However, to test if it is the length of the schema that is the issue I cut out some of the tables from the larger schema and eventually cut out enough so that it worked and generated SQL, so clearly it was the length of the schema that was the issue. Note I ran these tests on an AWS g5.12xlarge, so that should be large enough (it was large enough to load the sqlcoder LLM without error, it was just that I got the out of memory error when trying to do inference on the large schema). Is there any way to get it to work for larger schemas (just run on larger GPU server with more GPU RAM?)

I also encountered the same issue. I found that as the input size to this model increases, the memory it requires surges dramatically, sometimes demanding up to 100G. I'm not sure if this is a normal situation. @durgesh-rathod-sf

[WillWillWong]

I'm sorry to trouble you. When I included remarks in a 1000 column data table. The same error occurred, also out of memory on a 4*4090 machine. I'd like to know how to solve this problem, thank you.

[WillWillWong]

finally,she tell me. Have a trouble is ‘RuntimeError:The size of tensor a (8192) must match the size of tensor b (24864)at non-singleton dimension 2’

[nuaabuaa07]

I found a solution that reduceing the table schema . See it here https://github.com/defog-ai/sql-eval/blob/main/utils/pruning.py .

[github-actions[bot]]

This issue is stale because it has been open for 30 days with no activity.

[github-actions[bot]]

This issue was closed because it has been inactive for 14 days since being marked as stale.