search

brunnurs / valuenet

ValueNet: A Neural Text-to-SQL Architecture Incorporating Values
Apache License 2.0
66 stars 22 forks source link

Facing error if its more than 512 tokens #9

Open yuvarajvc opened 3 years ago

yuvarajvc commented 3 years ago

If i want to use more than 512 tokens bert is not allowing and i am facing an error, can anyone help me how to use bigger database which is having 2000+ tokens.

brunnurs commented 3 years ago

That is indeed a problem. You can use a more recent transformer (e.g. BART) which supports up to 1024 tokens, but you could also try to simplify your schema - most probably there are quite a few tables / columns which are purely technical and a user will never use.

yuvarajvc commented 3 years ago

Thanks for your reply, we are using larger database so the token are going beyond 512, we are planning to do exact column and table match as well as partial column match and table match and will pass only those tables and columns to BERT MODEL, do you think this approach will work or we have to send all the tables and columns