search

facebookresearch / TaBERT

This repository contains source code for the TaBERT model, a pre-trained language model for learning joint representations of natural language utterances and (semi-)structured tables for semantic parsing. TaBERT is pre-trained on a massive corpus of 26M Web tables and their associated natural language context, and could be used as a drop-in replacement of a semantic parsers original encoder to compute representations for utterances and table schemas (columns).
Other
580 stars 63 forks source link

How exactly the model works off the shelf? #2

Open bhushanpant opened 3 years ago

bhushanpant commented 3 years ago

Amazing paper!

Thanks for publishing the code.

I wanted to check, if it is possible to test out Tabert off the shelf for my documents/images. If yes, can you share some steps/guidelines on how to do that?

pcyin commented 3 years ago

Thanks for your interest! What kinds of tasks are you considering? If you'd like to use TaBERT as the encoder of text and tables in your end-to-end task-specific model, you could just fine-tune TaBERT with the rest of the parameters in the network.

If you'd like to pre-train TaBERT on your in-domain text and tabular data, sorry we haven't released the pre-training code yet. I will prepare a minimal working example ASAP. Thanks!

Lucianod28 commented 3 years ago

I was wondering how to use the pretrained model for other tasks. I want to build a column classifier based on TaBERT, using the column representations with a few additional layers including a classifier on the end. I figure this could be trained (fine-tuned) end-to-end from a pretrained model but I'm not sure how to put it together. I see on https://github.com/pcyin/pytorch_neural_symbolic_machines it says training with pretrained TaBERT models is to be released, is there any other example or resource you could point me to that might help me get started? Thanks again for the paper and code!