First training task: supervised prediction of 1 class with patent-BERT

[Oblynx]

Let's use one of these models:

• https://huggingface.co/anferico/bert-for-patents
• https://github.com/ec-jrc/Patents4IPPC

Let's generate a small test dataset.

Goal: supervised classification for the class "CLIMATE CHANGE MITIGATION TECHNOLOGIES RELATED TO WASTEWATER TREATMENT OR WASTE MANAGEMENT": Y02W

[MikeDoes]

https://worldwide.espacenet.com/patent/cpc-browser#!/CPC=Y02W

[MikeDoes]

Source: claims Label: 1 or 0 for wether it's part of the Y02W

[Oblynx]

Ambitious goal: unsupervised training.

• We would need to train a concept/embedding of various classes, including the target classY02W. Using a natural language model, we map the description of the class (natural lang) to an embedding.
• Train the patent model to minimize the distance of the patent to the embedding of its tags/classes.
• Hopefully, for a newly-seen embedding it would also manage to identify patents with a short distance.

Example: Word2Vec creates an embedding of the class description, and we train BERT using this embedding in the loss function.

Workflow:

• For the moment we can use Colab
• Pytorch lightning