Text2Text Language Modeling Toolkit

Colab Notebooks

• Assistant (free private ChatGPT LLM alternative)
• STF-IDF multilingual search
• All examples

Installation Requirements

pip install -qq -U text2text

• Examples run with <16 GB RAM on free Colab GPUs.

Quick Start Guide

Examples

Assistant

• Free private open source alternative to commercial LLMs.
• Commercial LLMs are costly, collect your data, impose quotas and rate limits that hinder development.
• Run at no cost on Google Colab free tier, so you don't even need your own device.

import text2text as t2t
asst = t2t.Assistant()

# Streaming example
chat_history = [
    {"role": "user",  "content": "Hi"},
    {"role": "assistant", "content": "Hello, how are you?"},
    {"role": "user", "content": "What should I do today?"}
]

result = asst.chat_completion(chat_history, stream=True) #{'role': 'assistant', 'content': '1. Make a list of things to be grateful for.\n2. Go outside and take a walk in nature.\n3. Practice mindfulness meditation.\n4. Connect with a loved one or friend.\n5. Do something kind for someone else.\n6. Engage in a creative activity like drawing or writing.\n7. Read an uplifting book or listen to motivational podcasts.'}
for chunk in result:
  print(chunk['message']['content'], end='', flush=True)

# Running conversation
messages = []
while True:
  user_input = input("User: ")
  print()
  messages.append({"role": "user", "content": user_input})
  print("Assistant: ")
  result = asst.chat_completion(messages, stream=False)
  print(result["message"]["content"])
  messages.append(result["message"])
  print()

# Schema for structured output
from pydantic import BaseModel

class Song(BaseModel):
    name: str
    artist: str

result = asst.chat_completion([
    {"role": "user",  "content": "What is Britney Spears's best song?"}
], schema=Song)
# Song(name='Toxic', artist='Britney Spears')

# Embeddings
asst.embed(["hello, world!", "this will be embedded"])

Tokenization

t2t.Tokenizer().transform([
  "Let's go hiking tomorrow",
  "안녕하세요.",
  "돼지꿈을 꾸세요~~"
])

# Sub-word tokens
[['▁Let', "'", 's', '▁go', '▁hik', 'ing', '▁tom', 'orrow'],
 ['▁안녕', '하세요', '.'],
 ['▁', '돼', '지', '꿈', '을', '▁꾸', '세요', '~~']]

Embedding / Vectorization

t2t.Vectorizer().transform([
  "Let's go hiking tomorrow",
  "안녕하세요.",
  "돼지꿈을 꾸세요~~"
])

# Embeddings
[[-0.00352954,  0.0260059 ,  0.00407429, ..., -0.04830331,
  -0.02540749, -0.00924972],
  [ 0.00043362,  0.00249816,  0.01755436, ...,  0.04451273,
    0.05118701,  0.01895813],
  [-0.03563676, -0.04856304,  0.00518898, ..., -0.00311068,
    0.00071953, -0.00216325]]

TF-IDF

t2t.Tfidfer().transform([
  "Let's go hiking tomorrow",
  "안녕하세요.",
  "돼지꿈을 꾸세요~~"
])

# TF-IDF values
[{'!': 0.22360679774997894,
  "'": 0.44721359549995787,
  ',': 0.22360679774997894,
  'ing': 0.22360679774997894,
  'orrow': 0.22360679774997894,
  's': 0.44721359549995787,
  '▁Let': 0.22360679774997894,
  '▁go': 0.44721359549995787,
  '▁hik': 0.22360679774997894,
  '▁let': 0.22360679774997894,
  '▁tom': 0.22360679774997894},
 {'.': 0.5773502691896258,
  '▁안녕': 0.5773502691896258,
  '하세요': 0.5773502691896258},
 {'~~': 0.3535533905932738,
  '▁': 0.3535533905932738,
  '▁꾸': 0.3535533905932738,
  '꿈': 0.3535533905932738,
  '돼': 0.3535533905932738,
  '세요': 0.3535533905932738,
  '을': 0.3535533905932738,
  '지': 0.3535533905932738}]

BM25

t2t.Bm25er().transform([
  "Let's go hiking tomorrow",
  "안녕하세요.",
  "돼지꿈을 꾸세요~~"
])

# BM25 values
[{"'": 1.2792257271403649,
  'ing': 1.2792257271403649,
  'orrow': 1.2792257271403649,
  's': 1.2792257271403649,
  '▁Let': 1.2792257271403649,
  '▁go': 1.2792257271403649,
  '▁hik': 1.2792257271403649,
  '▁tom': 1.2792257271403649},
 {'.': 1.751071282233123, '▁안녕': 1.751071282233123, '하세요': 1.751071282233123},
 {'~~': 1.2792257271403649,
  '▁': 1.2792257271403649,
  '▁꾸': 1.2792257271403649,
  '꿈': 1.2792257271403649,
  '돼': 1.2792257271403649,
  '세요': 1.2792257271403649,
  '을': 1.2792257271403649,
  '지': 1.2792257271403649}]

Index

index = t2t.Indexer().transform([
  "Let's go hiking tomorrow, let's go!",
  "안녕하세요.",
  "돼지꿈을 꾸세요~~",
])

index.retrieve(["돼지"], k=1) #[['"돼지꿈을 꾸세요~~"']]

# Add documents
index.add(["Hello, World! 你好,世界!"])

# Remove by ids
index.remove([2]) #Removes "돼지꿈을 꾸세요~~"

# Retrieve k results per query sorted by distance
index.retrieve(["你好, World"], k=3)

To learn more, see STF-IDF.

Levenshtein Sub-word Edit Distance

t2t.Measurer().transform([
  "Hello, World! [SEP] Hello, what?",
  "안녕하세요. [SEP] 돼지꿈을 꾸세요~~"
], metric="levenshtein_distance")

# Distances
 [2, 8]

Translation

# Sample texts
article_en = 'The Secretary-General of the United Nations says there is no military solution in Syria.'

notre_dame_str = "As at most other universities, Notre Dame's students run a number of news media outlets. The nine student - run outlets include three newspapers, both a radio and television station, and several magazines and journals. Begun as a one - page journal in September 1876, the Scholastic magazine is issued twice monthly and claims to be the oldest continuous collegiate publication in the United States. The other magazine, The Juggler, is released twice a year and focuses on student literature and artwork. The Dome yearbook is published annually. The newspapers have varying publication interests, with The Observer published daily and mainly reporting university and other news, and staffed by students from both Notre Dame and Saint Mary's College. Unlike Scholastic and The Dome, The Observer is an independent publication and does not have a faculty advisor or any editorial oversight from the University. In 1987, when some students believed that The Observer began to show a conservative bias, a liberal newspaper, Common Sense was published. Likewise, in 2003, when other students believed that the paper showed a liberal bias, the conservative paper Irish Rover went into production. Neither paper is published as often as The Observer; however, all three are distributed to all students. Finally, in Spring 2008 an undergraduate journal for political science research, Beyond Politics, made its debut."

bacteria_str = "Bacteria are a type of biological cell. They constitute a large domain of prokaryotic microorganisms. Typically a few micrometres in length, bacteria have a number of shapes, ranging from spheres to rods and spirals. Bacteria were among the first life forms to appear on Earth, and are present in most of its habitats."

bio_str = "Biology is the science that studies life. What exactly is life? This may sound like a silly question with an obvious answer, but it is not easy to define life. For example, a branch of biology called virology studies viruses, which exhibit some of the characteristics of living entities but lack others. It turns out that although viruses can attack living organisms, cause diseases, and even reproduce, they do not meet the criteria that biologists use to define life."

t2t.Translator().transform([
  article_en,
  notre_dame_str,
  bacteria_str,
  bio_str
], src_lang='en', tgt_lang='zh')

# Translations
['联合国秘书长说,叙利亚没有军事解决方案。',
 '与大多数其他大学一样,Notre Dame的学生运行的新闻媒体渠道的数量。九个学生 - 运行的渠道包括三份报纸,两台广播电视台,以及几本杂志和杂志。 开始作为一个一页的杂志在1876年9月,该杂志的Schoolistic发行了每月两次,并声称是美国最古老的连续的大学新闻出版物,和其他杂志,TheJuggler,每年发行两次,并专注于学生文学和艺术作品。 多姆年刊每年发行。 报纸有不同的出版利益,与The Observer发表每日,主要报道大学和其他新闻,并由学生从Notre Dame和圣玛丽的学院。 与Scholastic和The Dome不同,The Observer是一个独立的公众作品,但没有教师顾',
 '细菌是生物细胞的一种类型. 它们构成一个大范围的亲生微生物. 通常几微米长,细菌有许多形状,从球到杖和螺旋。 细菌是地球上出现的第一个生命形式之一,并且存在于其大多数栖息地。',
 '生物学是研究生命的科学. 究竟什么是生命? 这可能听起来像一个愚蠢的问题,有一个显而易见的答案,但它并不容易定义生命. 例如,一个名为病毒学的生物学分支研究病毒,这些病毒表现出一些活体的特征,但缺乏其他。']

BYOT: Bring Your Own Translator

• The default translator requires no more than 16GB of memory.
• You can specify other pretrained translators at your own risk.
• Make sure src_lang and tgt_lang codes conform to that model.

Data Augmentation / Back-Translation

Back-translations useful for augmenting training data

t2t.Variator().transform([bacteria_str], src_lang='en')

Training / Finetuning

Finetune cross-lingual model on your data

result = t2t.Fitter().transform(["Hello, World! [TGT] 你好,世界!"],
  src_lang="en",
  tgt_lang="zh",
  num_epochs=10,
  save_directory="model_dir"
)

# load and use model from saved directory
t2t.Transformer.PRETRAINED_TRANSLATOR = "model_dir"
t2t.Translator().transform("Hello, World!", src_lang="en", tgt_lang="zh")

Identification

Identify the language of a text. Not yet accurate for short sequences (<10 tokens)

t2t.Identifier().transform(["Aj keď sa Buzz Aldrin stal až „druhým človekom“, ktorý otlačil svoju nohu do povrchu Mesiaca...",])

# Prediction
`['sk', 'Slovak']`

Serving

We aim to serve all functionality above

# Start web server
t2t.Server(host='0.0.0.0', port=80)

# Make POST requests
import socket
import requests

address = socket.gethostbyname(socket.getfqdn(socket.gethostname()))
url = f"http://{address}"
transformer = "Translator"
payload = {
  "input_lines": ["hello", "world"],
  "src_lang": "en",
  "tgt_lang": "ko",
}
r = requests.post(f"{url}/{transformer}", json=payload)
print(r.json()) #{'result': ['안녕하세요', '세계']}

# Indexer actions
r = requests.post(f"{url}/Indexer/add", json=payload)
r = requests.post(f"{url}/Indexer/size")
r = requests.post(f"{url}/Indexer/search", json=payload)
r = requests.post(f"{url}/Indexer/retrieve", json=payload)
payload["ids"] = [0,1]
r = requests.post(f"{url}/Indexer/remove", json=payload)

Questions?

For questions or help using Text2Text, please submit a GitHub issue.

Citation

To cite this work, use the following BibTeX citations.

@article{stfidf,
  doi = {10.48550/ARXIV.2209.14281},
  url = {https://arxiv.org/abs/2209.14281},
  author = {Wangperawong, Artit},
  title = {Multilingual Search with Subword TF-IDF},
  publisher = {arXiv},
  year = {2022},
}

@misc{text2text@2020,
  author={Wangperawong, Artit},
  title={Text2Text Language Modeling Toolkit},
  year={2020},
  publisher = {GitHub},
  journal = {GitHub repository},
  howpublished = {\url{https://github.com/artitw/text2text}},
  url = {https://github.com/artitw/text2text}
}

Contributing

There are many ways you can contribute:

1. Ask or answer a question in Issues
2. Share your experiences on using Text2Text
3. Report bugs with information to reproduce
4. Request for new features or functionality
5. Improve code by submitting a pull request with outputs demonstrating the change
6. See projects, comment on a task issue to get involved

Code of Conduct

Please adhere to our code of conduct when participating in this project.