search

pcwerk / lp-toolkit

Language Processing Toolkit
5 stars 0 forks source link

Research - FrontEnd store as Tokens with Bigram #10

Closed deep06bhakta closed 11 months ago

deep06bhakta commented 11 months ago

Synopsis

Summarize research here

Reference

deep06bhakta commented 11 months ago

https://colab.research.google.com/drive/1JMLa53HDuA-i7ZBmqV7ZnA3c_fvtXnx-?usp=sharing

Here is the code for tokenization of the data that is provided to the model.

deep06bhakta commented 11 months ago

https://github.com/karpathy/ng-video-lecture/blob/master/bigram.py

This is a simple bigram that would help us with the analogy of the data processing

Irha1520 commented 11 months ago

documented into wiki