Make token numpy arrays - Githubissues

artificialwisdomai / origin

Artificial Wisdom™ Cloud Platform

Apache License 2.0

2 stars 4 forks source link

Make token numpy arrays #168

Closed sdake closed 2 months ago

sdake commented 3 months ago

This code does the following using a source dataset:

tokenize
determine sequence to token
determine token to sequence
generate embedded tokens