search

simonw / ttok

Count and truncate text based on tokens
Apache License 2.0
247 stars 7 forks source link

Does `ttok` need an internet connection? #11

Open NightMachinery opened 7 months ago

NightMachinery commented 7 months ago

ttok either does not work or is very very slow when the internet connection is bad/non-existent. Is this a known design behavior or a bug?

Related:

NightMachinery commented 3 months ago

@simonw Bumping this; I am considering disabling token counting, but I am not sure if the culprit is really this.

NightMachinery commented 1 month ago

Bumping this again.

NightMachinery commented 1 month ago

ttok seems to use HuggingFace, which is most probably the culprit. See this issue. I suspect some similar env flag must be set for offline usage.