Token file not found "token"

[harshakhmk]

I am using fastembed-rs package, to create embeddings for text input While I am running my application as a shared library on Android, it is giving below error

hf_hub: Token file not found "token"
ureq:::stream connecting to huggingface.co 

use hf_hub::api::sync::Api;

let api = Api::new().unwrap();

let repo = api.model("bert-base-uncased".to_string());
let _filename = repo.get("config.json").unwrap();

// filename  is now the local location within hf cache of the config.json file

[harshakhmk]

Using the above code in my sample rust crate which is built as a shared library and used in Android app, app is crashing

[harshakhmk]

@Narsil any leads on this would be highly helpful

[polarathene]

Here is the rough flow of what's happening when you call Api::new():

Click to view

https://github.com/huggingface/hf-hub/blob/9d6502f5bc2e69061c132f523c76a76dad470477/src/api/sync.rs#L258-L262 https://github.com/huggingface/hf-hub/blob/9d6502f5bc2e69061c132f523c76a76dad470477/src/api/sync.rs#L99-L108 https://github.com/huggingface/hf-hub/blob/9d6502f5bc2e69061c132f523c76a76dad470477/src/lib.rs#L194-L208 https://github.com/huggingface/hf-hub/blob/9d6502f5bc2e69061c132f523c76a76dad470477/src/api/sync.rs#L117-L129 https://github.com/huggingface/hf-hub/blob/9d6502f5bc2e69061c132f523c76a76dad470477/src/api/sync.rs#L160-L182 For the `cache.token()` call in `from_cache()`: https://github.com/huggingface/hf-hub/blob/9d6502f5bc2e69061c132f523c76a76dad470477/src/lib.rs#L41-L68

---

So what's happening is:

1. A cache directory for HF to use is checked via the ENV HF_HOME, otherwise it defaults to ~/.cache/huggingface/hub for the cache directory.
2. Then when the API struct is created, it takes this path and checks the parent dir (omitting hub) to look for a file named token, thus default path is ~/.cache/huggingface/token.
3. That token file check is part of calling from_cache() internally to create the API builder struct, which itself calls cache.token() on that path value from step 1. When this does not exist you get the info log you encountered.
4. That presently cannot be avoided easily, you'd have to manually add a lot more code to create an API builder without the cache.token() call.

That shouldn't be causing a crash though.