search

bjascob / amrlib

A python library that makes AMR parsing, generation and visualization simple.
MIT License
216 stars 33 forks source link

load_stog_model load model fail #71

Closed ttjjlw closed 3 days ago

ttjjlw commented 1 week ago

stog = amrlib.load_stog_model(model_dir='/finetune-data/test/model_final/model_parse_xfm_bart_base-v0_1_0') raise error: OSError: We couldn't connect to 'https://huggingface.co' to load this file, couldn't find it in the cached files and it looks like facebook/bart-base is not the path to a directory containing a file named config.json.

model_dir has file as follow:

image

bjascob commented 6 days ago

This works fine for me. I'm guessing you can't connect to the HF hub to download the tokenizer associated with the model. You can test this manually with...

>>> from transformers import AutoTokenizer
>>> tokenizer = AutoTokenizer.from_pretrained('facebook/bart-base')

For debugging, it would be helpful to include which line in the code the OSError was raised so we know exactly what it was trying to do when the error occurred (ie.. provide the full traceback).

ttjjlw commented 4 days ago

Indeed, I cannot access the HF hub in my environment, but I have a question: why is 'facebook/bart-base' required when model_dir='/finetune-data/test/model_final/model_parse_xfm_bart_base-v0_1_0' is specified?

bjascob commented 4 days ago

If you look at the actual line that is failing, I suspect you'll find the issue is in loading the tokenizer, not the model (which is why I asked for the full traceback).

The tokenizer is named facebook/bart-base as well as the base model. The tokenizer is normally loaded/cached using the hub since it is not changed by the fine-tuning process. You can override load_stog_model() with the parameter tok_name_or_path, if you want to specify a disk location manually instead of using the hub logic to download and cache it for you.

ttjjlw commented 3 days ago

Thank you for clarifying the confusion, it works