Missing modules: There should be eval.py in the ultra folder but there isn't.

[HuieL]

Missing eval.py, the pretrianed models are not accessable directly by git or huggingface.

[migalkin]

I don't see why a specific eval.py should be in the folder - could you elaborate on that?

The pretrained models are literally in the ckpts folder here in the git repo and in the model.safetensors on the huggingface hub, you can download them when cloning the repo

[HuieL]

I guess I loaded the model in one wrong way. I'll be appreciate if you can help me to do that.

Now, I am trying to load the model with (1) huggingface model by model = AutoModel.from_pretrained("mgalkin/ultra_50g", trust_remote_code=True) and (2) cloned repo to load checkpoint locally. With the first way, I got the ImportError and it requires a pip install ultra. And with the second way, I got the error of missing modules.

Actully, I prefer to load models directly with transformers. Did I load the model with the wrong way or wrong code?

[migalkin]

Oh I see, so you are trying to get the HF Hub setup running. In that case, you should run the code from the cloned ultra_50g folder, eg, after cloning the repo from the HF Hub and installing the requirements, you can create a script ex.py with smth like this:

from modeling import UltraForKnowledgeGraphReasoning
from transformers import AutoModel
from ultra.datasets import CoDExSmall
from ultra.eval import test

#model = UltraForKnowledgeGraphReasoning.from_pretrained("mgalkin/ultra_50g")
model = AutoModel.from_pretrained("mgalkin/ultra_50g", trust_remote_code=True)
dataset = CoDExSmall(root="./datasets/")
test(model, mode="test", dataset=dataset, gpus=None)
# Expected results
# mrr:      0.497697
# hits@10:  0.685175

Running python ex.py you should get the numbers in the expected results comment.

[HuieL]

Thanks a lot. It works on the datasets now!