search

aws / fmeval

Foundation Model Evaluations Library
http://aws.github.io/fmeval
Apache License 2.0
153 stars 40 forks source link

Cannot import FactualKnowledge module #226

Closed dferguson992 closed 3 months ago

dferguson992 commented 3 months ago

Importing FactualKnowledge module into my jupyter notebook on sagemaker studio gives the following:

image

nathanng17 commented 3 months ago

Thanks for reaching out @dferguson992! Some users have also reported similar issues when using the FMEval library with the newer SageMaker Studio experience. The issue seems to come from the pip package installations. One workaround for this issue would be to force reinstall the dependency packages causing the issue, which include fsspec and pytz from my experience, with the following commands: 

%pip install --force-reinstall fsspec
%pip install --force-reinstall pytz

You may encounter errors with uninstalling a previous version of those packages because of missing files, so you may need to manually remove those old package version directories similar to below (you can find the affected directories using the filepaths of the missing files from pip's warning messages): 

rm -r /opt/conda/lib/python3.10/site-packages/fsspec-2023.6.0.dist-info/
rm -r /opt/conda/lib/python3.10/site-packages/pytz-2023.3.dist-info/

The newer versions of the packages will still be installed, so you should be able to import the FactualKnowledge module in your notebook.

I've noticed that this issue does tend to come back, so a more permanent workaround would be to instead use SageMaker Studio Classic to run your notebook. The issue does not seem to affect notebooks run in SageMaker Studio Classic, so you should be able to run the FMEval library without an issue.

dferguson992 commented 3 months ago

Thanks that seemed to work for awhile until I got to the next few cells.

Without modifying the default example, I got up to the "Run Evaluation" section where I received this error: image

The stack trace ends with: image

I haven't modified anything in the example, except to print the ModelRunner object confirming it is not None.
Is there a way to fix this?

danielezhu commented 3 months ago

The example notebooks are currently using the $model_input placeholder, as the latest code in the main branch of this repo have renamed the placeholder from $feature to $model_input. However, this change has not been released to pypi yet, so to fix this, just change $model_input in the notebook to $feature.

dferguson992 commented 3 months ago

That worked! Thank you!