Integration with SummEval

[niansong1996]

@MuroriM Alex just sent out an email about SummEval being pip installable now, can you give some progress information here about integrating it with SummerTime?

[zhangir-azerbayev]

There are currently two main issues with SummEval integration:

1. The need to manually set an environment variable to use ROUGE
2. Overly verbose console output.

[niansong1996]

@zhangir-azerbayev Okay. We will discuss how big those issues are and whether we should use another evaluation library instead if a lot of us spending more time trying to fix it than it helping us.

[niansong1996]

@troyfeng116 can you describe the dependency issue you have with summEval? so that we can keep record

[zhangir-azerbayev]

@troyfeng116 Is the rouge environment variable still causing us problems? If so, we could rewrite st.evaluation.Rouge to import pyrouge instead of summeval (summeval rouge is a wrapper around pyrouge anyway). What do you think?

[niansong1996]

@zhangir-azerbayev I think it was working for me for the current main branch. What is the error you were getting?

[troyfeng116]

@zhangir-azerbayev The line we added to the notebook %env ROUGE_HOME = [...] works, but I wasn't able to get it working by following the console output and exporting it locally. But yeah since it's hard to explain how to configure the env variable we might want to consider cutting out the summeval dependency there and directly use pyrouge if we can?

[niansong1996]

I think any ROUGE package based on perl would need to do something similar. But maybe @zhangir-azerbayev can try to replace summEval with pyrouge if you find it easy to do?

[zhangir-azerbayev]

@niansong1996 @troyfeng116 @chatc It seems that you can get around the rouge problem on unix systems by including

py_path = sys.executable
os.environ["ROUGE_HOME"] = py_path + '/site-packages/summ_eval/ROUGE-1.5.5/'

in evaluation/__init__.py.

Generalizing to windows systems should be relatively straightforward using import os; os.name.

Is this solution robust? In particular, is site-packages/summ_eval/ROUGE-1.5.5 always the right directory to set the environment variable to? (the summ_eval error messages seems to suggest so).

Does anyone run python on non-unix servers?

[niansong1996]

This thread was discussed offline, and our current solution is in #32

[niansong1996]

I am keeping this issue alive since this seems to be a longer-term issue that will appear from time to time

[niansong1996]

This is more or less fixed and didn't appear to be buggy for a while, hence closing it now