brain-score / core

MIT License
2 stars 5 forks source link

Fixed an I/O bug where 'get_scoring_info' would generate a non-JSON serializable string #53

Closed shehadak closed 7 months ago

shehadak commented 7 months ago

This code fixes a bug where the output printed by get_scoring_info is not JSON-serializable. In particular, running a simple command like:

python -c 'from brainscore_core.plugin_management.parse_plugin_changes import get_scoring_info; get_scoring_info("brainscore_language/models/gpt/__init__.py ", "brainscore_language")'

prints:

{'modifies_plugins': True, 'changed_plugins': {'models': ['gpt'], 'benchmarks': [], 'data': [], 'metrics': []}, 'is_automergeable': True, 'run_score': 'True', 'new_models': 'distilgpt2 gpt2 gpt2-medium gpt2-large gpt2-xl gpt-neo-2.7B gpt-neo-1.3B', 'new_benchmarks': ''}

This output doesn't follow JSON format, particularly because it uses single quotes ' instead of double quotes ", and because it uses True and False instead of true and false. Therefore, attempting to parse it using, for example, jq, results in an error:

echo "{'modifies_plugins': True, 'changed_plugins': {'models': ['gpt'], 'benchmarks': [], 'data': [], 'metrics': []}, 'is_automergeable': True, 'run_score': 'True', 'new_models': 'distilgpt2 gpt2 gpt2-medium gpt2-large gpt2-xl gpt-neo-2.7B gpt-neo-1.3B', 'new_benchmarks': ''}" | jq .

results in:

parse error: Invalid numeric literal at line 1, column 20

This PR fixes this issue by ensuring that the output from get_scoring_info is JSON-serializable.

Additionally, since the primary use case of this method is communicating information with a JSON decoder, this PR updates the unit tests for get_scoring_info to use json.loads instead of ast.literal_eval.