search

brain-score / vision

A framework for evaluating models on their alignment to brain and behavioral measurements (50+ benchmarks)
http://brain-score.org
MIT License
115 stars 67 forks source link

Change brainio #963

Open YingtianDt opened 1 week ago

YingtianDt commented 1 week ago

This is not a PR

This is just to test the new brainio. Don't merge this branch.

YingtianDt commented 4 days ago

The tests on github include the tests on models but only one benchmark test (alexnet on MajajHong2015).

Therefore, here I add tests:

pytest brainscore_vision/benchmarks/freemanziemba2013/* 
brainscore_vision/benchmarks/geirhos2021/* 
brainscore_vision/benchmarks/imagenet_c/* 
brainscore_vision/benchmarks/majajhong2015/*

And the test report:

================================================ warnings summary =================================================
brainscore_vision/benchmarks/freemanziemba2013/test.py: 2 warnings
brainscore_vision/benchmarks/geirhos2021/test.py: 17 warnings
brainscore_vision/benchmarks/majajhong2015/test.py: 2 warnings
  /home/ytang/workspace/gather_index/brain-score/brainscore_vision/benchmark_helpers/screen.py:72: UserWarning: Pandas doesn't allow columns to be created via a new attribute name - see https://pandas.pydata.org/pandas-docs/stable/indexing.html#attribute-access
    converted_stimuli.original_paths = copy.deepcopy(stimulus_set.stimulus_paths)

brainscore_vision/benchmarks/geirhos2021/test.py: 60 warnings
  /home/ytang/workspace/gather_index/brain-score/brainscore_vision/benchmarks/geirhos2021/benchmark.py:119: SettingWithCopyWarning: 
  A value is trying to be set on a copy of a slice from a DataFrame.
  Try using .loc[row_indexer,col_indexer] = value instead

  See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
    assembly.attrs['stimulus_set']['condition'] = assembly.attrs['stimulus_set']['condition'].astype(str)

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
============================================= short test summary info =============================================
FAILED brainscore_vision/benchmarks/freemanziemba2013/test.py::test_ceilings[FreemanZiemba2013.V1-pls-expected0] - Exception: [ClientError('An error occurred (403) when calling the HeadObject operation: Forbidden'), ClientErr...
FAILED brainscore_vision/benchmarks/freemanziemba2013/test.py::test_ceilings[FreemanZiemba2013.V2-pls-expected1] - Exception: [ClientError('An error occurred (403) when calling the HeadObject operation: Forbidden'), ClientErr...
FAILED brainscore_vision/benchmarks/freemanziemba2013/test.py::test_self_regression[FreemanZiemba2013.V1-pls-4-expected0] - Exception: [ClientError('An error occurred (403) when calling the HeadObject operation: Forbidden'), ClientErr...
FAILED brainscore_vision/benchmarks/freemanziemba2013/test.py::test_self_regression[FreemanZiemba2013.V2-pls-4-expected1] - Exception: [ClientError('An error occurred (403) when calling the HeadObject operation: Forbidden'), ClientErr...
FAILED brainscore_vision/benchmarks/freemanziemba2013/test.py::test_FreemanZiemba2013[FreemanZiemba2013.V1-pls-expected0] - Exception: [ClientError('An error occurred (403) when calling the HeadObject operation: Forbidden'), ClientErr...
FAILED brainscore_vision/benchmarks/freemanziemba2013/test.py::test_FreemanZiemba2013[FreemanZiemba2013.V2-pls-expected1] - Exception: [ClientError('An error occurred (403) when calling the HeadObject operation: Forbidden'), ClientErr...
FAILED brainscore_vision/benchmarks/freemanziemba2013/test.py::test_amount_gray[FreemanZiemba2013.V1-pls-14-c3a633a13e736394f213ddf44bf124fe80cabe07-expected0] - Exception: [ClientError('An error occurred (403) when calling the HeadObject operation: Forbidden'), ClientErr...
FAILED brainscore_vision/benchmarks/freemanziemba2013/test.py::test_amount_gray[FreemanZiemba2013.V1-pls-6-c3a633a13e736394f213ddf44bf124fe80cabe07-expected1] - Exception: [ClientError('An error occurred (403) when calling the HeadObject operation: Forbidden'), ClientErr...
FAILED brainscore_vision/benchmarks/freemanziemba2013/test.py::test_amount_gray[FreemanZiemba2013.V2-pls-14-c3a633a13e736394f213ddf44bf124fe80cabe07-expected4] - Exception: [ClientError('An error occurred (403) when calling the HeadObject operation: Forbidden'), ClientErr...
FAILED brainscore_vision/benchmarks/freemanziemba2013/test.py::test_amount_gray[FreemanZiemba2013.V2-pls-6-c3a633a13e736394f213ddf44bf124fe80cabe07-expected5] - Exception: [ClientError('An error occurred (403) when calling the HeadObject operation: Forbidden'), ClientErr...
FAILED brainscore_vision/benchmarks/freemanziemba2013/test.py::test_repetitions[FreemanZiemba2013.V1-pls] - Exception: [ClientError('An error occurred (403) when calling the HeadObject operation: Forbidden'), ClientErr...
FAILED brainscore_vision/benchmarks/freemanziemba2013/test.py::test_repetitions[FreemanZiemba2013.V2-pls] - Exception: [ClientError('An error occurred (403) when calling the HeadObject operation: Forbidden'), ClientErr...
FAILED brainscore_vision/benchmarks/imagenet_c/test.py::TestImagenetC::test_groundtruth - Exception: [ClientError('An error occurred (403) when calling the HeadObject operation: Forbidden'), ClientErr...
FAILED brainscore_vision/benchmarks/imagenet_c/test.py::TestImagenetC_Category::test_groundtruth - Exception: [ClientError('An error occurred (403) when calling the HeadObject operation: Forbidden'), ClientErr...
FAILED brainscore_vision/benchmarks/majajhong2015/test.py::test_ceilings[MajajHong2015.V4-pls-expected0] - Exception: [ClientError('An error occurred (403) when calling the HeadObject operation: Forbidden'), ClientErr...
FAILED brainscore_vision/benchmarks/majajhong2015/test.py::test_ceilings[MajajHong2015.IT-pls-expected1] - Exception: [ClientError('An error occurred (403) when calling the HeadObject operation: Forbidden'), ClientErr...
FAILED brainscore_vision/benchmarks/majajhong2015/test.py::test_self_regression[MajajHong2015.V4-pls-8-expected0] - Exception: [ClientError('An error occurred (403) when calling the HeadObject operation: Forbidden'), ClientErr...
FAILED brainscore_vision/benchmarks/majajhong2015/test.py::test_self_regression[MajajHong2015.IT-pls-8-expected1] - Exception: [ClientError('An error occurred (403) when calling the HeadObject operation: Forbidden'), ClientErr...
FAILED brainscore_vision/benchmarks/majajhong2015/test.py::test_MajajHong2015[MajajHong2015.V4-pls-expected0] - Exception: [ClientError('An error occurred (403) when calling the HeadObject operation: Forbidden'), ClientErr...
FAILED brainscore_vision/benchmarks/majajhong2015/test.py::test_MajajHong2015[MajajHong2015.IT-pls-expected1] - Exception: [ClientError('An error occurred (403) when calling the HeadObject operation: Forbidden'), ClientErr...
FAILED brainscore_vision/benchmarks/majajhong2015/test.py::test_amount_gray[MajajHong2015.V4-pls-14-40a786ed8e13db10185ddfdbe07759d83a589e1c-expected0] - Exception: [ClientError('An error occurred (403) when calling the HeadObject operation: Forbidden'), ClientErr...
FAILED brainscore_vision/benchmarks/majajhong2015/test.py::test_amount_gray[MajajHong2015.V4-pls-6-40a786ed8e13db10185ddfdbe07759d83a589e1c-expected1] - Exception: [ClientError('An error occurred (403) when calling the HeadObject operation: Forbidden'), ClientErr...
FAILED brainscore_vision/benchmarks/majajhong2015/test.py::test_amount_gray[MajajHong2015.IT-pls-14-40a786ed8e13db10185ddfdbe07759d83a589e1c-expected4] - Exception: [ClientError('An error occurred (403) when calling the HeadObject operation: Forbidden'), ClientErr...
FAILED brainscore_vision/benchmarks/majajhong2015/test.py::test_amount_gray[MajajHong2015.IT-pls-6-40a786ed8e13db10185ddfdbe07759d83a589e1c-expected5] - Exception: [ClientError('An error occurred (403) when calling the HeadObject operation: Forbidden'), ClientErr...
FAILED brainscore_vision/benchmarks/majajhong2015/test.py::test_repetitions[MajajHong2015.V4-pls] - Exception: [ClientError('An error occurred (403) when calling the HeadObject operation: Forbidden'), ClientErr...
FAILED brainscore_vision/benchmarks/majajhong2015/test.py::test_repetitions[MajajHong2015.IT-pls] - Exception: [ClientError('An error occurred (403) when calling the HeadObject operation: Forbidden'), ClientErr...
======================= 26 failed, 120 passed, 1 skipped, 81 warnings in 1106.06s (0:18:26) =======================

It seems all failed tests are due to private access permission, and I am not sure about the one skipped test.

@mschrimpf Could you check if this is a good indication of ready to merge?

mschrimpf commented 3 days ago

indeed looks like failures are only due to private access, so I think this should be good.