Added results to benchmark suffix

[MatteoWohlrapp]

Added time when the benchmark script was executed as a suffix. For that, I am getting the folder name from the configuration file and extending this name with the timestamp, then adding it to the snakemake command to overwrite the existing output folder path. See format below:

[smilesun]

# only plot
# snakemake --cores 1 -s "domainlab/exp_protocol/benchmark.smk" --configfile "examples/yaml/demo_benchmark.yaml" --keep-going --allowed-rules agg_partial_results
# snakemake --cores 1 -s "domainlab/exp_protocol/benchmark.smk" --configfile "examples/yaml/demo_benchmark.yaml" --keep-going --allowed-rules gen_plots
,method,model,params
0,method_diva,diva,"{'gamma_d': 10000.0, 'zy_dim': 32}"
1,method_diva,diva,"{'gamma_d': 10000.0, 'zy_dim': 64}"
2,method_diva,diva,"{'gamma_d': 310000.0, 'zy_dim': 32}"
3,method_diva,diva,"{'gamma_d': 310000.0, 'zy_dim': 64}"
4,method_hduva,hduva,{'zy_dim': 32}
5,method_hduva,hduva,{'zy_dim': 64}
6,erm,erm,{'lr': 0.001}
7,erm,erm,{'lr': 0.0001}
cat: zoutput/benchmarks/mnist_benchmark_grid/results.csv: No such file or directory
Error: Process completed with exit code 1.

[MatteoWohlrapp]

When did you get this output?

[smilesun]

When did you get this output?

i just copied the CI log, maybe it gives you a hint what went wrong.

[codecov-commenter]

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 95.17%. Comparing base (44271c8) to head (e67e212).

Additional details and impacted files

```diff @@ Coverage Diff @@ ## master #821 +/- ## ======================================= Coverage 95.17% 95.17% ======================================= Files 128 128 Lines 5080 5080 ======================================= Hits 4835 4835 Misses 245 245 ``` | [Flag](https://app.codecov.io/gh/marrlab/DomainLab/pull/821/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=marrlab) | Coverage Δ | | |---|---|---| | [unittests](https://app.codecov.io/gh/marrlab/DomainLab/pull/821/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=marrlab) | `95.17% <ø> (ø)` | | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=marrlab#carryforward-flags-in-the-pull-request-comment) to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

[smilesun]

When did you get this output?

i just copied the CI log, maybe it gives you a hint what went wrong.

I just added a wildcard to the benchmark results folder in the CI script, hopefuly it can fix the issue

[smilesun]

When did you get this output?

i just copied the CI log, maybe it gives you a hint what went wrong.

I just added a wildcard to the benchmark results folder in the CI script, hopefuly it can fix the issue

When did you get this output?

i just copied the CI log, maybe it gives you a hint what went wrong.

I just added a wildcard to the benchmark results folder in the CI script, hopefuly it can fix the issue

CI passed now

[smilesun]

I clicked two times rerun the CI, but it looks like it did not re-run

[smilesun]

still the same error with poetry

NameError: name 'extend' is not defined Error: Process completed with exit code 1.

@agisga , do you have any idea what is going on?

[agisga]

@smilesun I just looked through the commit history of DomId, and I had solved an issue that seems to be the same error with "name 'extend' is not defined"! What worked for me was adding the following to pyproject.toml:

setuptools = "^69.5.1"

There seems to be a compatibility bug between backpack and newer versions of setuptools (70.x). Hope that helps!

[smilesun]

=========================== short test summary info ============================
FAILED tests/test_benchmark_plots.py::test_benchm_plots - AttributeError: 'Le...
FAILED tests/test_exp_protocol.py::test_agg_main - AttributeError: 'Legend' o...
=========== 2 failed, 96 passed, [581](https://github.com/marrlab/DomainLab/actions/runs/9697814097/job/26762854758?pr=821#step:7:582)7 warnings in 2332.02s (0:38:52) ===========

[smilesun]

2024-06-27T14:37:42.3965653Z tests/test_utils.py ..                                                   [100%]
2024-06-27T14:37:42.3966347Z 
2024-06-27T14:37:42.3966597Z =================================== FAILURES ===================================
2024-06-27T14:37:42.3967304Z ______________________________ test_benchm_plots _______________________________
2024-06-27T14:37:42.3967882Z 
2024-06-27T14:37:42.3968070Z     def test_benchm_plots():
2024-06-27T14:37:42.3968605Z         """
2024-06-27T14:37:42.3969007Z         test benchmark plots
2024-06-27T14:37:42.3969481Z         """
2024-06-27T14:37:42.3969870Z >       gen_benchmark_plots(
2024-06-27T14:37:42.3970499Z             "domainlab/zdata/ztest_files/aggret_res_test1",
2024-06-27T14:37:42.3971238Z             "zoutput/benchmark_plots_test/outp1",
2024-06-27T14:37:42.3971858Z             use_param_index=False,
2024-06-27T14:37:42.3972340Z         )
2024-06-27T14:37:42.3972535Z 
2024-06-27T14:37:42.3972729Z tests/test_benchmark_plots.py:11: 
2024-06-27T14:37:42.3973401Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
2024-06-27T14:37:42.3974363Z domainlab/utils/generate_benchmark_plots.py:57: in gen_benchmark_plots
2024-06-27T14:37:42.3975233Z     gen_plots(raw_df, output_dir, use_param_index)
2024-06-27T14:37:42.3976013Z domainlab/utils/generate_benchmark_plots.py:102: in gen_plots
2024-06-27T14:37:42.3976732Z     scatterplot_matrix(
2024-06-27T14:37:42.3977820Z domainlab/utils/generate_benchmark_plots.py:303: in scatterplot_matrix
2024-06-27T14:37:42.3978870Z     sns.move_legend(g_p, loc="upper right", bbox_to_anchor=(1.0, 1.0), ncol=1)
2024-06-27T14:37:42.3979839Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
2024-06-27T14:37:42.3980416Z 
2024-06-27T14:37:42.3981270Z obj = <seaborn.axisgrid.PairGrid object at 0x7f29cc9a57e0>, loc = 'upper right'
2024-06-27T14:37:42.3982302Z kwargs = {'bbox_to_anchor': (1.0, 1.0), 'ncol': 1}
2024-06-27T14:37:42.3983080Z Grid = <class 'seaborn.axisgrid.Grid'>
2024-06-27T14:37:42.3983887Z old_legend = <matplotlib.legend.Legend object at 0x7f29cc24fa00>
2024-06-27T14:37:42.3985032Z legend_func = <bound method FigureBase.legend of <Figure size 1250x1200 with 14 Axes>>
2024-06-27T14:37:42.3985812Z 
2024-06-27T14:37:42.3986016Z     def move_legend(obj, loc, **kwargs):
2024-06-27T14:37:42.3986573Z         """
2024-06-27T14:37:42.3987511Z         Recreate a plot's legend at a new location.
2024-06-27T14:37:42.3988155Z     
2024-06-27T14:37:42.3988806Z         The name is a slight misnomer. Matplotlib legends do not expose public
2024-06-27T14:37:42.3990011Z         control over their position parameters. So this function creates a new legend,
2024-06-27T14:37:42.3991262Z         copying over the data from the original object, which is then removed.
2024-06-27T14:37:42.3992500Z     
2024-06-27T14:37:42.3992894Z         Parameters
2024-06-27T14:37:42.3993353Z         ----------
2024-06-27T14:37:42.3993807Z         obj : the object with the plot
2024-06-27T14:37:42.3994582Z             This argument can be either a seaborn or matplotlib object:
2024-06-27T14:37:42.3995309Z     
2024-06-27T14:37:42.3995942Z             - :class:`seaborn.FacetGrid` or :class:`seaborn.PairGrid`
2024-06-27T14:37:42.3997056Z             - :class:`matplotlib.axes.Axes` or :class:`matplotlib.figure.Figure`
2024-06-27T14:37:42.3997854Z     
2024-06-27T14:37:42.3998223Z         loc : str or int
2024-06-27T14:37:42.3998934Z             Location argument, as in :meth:`matplotlib.axes.Axes.legend`.
2024-06-27T14:37:42.3999672Z     
2024-06-27T14:37:42.4000014Z         kwargs
2024-06-27T14:37:42.4000726Z             Other keyword arguments are passed to :meth:`matplotlib.axes.Axes.legend`.
2024-06-27T14:37:42.4001526Z     
2024-06-27T14:37:42.4001834Z         Examples
2024-06-27T14:37:42.4002236Z         --------
2024-06-27T14:37:42.4002573Z     
2024-06-27T14:37:42.4003001Z         .. include:: ../docstrings/move_legend.rst
2024-06-27T14:37:42.4003550Z     
2024-06-27T14:37:42.4003844Z         """
2024-06-27T14:37:42.4004463Z         # This is a somewhat hackish solution that will hopefully be obviated by
2024-06-27T14:37:42.4005495Z         # upstream improvements to matplotlib legends that make them easier to
2024-06-27T14:37:42.4006290Z         # modify after creation.
2024-06-27T14:37:42.4006738Z     
2024-06-27T14:37:42.4007254Z         from seaborn.axisgrid import Grid  # Avoid circular import
2024-06-27T14:37:42.4007959Z     
2024-06-27T14:37:42.4008483Z         # Locate the legend object and a method to recreate the legend
2024-06-27T14:37:42.4009231Z         if isinstance(obj, Grid):
2024-06-27T14:37:42.4009750Z             old_legend = obj.legend
2024-06-27T14:37:42.4010311Z             legend_func = obj.figure.legend
2024-06-27T14:37:42.4010932Z         elif isinstance(obj, mpl.axes.Axes):
2024-06-27T14:37:42.4011552Z             old_legend = obj.legend_
2024-06-27T14:37:42.4012111Z             legend_func = obj.legend
2024-06-27T14:37:42.4012731Z         elif isinstance(obj, mpl.figure.Figure):
2024-06-27T14:37:42.4013361Z             if obj.legends:
2024-06-27T14:37:42.4013961Z                 old_legend = obj.legends[-1]
2024-06-27T14:37:42.4014532Z             else:
2024-06-27T14:37:42.4014948Z                 old_legend = None
2024-06-27T14:37:42.4015470Z             legend_func = obj.legend
2024-06-27T14:37:42.4015998Z         else:
2024-06-27T14:37:42.4016689Z             err = "`obj` must be a seaborn Grid or matplotlib Axes or Figure instance."
2024-06-27T14:37:42.4017843Z             raise TypeError(err)
2024-06-27T14:37:42.4018334Z     
2024-06-27T14:37:42.4018703Z         if old_legend is None:
2024-06-27T14:37:42.4019268Z             err = f"{obj} has no legend attached."
2024-06-27T14:37:42.4019906Z             raise ValueError(err)
2024-06-27T14:37:42.4020406Z     
2024-06-27T14:37:42.4020909Z         # Extract the components of the legend we need to reuse
2024-06-27T14:37:42.4021678Z >       handles = old_legend.legendHandles
2024-06-27T14:37:42.4022630Z E       AttributeError: 'Legend' object has no attribute 'legendHandles'

[smilesun]

I tried sometime to fix the seborn problem above: currently, matplotlib==3.8.2 ; python_version >= "3.9" and python_version < "4.0" and seaborn==0.12.2 ; python_version >= "3.9" and python_version < "4.0" in the requirements.txt, but i could not reproduce the error on my laptop after chaning the version of matplotlib and seaborn, any idea @MatteoWohlrapp @agisga

[smilesun]

the error occurs at rule:

    - name: Upload coverage to Codecov
      uses: codecov/codecov-action@v1

[smilesun]

2024-07-01T14:18:47.4395753Z old_legend = <matplotlib.legend.Legend object at 0x7f71b8923a00>
2024-07-01T14:18:47.4396366Z legend_func = <bound method FigureBase.legend of <Figure size 1250x1200 with 14 Axes>>
2024-07-01T14:18:47.4396787Z 
2024-07-01T14:18:47.4396904Z     def move_legend(obj, loc, **kwargs):
2024-07-01T14:18:47.4397217Z         """
2024-07-01T14:18:47.4397812Z         Recreate a plot's legend at a new location.
2024-07-01T14:18:47.4398176Z     
2024-07-01T14:18:47.4398540Z         The name is a slight misnomer. Matplotlib legends do not expose public
2024-07-01T14:18:47.4399189Z         control over their position parameters. So this function creates a new legend,
2024-07-01T14:18:47.4399834Z         copying over the data from the original object, which is then removed.
2024-07-01T14:18:47.4400273Z     
2024-07-01T14:18:47.4400475Z         Parameters
2024-07-01T14:18:47.4400726Z         ----------
2024-07-01T14:18:47.4400979Z         obj : the object with the plot
2024-07-01T14:18:47.4401409Z             This argument can be either a seaborn or matplotlib object:
2024-07-01T14:18:47.4401804Z     
2024-07-01T14:18:47.4402163Z             - :class:`seaborn.FacetGrid` or :class:`seaborn.PairGrid`
2024-07-01T14:18:47.4402771Z             - :class:`matplotlib.axes.Axes` or :class:`matplotlib.figure.Figure`
2024-07-01T14:18:47.4403205Z     
2024-07-01T14:18:47.4403412Z         loc : str or int
2024-07-01T14:18:47.4403802Z             Location argument, as in :meth:`matplotlib.axes.Axes.legend`.
2024-07-01T14:18:47.4404210Z     
2024-07-01T14:18:47.4404401Z         kwargs
2024-07-01T14:18:47.4404801Z             Other keyword arguments are passed to :meth:`matplotlib.axes.Axes.legend`.
2024-07-01T14:18:47.4405267Z     
2024-07-01T14:18:47.4405452Z         Examples
2024-07-01T14:18:47.4405695Z         --------
2024-07-01T14:18:47.4406093Z     
2024-07-01T14:18:47.4406361Z         .. include:: ../docstrings/move_legend.rst
2024-07-01T14:18:47.4406689Z     
2024-07-01T14:18:47.4406866Z         """
2024-07-01T14:18:47.4407237Z         # This is a somewhat hackish solution that will hopefully be obviated by
2024-07-01T14:18:47.4407847Z         # upstream improvements to matplotlib legends that make them easier to
2024-07-01T14:18:47.4408304Z         # modify after creation.
2024-07-01T14:18:47.4408576Z     
2024-07-01T14:18:47.4408883Z         from seaborn.axisgrid import Grid  # Avoid circular import
2024-07-01T14:18:47.4409271Z     
2024-07-01T14:18:47.4409580Z         # Locate the legend object and a method to recreate the legend
2024-07-01T14:18:47.4410006Z         if isinstance(obj, Grid):
2024-07-01T14:18:47.4410308Z             old_legend = obj.legend
2024-07-01T14:18:47.4410635Z             legend_func = obj.figure.legend
2024-07-01T14:18:47.4411000Z         elif isinstance(obj, mpl.axes.Axes):
2024-07-01T14:18:47.4411344Z             old_legend = obj.legend_
2024-07-01T14:18:47.4411649Z             legend_func = obj.legend
2024-07-01T14:18:47.4411992Z         elif isinstance(obj, mpl.figure.Figure):
2024-07-01T14:18:47.4412333Z             if obj.legends:
2024-07-01T14:18:47.4412673Z                 old_legend = obj.legends[-1]
2024-07-01T14:18:47.4412986Z             else:
2024-07-01T14:18:47.4413214Z                 old_legend = None
2024-07-01T14:18:47.4413507Z             legend_func = obj.legend
2024-07-01T14:18:47.4413898Z         else:
2024-07-01T14:18:47.4414548Z             err = "`obj` must be a seaborn Grid or matplotlib Axes or Figure instance."
2024-07-01T14:18:47.4415393Z             raise TypeError(err)
2024-07-01T14:18:47.4415851Z     
2024-07-01T14:18:47.4416109Z         if old_legend is None:
2024-07-01T14:18:47.4416712Z             err = f"{obj} has no legend attached."
2024-07-01T14:18:47.4417350Z             raise ValueError(err)
2024-07-01T14:18:47.4417811Z     
2024-07-01T14:18:47.4418286Z         # Extract the components of the legend we need to reuse
2024-07-01T14:18:47.4419012Z >       handles = old_legend.legendHandles
2024-07-01T14:18:47.4419922Z E       AttributeError: 'Legend' object has no attribute 'legendHandles'
2024-07-01T14:18:47.4420541Z 
2024-07-01T14:18:47.4421168Z ../../../.cache/pypoetry/virtualenvs/domainlab-okngg5wM-py3.10/lib/python3.10/site-packages/seaborn/utils.py:456: AttributeError
2024-07-01T14:18:47.4421977Z ________________________________ test_agg_main _________________________________
2024-07-01T14:18:47.4422291Z 
2024-07-01T14:18:47.4422775Z bm_config = 'zoutput/test', agg_output_file = 'zoutput/test/results.csv'
2024-07-01T14:18:47.4424194Z agg_expected_output = 'param_index, method, algo, epos, te_d, seed, params, acc, precision, recall, specificity, f1, aurocy\n0, diva, diva, ..., caltech, 1, "{\'gamma_y\': 70037, \'zy_dim\': 48}", 0.7307692, 0.557971, 0.5333333, 0.5333333, 0.5297158, 0.73333335'
2024-07-01T14:18:47.4425202Z 
2024-07-01T14:18:47.4425427Z     def test_agg_main(bm_config, agg_output_file, agg_expected_output):
2024-07-01T14:18:47.4425958Z         """Testing the csv aggregation from a full directory."""
2024-07-01T14:18:47.4426394Z >       agg_main(bm_config, skip_plotting=True)
2024-07-01T14:18:47.4426643Z 
2024-07-01T14:18:47.4426757Z tests/test_exp_protocol.py:148: 

[smilesun]

I tried sometime to fix the seborn problem above: currently, matplotlib==3.8.2 ; python_version >= "3.9" and python_version < "4.0" and seaborn==0.12.2 ; python_version >= "3.9" and python_version < "4.0" in the requirements.txt, but i could not reproduce the error on my laptop after chaning the version of matplotlib and seaborn, any idea @MatteoWohlrapp @agisga

@MatteoWohlrapp @agisga , I FIXED it, in fact, the unit test is run with the poetry dependencies, instead of requirements.txt, i have updated pyproj.yaml and now the unit test passed for seaborn and matplotlib

[smilesun]

Thanks for the nice work @MatteoWohlrapp , i will now test it on the cluster if the benchmark works as expected.

It works on the cluster!