search

MilesCranmer / AirspeedVelocity.jl

Easily benchmark a Julia package over its commit history
https://astroautomata.com/AirspeedVelocity.jl/
Apache License 2.0
93 stars 5 forks source link

Error: Key "25" not found #48

Open AZEY4 opened 3 months ago

AZEY4 commented 3 months ago

When creating plots using workflows, benchpkgplot fails with the error:

ERROR: LoadError: KeyError: key "25" not found Stacktrace: [1] getindex @ ./dict.jl:498 [inlined] [2] (::AirspeedVelocity.PlotUtils.var"#2#5")(d::Dict{String, Real}) @ AirspeedVelocity.PlotUtils ./none:0 [3] iterate @ ./generator.jl:47 [inlined] [4] collect_to!(dest::Vector{Float64}, itr::Base.Generator{Vector{Dict{String, Real}}, AirspeedVelocity.PlotUtils.var"#2#5"}, offs::Int64, st::Int64) @ Base ./array.jl:892 [5] collect_to_with_first!(dest::Vector{Float64}, v1::Float64, itr::Base.Generator{Vector{Dict{String, Real}}, AirspeedVelocity.PlotUtils.var"#2#5"}, st::Int64) @ Base ./array.jl:870 [6] collect(itr::Base.Generator{Vector{Dict{String, Real}}, AirspeedVelocity.PlotUtils.var"#2#5"}) @ Base ./array.jl:844 [7] create_line_plot(data::Vector{Dict{String, Real}}, names::Vector{String}, title::String) @ AirspeedVelocity.PlotUtils ~/.julia/packages/AirspeedVelocity/4cFoN/src/PlotUtils.jl:35 [8] combined_plots(combined_results::OrderedCollections.OrderedDict{String, OrderedCollections.OrderedDict}; npart::Int64) @ AirspeedVelocity.PlotUtils ~/.julia/packages/AirspeedVelocity/4cFoN/src/PlotUtils.jl:100 [9] combined_plots @ ~/.julia/packages/AirspeedVelocity/4cFoN/src/PlotUtils.jl:93 [inlined] [10] benchpkgplot(package_name::String; rev::String, input_dir::String, output_dir::String, npart::Int64, format::String) @ AirspeedVelocity.BenchPkgPlot ~/.julia/packages/AirspeedVelocity/4cFoN/src/BenchPkgPlot.jl:42 [11] command_main(ARGS::Vector{String}) @ AirspeedVelocity.BenchPkgPlot ~/.julia/packages/Comonicon/F3QqZ/src/codegen/julia.jl:343 [12] command_main() @ AirspeedVelocity.BenchPkgPlot ~/.julia/packages/Comonicon/F3QqZ/src/codegen/julia.jl:90 [13] top-level scope @ ~/.julia/bin/benchpkgplot:14 in expression starting at /home/lfaria/.julia/bin/benchpkgplot:14

https://github.com/IntegralEquations/HMatrices.jl/actions/runs/9763110081/job/26948180966?pr=65

MilesCranmer commented 3 months ago

Thanks. Yes seems like there should be a fallback strategy in these cases.

However it also looks like some of your benchmarks don’t have error bars, which might be an issue in its own right: IMG_1239 meaning they only run a single time. Is that expected? If not, maybe run with --tune? (Or manually set the settings in the @benchmarkable, like a longer allowed time for running a benchmark)

AZEY4 commented 3 months ago

So, When I run this workflow it fails mostly or passes in less cases. I use --tune when I run the benchmarks. I suppose it falls because there are no error bars and some statistics for the key "25" is not calculated. At first sight, I guess it should be checked if the key "25" exists as the same way as "75". https://github.com/MilesCranmer/AirspeedVelocity.jl/blob/493a0e01a32b2dbd0b81698bfd6259918e6ce5f6/src/PlotUtils.jl#L34