search

yeesian / ArchGDAL.jl

A high level API for GDAL - Geospatial Data Abstraction Library
https://yeesian.github.io/ArchGDAL.jl/stable/
Other
142 stars 26 forks source link

Add benchmarks to ArchGDAL CI #248

Closed mathieu17g closed 3 years ago

mathieu17g commented 3 years ago

Fixes #193

Here is a first step with one PkgBenchmark.jl benchmark suite, benchmarking shapefile reading and converting to table with GDAL.jl (w or w/o vsizip) and Shapefile.jl

julia> using PkgBenchmark

julia> using ArchGDAL

julia> benchmarkpkg(ArchGDAL)
PkgBenchmark: Running benchmarks...
PkgBenchmark: using benchmark tuning data in /Users/Mathieu/.../dev/ArchGDAL/benchmark/tune.json

(1/1) benchmarking "shapefile_to_table"...
  (1/3) benchmarking "frenchroads_with_GDAL.jl_via_vsizip"...
  done (took 5.725868093 seconds)
  (2/3) benchmarking "frenchroads_with_GDAL.jl"...
  done (took 5.672228116 seconds)
  (3/3) benchmarking "frenchroads_with_Shapefile.jl"...
  done (took 5.685400644 seconds)
done (took 17.703220046 seconds)
Benchmarking 100%|█████████████████████████████████████████████████████| Time: 0:00:19
Benchmarkresults:
    Package: /Users/Mathieu/.../dev/ArchGDAL
    Date: 5 Oct 2021 - 10:47
    Package commit: 1e5fe5
    Julia commit: ae8452
    BenchmarkGroup:
        1-element BenchmarkTools.BenchmarkGroup:
          tags: []
          "shapefile_to_table" => 3-element BenchmarkTools.BenchmarkGroup:
                  tags: []
                  "frenchroads_with_GDAL.jl_via_vsizip" => Trial(304.691 ms)
                  "frenchroads_with_GDAL.jl" => Trial(194.496 ms)
                  "frenchroads_with_Shapefile.jl" => Trial(66.557 ms)

julia>

Next step: integrate it with Github actions via BenchmarkCI.jl

felixcremer commented 3 years ago

Could we use the same code as in test/remotefiles.jl to get the actual data, instead of putting the whole shapefile into this repository?

mathieu17g commented 3 years ago

@felixcremer sure. For now, I'm just trying to make it work as it is.

mathieu17g commented 3 years ago

Well, it works locally but not on Github...

On Github, the commit string is not recognized (see checks' results):

[ Info: Using temporary project `/tmp/BenchmarkCI_jl_hAQsda`.
fatal: invalid object name '175600ee'.
fatal: invalid object name '175600ee'.
fatal: invalid object name '175600ee'.
ERROR: failed process: Process(`git --no-pager rev-parse 175600ee:`, ProcessExited(128)) [128]

Locally, there is no issue:

julia> BenchmarkCI.judge(; target = "175600ee", baseline = "ac838398")
   Resolving package versions...
    Updating `/private/var/folders/q5/_tzrxy_j71vckk28xwlh1sth0000gn/T/BenchmarkCI_jl_Xhxz9d/Project.toml`
  [c9ce4bd3] + ArchGDAL v0.7.4 `~/.../dev/ArchGDAL`
  [6e4b80f9] + BenchmarkTools v1.2.0
  [32113eaa] + PkgBenchmark v0.2.12
  [8e980c4a] + Shapefile v0.7.1
  [bd369af6] + Tables v1.6.0
  [a5390f91] + ZipFile v0.9.4
    Updating `/private/var/folders/q5/_tzrxy_j71vckk28xwlh1sth0000gn/T/BenchmarkCI_jl_Xhxz9d/Manifest.toml`
  [621f4979] + AbstractFFTs v1.0.1
  [1520ce14] + AbstractTrees v0.3.4
  [79e6a3ab] + Adapt v3.3.1
  [c9ce4bd3] + ArchGDAL v0.7.4 `~/.../dev/ArchGDAL`
  [6e4b80f9] + BenchmarkTools v1.2.0
  [fa961155] + CEnum v0.4.1
  [d360d2e6] + ChainRulesCore v1.7.2
  [3da002f7] + ColorTypes v0.11.0
  [c3611d14] + ColorVectorSpace v0.9.7
  [5ae59095] + Colors v0.12.8
  [34da2185] + Compat v3.39.0
  [75c7ada1] + DBFTables v0.2.3
  [9a962f9c] + DataAPI v1.9.0
  [e2d170a0] + DataValueInterfaces v1.0.0
  [3c3547ce] + DiskArrays v0.2.12
  [ffbed154] + DocStringExtensions v0.8.5
  [53c48c17] + FixedPointNumbers v0.8.4
  [add2ef01] + GDAL v1.2.4
  [68eda718] + GeoFormatTypes v0.3.0
  [cf35fbd7] + GeoInterface v0.5.5
  [a2bd30eb] + Graphics v1.1.1
  [a09fc81d] + ImageCore v0.9.3
  [92d709cd] + IrrationalConstants v0.1.0
  [82899510] + IteratorInterfaceExtensions v1.0.0
  [692b3bcd] + JLLWrappers v1.3.0
  [682c06a0] + JSON v0.21.2
  [1d6d02ad] + LeftChildRightSiblingTrees v0.1.2
  [2ab3a3ac] + LogExpFunctions v0.3.3
  [dbb5928d] + MappedArrays v0.4.1
  [e94cdb99] + MosaicViews v0.3.3
  [77ba4419] + NaNMath v0.3.5
  [6fe1bfb0] + OffsetArrays v1.10.7
  [5432bcbf] + PaddedViews v0.5.10
  [69de0a69] + Parsers v2.0.5
  [32113eaa] + PkgBenchmark v0.2.12
  [21216c6a] + Preferences v1.2.2
  [33c8b6b6] + ProgressLogging v0.1.4
  [3cdcf5f2] + RecipesBase v1.1.2
  [189a3867] + Reexport v1.2.2
  [8e980c4a] + Shapefile v0.7.1
  [276daf66] + SpecialFunctions v1.7.0
  [cae243ae] + StackViews v0.1.1
  [3783bdb8] + TableTraits v1.0.1
  [bd369af6] + Tables v1.6.0
  [62fd8b95] + TensorCore v0.1.1
  [5d786b92] + TerminalLoggers v0.1.4
  [ea10d353] + WeakRefStrings v0.6.2
  [a5390f91] + ZipFile v0.9.4
  [2e619515] + Expat_jll v2.2.10+0
  [a7073274] + GDAL_jll v300.202.100+0
  [d604d12d] + GEOS_jll v3.9.0+0
  [aacddb02] + JpegTurbo_jll v2.1.0+0
  [89763e89] + Libtiff_jll v4.3.0+0
  [d3a379c0] + LittleCMS_jll v2.12.0+0
  [643b3616] + OpenJpeg_jll v2.4.0+0
  [efe28fd5] + OpenSpecFun_jll v0.5.5+0
  [58948b4f] + PROJ_jll v700.202.100+0
  [76ed43ae] + SQLite_jll v3.36.0+0
  [3161d3a3] + Zstd_jll v1.5.0+0
  [06c338fa] + libgeotiff_jll v1.6.0+1
  [b53b4c65] + libpng_jll v1.6.38+0
  [0dad84c5] + ArgTools
  [56f22d72] + Artifacts
  [2a0f44e3] + Base64
  [ade2ca70] + Dates
  [8bb1440f] + DelimitedFiles
  [8ba89e20] + Distributed
  [f43a241f] + Downloads
  [b77e0a4c] + InteractiveUtils
  [b27032c2] + LibCURL
  [76f85450] + LibGit2
  [8f399da3] + Libdl
  [37e2e46d] + LinearAlgebra
  [56ddb016] + Logging
  [d6f4376e] + Markdown
  [a63ad114] + Mmap
  [ca575930] + NetworkOptions
  [44cfe95a] + Pkg
  [de0858da] + Printf
  [9abbd945] + Profile
  [3fa0cd96] + REPL
  [9a3f8284] + Random
  [ea8e919c] + SHA
  [9e88b42a] + Serialization
  [1a1011a3] + SharedArrays
  [6462fe0b] + Sockets
  [2f01184e] + SparseArrays
  [10745b16] + Statistics
  [fa267f1f] + TOML
  [a4e569a6] + Tar
  [8dfed614] + Test
  [cf7118a7] + UUIDs
  [4ec0a83e] + Unicode
  [e66e0078] + CompilerSupportLibraries_jll
  [deac9b47] + LibCURL_jll
  [29816b5a] + LibSSH2_jll
  [c8ffd9c3] + MbedTLS_jll
  [14a3606d] + MozillaCACerts_jll
  [05823500] + OpenLibm_jll
  [83775a58] + Zlib_jll
  [8e850ede] + nghttp2_jll
  [3f19e933] + p7zip_jll
[ Info: Using temporary project `/var/folders/q5/_tzrxy_j71vckk28xwlh1sth0000gn/T/BenchmarkCI_jl_Xhxz9d`.
PkgBenchmark: Running benchmarks...
  Activating environment at `/var/folders/q5/_tzrxy_j71vckk28xwlh1sth0000gn/T/BenchmarkCI_jl_Xhxz9d/Project.toml`
  No Changes to `/private/var/folders/q5/_tzrxy_j71vckk28xwlh1sth0000gn/T/BenchmarkCI_jl_Xhxz9d/Project.toml`
  No Changes to `/private/var/folders/q5/_tzrxy_j71vckk28xwlh1sth0000gn/T/BenchmarkCI_jl_Xhxz9d/Manifest.toml`
PkgBenchmark: using benchmark tuning data in /Users/Mathieu/.../dev/ArchGDAL/benchmark/tune.json
(1/1) benchmarking "shapefile_to_table"...
  (1/3) benchmarking "frenchroads_with_GDAL.jl_via_vsizip"...
  done (took 5.748433276 seconds)
  (2/3) benchmarking "frenchroads_with_GDAL.jl"...
  done (took 5.731729496 seconds)
  (3/3) benchmarking "frenchroads_with_Shapefile.jl"...
  done (took 5.688544235 seconds)
done (took 17.816182621 seconds)
Benchmarking 100%|█████████████████████████████████████████████████████| Time: 0:00:19
PkgBenchmark: benchmark results written to .benchmarkci/result-target.json
PkgBenchmark: Running benchmarks...
  Activating environment at `/var/folders/q5/_tzrxy_j71vckk28xwlh1sth0000gn/T/BenchmarkCI_jl_Xhxz9d/Project.toml`
  No Changes to `/private/var/folders/q5/_tzrxy_j71vckk28xwlh1sth0000gn/T/BenchmarkCI_jl_Xhxz9d/Project.toml`
  No Changes to `/private/var/folders/q5/_tzrxy_j71vckk28xwlh1sth0000gn/T/BenchmarkCI_jl_Xhxz9d/Manifest.toml`
PkgBenchmark: using benchmark tuning data in /Users/Mathieu/.../dev/ArchGDAL/benchmark/tune.json
(1/1) benchmarking "shapefile_to_table"...
  (1/3) benchmarking "frenchroads_with_GDAL.jl_via_vsizip"...
  done (took 5.641582915 seconds)
  (2/3) benchmarking "frenchroads_with_GDAL.jl"...
  done (took 5.703490303 seconds)
  (3/3) benchmarking "frenchroads_with_Shapefile.jl"...
  done (took 5.662686884 seconds)
done (took 17.641346895 seconds)
Benchmarking 100%|█████████████████████████████████████████████████████| Time: 0:00:19
PkgBenchmark: benchmark results written to .benchmarkci/result-baseline.json
┌ Info: Finish running benchmarks.
│ * Target: 37 seconds
└ * Baseline: 37 seconds
Benchmarkjudgement (target / baseline):
    Package: /Users/Mathieu/.../dev/ArchGDAL
    Dates: 5 Oct 2021 - 18:25 / 5 Oct 2021 - 18:26
    Package commits: 175600 / ac8383
    Julia commits: ae8452 / ae8452

julia>

Let's try with the full commit ids:

mathieu17g commented 3 years ago

It still does not work but the error message is different:

[ Info: Using temporary project `/tmp/BenchmarkCI_jl_5RFnD0`.
fatal: path 'benchmark' exists on disk, but not in '175600eef7cb1420176eae0d6d4490ec941727bf'
fatal: path 'src' exists on disk, but not in '175600eef7cb1420176eae0d6d4490ec941727bf'
fatal: path '' does not exist in '175600eef7cb1420176eae0d6d4490ec941727bf'
ERROR: failed process: Process(`git --no-pager rev-parse 175600eef7cb1420176eae0d6d4490ec941727bf:`, ProcessExited(128)) [128]
mathieu17g commented 3 years ago

At last! It works :-)

mathieu17g commented 3 years ago

Could we use the same code as in test/remotefiles.jl to get the actual data, instead of putting the whole shapefile into this repository?

@felixcremer I have cleaned the benchmark/data/ directory and added it to .gitignore. So now the repository should not be burdened

Since it is a mockup needing a first merge, before being able to modifying the baseline to master, it should do.

We may convert the download of https://www.data.gouv.fr/fr/datasets/r/ea2b85d7-b29e-4bb4-baa5-ac3bb618c67e to a download of a copy put in https://github.com/yeesian/ArchGDALDatasets/blob/master/ , after this first merge.

mathieu17g commented 3 years ago

@mathieu17g I've granted you access to https://github.com/yeesian/ArchGDALDatasets now

Thanks, I will duplicate test/remotefiles.jl in benchmark/ and adapt the list of remote files to be downloaded for benchmark usage

yeesian commented 3 years ago

Thank you!