search

borglab / SwiftFusion

Apache License 2.0
115 stars 13 forks source link

Using TensorBoard.dev for the benchmarks #81

Open ProfFan opened 4 years ago

ProfFan commented 4 years ago

Before we can use tensorboard.dev, we need to figure out how to write the tensorboard format in Swift, as unlike comet.ml in tensorboard.dev we do not have REST APIs for uploading data.

marcrasi commented 4 years ago

Are there Python libraries that can write these formats? If so, one quick way to get started would be to import those using Swift's Python interop.

ProfFan commented 4 years ago

@marcrasi Great point, I'll browse around and see :)

ProfFan commented 4 years ago

Tested this one: https://github.com/KrakenCL/KrakenKit and it seems to be perfect for our usage

shabalind commented 4 years ago

swift-benchmark is one step away from supporting JSON output (https://github.com/google/swift-benchmark/pull/43).

Once it is merged one can do something along the lines:

  1. Create a python wrapper that invokes swift benchmarks as a subcommand with --format json flag.
  2. Consume JSON output from subprocess.
  3. Call the python-based APIs to report it to the dashboarding service (e.g. TensorBoard here)

That's how we integrate with TensorFlow's perfzero infra in swift-models for example.

ProfFan commented 4 years ago

@shabalind Wow this seems to be pretty straightforward :) Also appears to be better in integrating into the current GTSAM benchmarks. Will try shortly.

Thank you very much for the suggestion! @shabalind

shabalind commented 4 years ago

@ProfFan Happy to help! 😃

marcrasi commented 4 years ago

@ProfFan, how is the benchmarking going? Do you need any help with anything? Would you be interested in having a meeting with @shabalind to discuss swift-benchmark features that could help?

ProfFan commented 4 years ago

@marcrasi I looked at google/swift-benchmark#43 and it's not merged? Should we meet today? I have 3 meetings tomorrow :(

marcrasi commented 4 years ago

How about Monday? I sent a proposed time in a calendar invite. @shabalind is in Europe and therefore probably not available for the rest of today.

I'm wondering if swift-benchmark currently has enough features to track the things that you want to track in the benchmarks. Have you investigated that? For example, I saw your benchmarks had things like a plot of error vs time and I don't see anything in swift-benchmark that supports things like that yet. swift-benchmark can collect single numbers from benchmarks (https://github.com/google/swift-benchmark/pull/40) but I don't see a way to collect time-varying numbers.

Yeah, google/swift-benchmark#43 is not merged. Is that blocking you? You could try locally updating Package.swift to point at that branch so that you can develop against it before it gets merged. Though it looks like its dependency got merged recently, so hopefully it'll be merged soon too.

ProfFan commented 4 years ago

I'm wondering if swift-benchmark currently has enough features to track the things that you want to track in the benchmarks.

I see the problem. I think I got sidetracked somehow, as our "benchmark" is actually "tracing", not "benchmarking". I think a meeting is still valuable, as we can discuss the possibility of adding such a functionality in swift-benchmark.

I'll just move the current Comet.ml code to use https://github.com/KrakenCL/KrakenKit for now and make sure that I have the benchmarks ready by our meeting tomorrow.

shabalind commented 4 years ago

Just a small update: google/swift-benchmark#43 is on its way to be merged by the end of the week.

shabalind commented 4 years ago

https://github.com/google/swift-benchmark/pull/43 is merged!