Add ifbo

[Neeratyoy]

Adding ifBO and refactoring a bunch of freeze-thaw based multi-fidelity algorithms. Working example here.

Existing issues or concerns:

• [x] Adding parallelization support
  • Need resolving some issues
• [ ] Tblogger from the example does not seem to work
  • Only config_0_0 has the tbevent files
  • (did not reproduce) On restart, there was an error from the tbevent file-not-there
• [ ] Resolving Python dependency #140

[Neeratyoy]

@eddiebergman @TarekAbouChakra is the tblogging supported currently? If you can install this branch and run this example and view Tensorboard, it is no longer working.

[eddiebergman]

Just fyi, [PR](#140) didn't work as intended, you can just do PR #140 :)

[eddiebergman]

As for the PR about python dependancies, just get ifbo in first and we can do the python dep thing after

[eddiebergman]

Regarding tblogger, I can't initially see what the issue is. Seems that the tblogger at each iteration will get the trial that is currently being evaluated from the runtime.py. The runtime.py will ask the optimizer for the next config to evaluate (which returns the config dict as well as the ID), sets that as the currently running trial and then sends it to run_pipeline(). Inside run_pipeline() is where the tblogger.log() is called, at which point it asks the runtime.py, "hey, what config is currently being evaluated?". Should be the exact same as what's used for determining the config directory.

[eddiebergman]

Think I found the issue:

This line: https://github.com/automl/neps/blob/0025d3d41db51a6198c7b19464513371eb9cb9ff/neps/plot/tensorboard_eval.py#L316-L317

Basically it only initializes once, where intialization is in charge of deciding where to write to: https://github.com/automl/neps/blob/0025d3d41db51a6198c7b19464513371eb9cb9ff/neps/plot/tensorboard_eval.py#L109-L135

[Neeratyoy]

@eddiebergman how should we go about this PR?

Parallelization works and I happy for now with local testing.

Tblogger is the biggest bummer here, along with test cases failing.