MMAP inconsistent results

[mroavi]

The UAI 2014 Inference competition does not seem to provide reference solutions for the MMAP task. Therefore, we are validating our results with those produced by the Merlin solver, which can be found here: https://github.com/radum2275/merlin.

The solutions produced by our tool and those generated by Merlin are inconsistent for several problems. It would be valuable to determine the reason for this discrepancy.

MWE:

using Revise, BenchmarkTools, Test, Artifacts
using TensorInference

import Pkg;
Pkg.ensure_artifact_installed("uai2014", "Artifacts.toml");

problem_sets = dataset_from_artifact("uai2014")["MMAP"]
problems = [
  ("Segmentation", 12, TreeSA(ntrials=1, niters=2, βs=1:0.1:40)),
  ("Segmentation", 13, TreeSA(ntrials=1, niters=2, βs=1:0.1:40)), # fails!
  #("Segmentation", 14, TreeSA(ntrials=1, niters=2, βs=1:0.1:40))  # fails!
]
for (problem_set_name, id, optimizer) in problems
  problem = problem_sets[problem_set_name][id]
  model = MMAPModel(
    read_model(problem); optimizer,
    evidence=read_evidence(problem),
    queryvars=read_queryvars(problem)
  )
  _, solution = most_probable_config(model)
  test_result = @test solution == read_solution(problem)
  println(test_result)
end

[mroavi]

One possible explanation for the inconsistency between our results and Merlin's could be the algorithms used. While our method produces exact results, Merlin supports the WMB, JGLP, IJGP, and GIBBS algorithms, all of which are approximate.