re-run EMNLP 2015 paper using PRA not SFE

[samehkamaleldin]

I'm trying to run the experiment from EMNLP 2015 using PRA with random walks but I can't.

I've used the following configuration file ~/pra/examples/experiment_specs/nell/final_emnlp2015/sfe_bfs_pra_anyrel_PRA.json

{
  "operation": {
    "type": "train and test",
    "features": {
      "path finder": {
        "walks per source": 100,
        "path finding iterations": 3,
        "path accept policy": "paired-only"
      },
      "path selector": {
        "number of paths to keep": 1000
      },
      "path follower": {
        "walks per path": 50,
        "matrix accept policy": "all-targets"
      }
    },
    "learning": {
      "l1 weight": 0.005,
      "l2 weight": 1
    }
  }
}

then I ran it like the following:

sbt "run ./examples/ sfe_bfs_pra_anyrel_PRA"

Then chosed experiment runner and it omited the following error

[info] Running edu.cmu.ml.rtw.pra.experiments.ExperimentRunner ./examples/ sfe_bfs_pra_anyrel_PRA
[info] Found 353 experiment specs, and kept 1 of them
[info] Running PRA from spec file ./examples/experiment_specs/nell/final_emnlp2015/sfe_bfs_pra_anyrel_PRA.json
[info] Split not found at ./examples/splits/nell_with_negatives/; creating it...
[error] Exception in thread "main" java.util.NoSuchElementException: None.get
[error]         at scala.None$.get(Option.scala:347)
[error]         at scala.None$.get(Option.scala:345)
[error]         at edu.cmu.ml.rtw.pra.data.SplitCreator.<init>(SplitCreator.scala:32)
[error]         at edu.cmu.ml.rtw.pra.experiments.Driver.createSplitIfNecessary(Driver.scala:286)
[error]         at edu.cmu.ml.rtw.pra.experiments.Driver.runPra(Driver.scala:57)
[error]         at edu.cmu.ml.rtw.pra.experiments.ExperimentRunner$.runPraFromSpec(ExperimentRunner.scala:70)
[error]         at edu.cmu.ml.rtw.pra.experiments.ExperimentRunner$$anonfun$runPra$1.apply(ExperimentRunner.scala:55)
[error]         at edu.cmu.ml.rtw.pra.experiments.ExperimentRunner$$anonfun$runPra$1.apply(ExperimentRunner.scala:55)
[error]         at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)
[error]         at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)
[error]         at scala.collection.mutable.ArraySeq.foreach(ArraySeq.scala:74)
[error]         at scala.collection.TraversableLike$class.map(TraversableLike.scala:245)
[error]         at scala.collection.AbstractTraversable.map(Traversable.scala:104)
[error]         at edu.cmu.ml.rtw.pra.experiments.ExperimentRunner$.runPra(ExperimentRunner.scala:55)
[error]         at edu.cmu.ml.rtw.pra.experiments.ExperimentRunner$.main(ExperimentRunner.scala:25)
[error]         at edu.cmu.ml.rtw.pra.experiments.ExperimentRunner.main(ExperimentRunner.scala)
java.lang.RuntimeException: Nonzero exit code returned from runner: 1
        at scala.sys.package$.error(package.scala:27)
[trace] Stack trace suppressed: run last compile:run for the full output.
[error] (compile:run) Nonzero exit code returned from runner: 1
[error] Total time: 3 s, completed 18-Apr-2016 13:38:56

How can I manage to make it working with PRA ?

[matt-gardner]

There are a few issues here. First, if you want to run PRA on the data from the EMNLP 2015 paper, you should use this experiment configuration - there's no need to create your own. Assuming you've downloaded the data from the paper website, that should fix the split creation issue you're seeing (and, for comparing to the experiments in the 2015 paper, you need to use paired-targets-only in the path follower parameters, not all-targets).

However, the default parameters have changed since the EMNLP 2015 paper, and I think this experiment spec will probably crash with an error about invalid parameters. If it does, the fix is to add two more parameters: (1) under "features", add "type": "pra", and (2) under "path finder", add "type": "RandomWalkPathFinder".

[samehkamaleldin]

Even the baseline.json file you mentioned is not working and it omits this error

and it also says that it's using subgraphs not pra

[info] Running edu.cmu.ml.rtw.pra.experiments.ExperimentRunner ./examples final_emnlp2015/baseline
[info] Found 329 experiment specs, and kept 2 of them
[info] Running PRA from spec file ./examples/experiment_specs/nell/final_emnlp2015/baseline.json
[info] Split found in ./examples/splits/final_nell_split_with_negatives/
[info] 
[info] 
[info] 
[info] 
[info] Running PRA for relation concept:riverflowsthroughcity
[info] feature type being used is subgraphs
[error] Exception in thread "main" java.lang.IllegalStateException: Malformed parameters under operation -> features: unexpected key: path selector
[error]         at com.mattg.util.JsonHelper$$anonfun$ensureNoExtras$1.apply(SpecFileReader.scala:126)
[error]         at com.mattg.util.JsonHelper$$anonfun$ensureNoExtras$1.apply(SpecFileReader.scala:123)
[error]         at scala.collection.immutable.List.map(List.scala:277)
[error]         at com.mattg.util.JsonHelper$.ensureNoExtras(SpecFileReader.scala:123)
[error]         at edu.cmu.ml.rtw.pra.features.SubgraphFeatureGenerator.<init>(SubgraphFeatureGenerator.scala:41)
[error]         at edu.cmu.ml.rtw.pra.features.NodePairSubgraphFeatureGenerator.<init>(SubgraphFeatureGenerator.scala:165)
[error]         at edu.cmu.ml.rtw.pra.features.FeatureGenerator$.create(FeatureGenerator.scala:96)
[error]         at edu.cmu.ml.rtw.pra.operations.TrainAndTest.runRelation(Operation.scala:76)
[error]         at edu.cmu.ml.rtw.pra.experiments.Driver$$anonfun$runPra$1.apply(Driver.scala:82)
[error]         at edu.cmu.ml.rtw.pra.experiments.Driver$$anonfun$runPra$1.apply(Driver.scala:76)
[error]         at scala.collection.immutable.Stream.foreach(Stream.scala:594)
[error]         at edu.cmu.ml.rtw.pra.experiments.Driver.runPra(Driver.scala:76)
[error]         at edu.cmu.ml.rtw.pra.experiments.ExperimentRunner$.runPraFromSpec(ExperimentRunner.scala:70)
[error]         at edu.cmu.ml.rtw.pra.experiments.ExperimentRunner$$anonfun$runPra$1.apply(ExperimentRunner.scala:55)
[error]         at edu.cmu.ml.rtw.pra.experiments.ExperimentRunner$$anonfun$runPra$1.apply(ExperimentRunner.scala:55)
[error]         at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)
[error]         at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)
[error]         at scala.collection.mutable.ArraySeq.foreach(ArraySeq.scala:74)
[error]         at scala.collection.TraversableLike$class.map(TraversableLike.scala:245)
[error]         at scala.collection.AbstractTraversable.map(Traversable.scala:104)
[error]         at edu.cmu.ml.rtw.pra.experiments.ExperimentRunner$.runPra(ExperimentRunner.scala:55)
[error]         at edu.cmu.ml.rtw.pra.experiments.ExperimentRunner$.main(ExperimentRunner.scala:25)
[error]         at edu.cmu.ml.rtw.pra.experiments.ExperimentRunner.main(ExperimentRunner.scala)
java.lang.RuntimeException: Nonzero exit code returned from runner: 1
        at scala.sys.package$.error(package.scala:27)
[trace] Stack trace suppressed: run last compile:run for the full output.
[error] (compile:run) Nonzero exit code returned from runner: 1
[error] Total time: 2 s, completed 18-Apr-2016 18:05:25

[samehkamaleldin]

I don't know why but the baseline.jsonfile in my local version is different from online rep version, maybe because I checked out v3.3.

I've create a new file with the content of online rep baseline.json file and is currently running okay. I'll close the issue as soon as the experiment end successfully

[matt-gardner]

This was what I mentioned in the second paragraph of my comment above - I told you it would crash, and that you had to change the parameters. After writing the comment, I made the change in the repo and pushed the changes, which is why the versions were different.

[samehkamaleldin]

Experiment ended successfully.