Undefined variant leads to full combination of instances and other variants

[duylethanh]

Could you provide the experiments.yml?

[fabratu]

Hm, I should have posted the content right away. I fixed the problem after posting the issue. However I think it was something related to the following entries:

matrix:
  include:
  ...
    - experiments: [linkpredict-sig-RW]
      instsets: [citeseer]
      axes: ['embeddings']
      variants: ['b65536-citeseer', 'b131072-citeseer', 'b262144-citeseer']
    - experiments: [linkpredict-sig-RW]
      instsets: [karate]
      axes: ['embeddings']
      variants: ['b65536-karate', 'b131072-karate', 'b262144-karate']
    - experiments: [linkpredict-sig-RW]
      instsets: [cora]
      axes: ['embeddings']
      variants: ['b65536-cora', 'b131072-cora', 'b262144-cora']
    - experiments: [linkpredict-sig-RW]
      instsets: [flickr]
      axes: ['embeddings']
      variants: ['b256-flickr', 'b65536-flickr', 'b131072-flickr', 'b262144-flickr']
    - experiments: [linkpredict-sig-RW]
      instsets: [pubmed]
      axes: ['embeddings']
      variants: ['b65536-pubmed', 'b131072-pubmed', 'b262144-pubmed']
    - experiments: [linkpredict-sig-RW]
      instsets: [com-Youtube]
      axes: ['embeddings']
      variants: ['b65536-com-Youtube', 'b131072-com-Youtube', 'b262144-com-Youtube']

One variant had a misspelling, which therefore led to an undefined variant name. As a result, all instsets and also all previously defined variants were used for this experiment.

[meffertj]

I believe this is intended behaviour, take a look at the "run matrix" page: https://simexpal.readthedocs.io/en/latest/run_matrix.html#runmatrix

here it says: "Simexpal will then build the cross product of each experiment, instance set, variant and revision for each entry in include. Omitting a key in include will select all possible elements of the respective key, e.g., omitting revisions is equivalent to specifying the revisions key with all possible revisions."

Edit: my bad I just realized I overlooked the last sentence

[bernlu]

We should check for undefined variant names and print errors instead of silently ignoring them.