'Ignore' not respected in schema

[daholste]

System information

• OS version/distro: Windows 10
• .NET Version (eg., dotnet --info): .NET Core 2.1

Issue

• What did you do? Ran MML command line: execgraph "C:\Benchmarking\automl_graph.json"

Contents of automl_.graph.json:

{
  "Inputs": {
    "file_train": "D:\\SplitDatasets\\ExcitementFG2_train.csv",
    "file_test": "D:\\SplitDatasets\\ExcitementFG2_valid.csv"
  },
  "Nodes": [
    {
      "Inputs": {
        "CustomSchema": "sep=, col=Label:R4:0 col=Features:R4:1-13 col=Cat:TX:14 col=Cat01:TX:15 col=Ignore:TX:16,25 col=Cat02:TX:17 col=Cat03:TX:18 col=Cat04:TX:19 col=Cat05:TX:20 col=Cat06:TX:21 col=Cat07:TX:22 col=Cat08:TX:23 col=Cat09:TX:24 col=Cat10:TX:26 col=Cat11:TX:27 col=Cat12:TX:28 col=Cat13:TX:29 col=Cat14:TX:30 col=Cat15:TX:31 col=Cat16:TX:32 col=Cat17:TX:33 col=Cat18:TX:34 col=Cat19:TX:35 col=Cat20:TX:36 col=Cat21:TX:37 col=Cat22:TX:38 col=Cat23:TX:39",
        "InputFile": "$file_train"
      },
      "Name": "Data.CustomTextLoader",
      "Outputs": {
        "Data": "$data_train"
      }
    },
    {
      "Inputs": {
        "CustomSchema": "sep=, col=Label:R4:0 col=Features:R4:1-13 col=Cat:TX:14 col=Cat01:TX:15 col=Ignore:TX:16,25 col=Cat02:TX:17 col=Cat03:TX:18 col=Cat04:TX:19 col=Cat05:TX:20 col=Cat06:TX:21 col=Cat07:TX:22 col=Cat08:TX:23 col=Cat09:TX:24 col=Cat10:TX:26 col=Cat11:TX:27 col=Cat12:TX:28 col=Cat13:TX:29 col=Cat14:TX:30 col=Cat15:TX:31 col=Cat16:TX:32 col=Cat17:TX:33 col=Cat18:TX:34 col=Cat19:TX:35 col=Cat20:TX:36 col=Cat21:TX:37 col=Cat22:TX:38 col=Cat23:TX:39",
        "InputFile": "$file_test"
      },
      "Name": "Data.CustomTextLoader",
      "Outputs": {
        "Data": "$data_test"
      }
    },
    {
      "Inputs": {
        "BatchSize": 3,
        "StateArguments": {
          "Name": "AutoMlState",
          "Settings": {
            "Engine": {
              "Name": "Rocket",
              "Settings": {}
            },
            "Metric": "Accuracy",
            "TerminatorArgs": {
              "Name": "IterationLimited",
              "Settings": {
                "FinalHistoryLength": 100
              }
            },
            "TrainerKind": "SignatureBinaryClassifierTrainer"
          }
        },
        "TestingData": "$data_test",
        "TrainingData": "$data_train"
      },
      "Name": "Models.PipelineSweeper",
      "Outputs": {
        "Results": "$output_data",
        "State": "$xyz"
      }
    }
  ],
  "Outputs": {
    "output_data": "C:\\Benchmarking\\01-ResultsOut.csv"
  }
}

• What happened? Even though Ignore:TX:16,25 is specified in the schema, the learners start expensive / slow text transforms / n-gram extractions on the columns

• What did you expect? The columns to be ignored

[najeeb-kazmi]

You can simply remove col=Ignore:TX:16,25 from your schema. This will not load those columns in the first place.

[daholste]

I did try that, but got an exception when trying to load col 17 (after first ignored col, ie w/o first loading/mentioning col 16)

[rogancarr]

Closing: Microsoft.ML.PipelineInference has been removed from the repository.