AutoXGBoostCARMA Error

[mbanco]

Hi Adrian, I don't know where the error is in this code, when I use GridTune = TRUE:

# # Load data
data <- data.table::fread('https://www.dropbox.com/s/2str3ek4f4cheqi/walmart_train.csv?dl=1')

data <- data[(Store == 1 & Dept <= 15) ,];

# Ensure series have no missing dates (also remove series with more than 25% missing values)
data <- TimeSeriesFill(
  data,
  DateColumnName = 'Date',
  GroupVariables = c('Store','Dept'),
  TimeUnit = 'weeks',
  FillType = 'maxmax',
  MaxMissingPercent = 0.25,
  SimpleImpute = TRUE);

data.table::setnames(data, "Weekly_Sales", "datos");

# Set negative numbers to 0
data <- data[, datos := data.table::fifelse(datos < 0, 0, datos)];

# Remove IsHoliday column
data[, IsHoliday := NULL];

# Create xregs (this is the include the categorical variables instead of utilizing only the interaction of them)
xregs <- data[, .SD, .SDcols = c('Date', 'Store', 'Dept')];

# Change data types
data[, ':=' (Store = as.character(Store), Dept = as.character(Dept))];
xregs[, ':=' (Store = as.character(Store), Dept = as.character(Dept))];

# Build forecast
XGBoostResults <- AutoXGBoostCARMA(

  # Data Artifacts
  data = data,
  NonNegativePred = FALSE,
  RoundPreds = FALSE,
  TargetColumnName = 'datos',
  DateColumnName = 'Date',
  HierarchGroups = NULL,
  GroupVariables = c('Store','Dept'),
  TimeUnit = 'weeks',
  TimeGroups = c('weeks','months'),

  # Data Wrangling Features
  ZeroPadSeries = NULL,
  DataTruncate = FALSE,
  SplitRatios = c(0.9, 0.1),
  PartitionType = 'timeseries',
  AnomalyDetection = NULL,
  EncodingMethod = 'binary',

  # Productionize
  FC_Periods = 52,
  TrainOnFull = TRUE,
  NThreads = parallel::detectCores()-1,
  Timer = TRUE,
  DebugMode = FALSE,
  SaveDataPath = NULL,
  PDFOutputPath = NULL,

  # Target Transformations
  TargetTransformation = TRUE,
  Methods = c('BoxCox', 'Asinh', 'Asin', 'Log', 'LogPlus1', 'Sqrt', 'Logit','YeoJohnson'),
  Difference = FALSE,

  # Features
  Lags = list('weeks' = seq(1L, 10L, 1L), 'months' = seq(1L, 5L, 1L)),
  MA_Periods = list('weeks' = seq(5L, 20L, 5L), 'months' = seq(2L, 10L, 2L)),
  SD_Periods = NULL,
  Skew_Periods = NULL,
  Kurt_Periods = NULL,
  Quantile_Periods = NULL,
  Quantiles_Selected = c('q10','q90'),
  XREGS = NULL,
  FourierTerms = 4,
  CalendarVariables = c('week', 'wom', 'month', 'quarter'),
  HolidayVariable = c('USPublicHolidays','EasterGroup','ChristmasGroup','OtherEcclesticalFeasts'),
  HolidayLookback = NULL,
  HolidayLags = 1,
  HolidayMovingAverages = 1:2,
  TimeTrendVariable = TRUE,

  # ML eval args
  TreeMethod = 'hist',
  EvalMetric = 'RMSE',
  LossFunction = 'reg:squarederror',

  # ML grid tuning
  GridTune = TRUE,
  GridEvalMetric = 'r2',
  ModelCount = 5,
  MaxRunsWithoutNewWinner = 20L,
  MaxRunMinutes = 2L,

  # ML args
  NTrees = seq(200L, 2000L, 200L),
  LearningRate = c(0.01,0.02,0.03,0.04),
  MaxDepth = seq(4L, 16L, 2L),
  MinChildWeight = seq(1.0, 10.0, 1.0),
  SubSample = seq(0.55, 1.0, 0.05),
  ColSampleByTree = seq(0.55, 1.0, 0.05));

Results <- XGBoostResults$Forecast;

Best Regards

Mauricio

[AdrianAntico]

@mbanco I just ran through your code example and it finished up without any issues.

> grid <- XGBoostResults$GridMetrics
> knitr::kable(grid[, .SD, .SDcols = c(names(grid)[1:5])])

| GridNumber| RunNumber| RunTime| EvalMetric| TreesBuilt|
|----------:|---------:|-------:|----------:|----------:|
|          0|         1|   35.40|          1|       2000|
|          1|         2|    7.58|          1|        200|
|          2|         3|   13.72|          1|        400|
|          3|         4|   21.97|          1|        600|
|          4|         5|   15.23|          1|        400|
|          5|         6|   35.79|          1|       1000|

How far along did the procedure run on your end? Did it stop training (which means it went on to create the evaluation metrics and interpretation plots)?

My best guess at this point is to have you upgrade ggplot2 to its newest version and rerun. I know there was a breaking change on ggplot2's side of things and I updated my plotting functions to incorporate the updated syntax.

PS - I updated your comment by adding the code into a code block.

https://docs.github.com/en/github/writing-on-github/working-with-advanced-formatting/creating-and-highlighting-code-blocks

[mbanco]

Hi Adrian, this is the error:

[1995] train-rmse:0.001685 [1996] train-rmse:0.001685 [1997] train-rmse:0.001685 [1998] train-rmse:0.001685 [1999] train-rmse:0.001685 [2000] train-rmse:0.001685 Error in xgb.DMatrix(newdata, missing = missing) : xgb.DMatrix does not support construction from NULL

I have installed ggplot2 version 3.3.5.

Thanks

Mauricio

[AdrianAntico]

@mbanco When is the last time you re-installed RemixAutoML?

[mbanco]

A while ago, I re-installed the package, but it gives the same error.

[AdrianAntico]

Edit:

I think I see the issue - if you want to run a grid tune (GridTune = TRUE) you need to have TrainOnFull = FALSE (in the Productionize section). When you get the results from the grid tune run you can evaluate the performances and then update your ML parameters based on the top performer and then set TrainOnFull = TRUE to rerun and generate a forecast.

[mbanco]

Thanks you Adrain, I'm going to try that way.

[AdrianAntico]

Keep me posted. I was able to get it to run on my side. When I first ran it, I had TrainOnFull set to TRUE and it worked.

[mbanco]

Thanks you. I tell you that I have run the code on two PCs and it gives the same error. Is it possible that there is some incompatibility with some installed package?

[AdrianAntico]

You could be running into memory errors. Can you provide you compute specs?

Also, give this script a shot, as it has some of the ML args for tuning toned down a bit:

# Load data
data <- data.table::fread('https://www.dropbox.com/s/2str3ek4f4cheqi/walmart_train.csv?dl=1')

# Ensure series have no missing dates (also remove series with more than 25% missing values)
data <- RemixAutoML::TimeSeriesFill(
  data,
  DateColumnName = 'Date',
  GroupVariables = c('Store','Dept'),
  TimeUnit = 'weeks',
  FillType = 'maxmax',
  MaxMissingPercent = 0.25,
  SimpleImpute = TRUE)

# Set negative numbers to 0
data <- data[, Weekly_Sales := data.table::fifelse(Weekly_Sales < 0, 0, Weekly_Sales)]

# Remove IsHoliday column
data[, IsHoliday := NULL]

# Create xregs (this is the include the categorical variables instead of utilizing only the interaction of them)
xregs <- data[, .SD, .SDcols = c('Date', 'Store', 'Dept')]

# Change data types
data[, ':=' (Store = as.character(Store), Dept = as.character(Dept))]
xregs[, ':=' (Store = as.character(Store), Dept = as.character(Dept))]

 # Build forecast
XGBoostResults <- RemixAutoML::AutoXGBoostCARMA(

  # Data Artifacts
  data = data,
  NonNegativePred = FALSE,
  RoundPreds = FALSE,
  TargetColumnName = 'Weekly_Sales',
  DateColumnName = 'Date',
  HierarchGroups = NULL,
  GroupVariables = c('Store','Dept'),
  TimeUnit = 'weeks',
  TimeGroups = c('weeks','months'),

  # Data Wrangling Features
  EncodingMethod = 'binary',
  ZeroPadSeries = NULL,
  DataTruncate = FALSE,
  SplitRatios = c(1 - 10 / 138, 10 / 138),
  PartitionType = 'timeseries',
  AnomalyDetection = NULL,

  # Productionize
  FC_Periods = 0,
  TrainOnFull = FALSE,
  NThreads = 8,
  Timer = TRUE,
  DebugMode = FALSE,
  SaveDataPath = NULL,
  PDFOutputPath = NULL,

  # Target Transformations
  TargetTransformation = TRUE,
  Methods = c('Asinh', 'Asin', 'Log', 'LogPlus1', 'Sqrt', 'Logit'),
  Difference = FALSE,

  # Features
  Lags = list('weeks' = seq(1L, 10L, 1L), 'months' = seq(1L, 5L, 1L)),
  MA_Periods = list('weeks' = seq(5L, 20L, 5L), 'months' = seq(2L, 10L, 2L)),
  SD_Periods = NULL,
  Skew_Periods = NULL,
  Kurt_Periods = NULL,
  Quantile_Periods = NULL,
  Quantiles_Selected = c('q5','q95'),
  XREGS = xregs,
  FourierTerms = 4,
  CalendarVariables = c('week', 'wom', 'month', 'quarter'),
  HolidayVariable = c('USPublicHolidays','EasterGroup', 'ChristmasGroup','OtherEcclesticalFeasts'),
  HolidayLookback = NULL,
  HolidayLags = 1,
  HolidayMovingAverages = 1:2,
  TimeTrendVariable = TRUE,

  # ML eval args
  TreeMethod = 'hist',
  EvalMetric = 'RMSE',
  LossFunction = 'reg:squarederror',

  # ML grid tuning
  GridTune = TRUE,
  ModelCount = 5,
  MaxRunsWithoutNewWinner = 20L,
  MaxRunMinutes = 24L*60L,

  # ML args
  NTrees = seq(200L, 300L, 10L),
  LearningRate = c(0.01,0.02,0.03,0.04),
  MaxDepth = seq(4L, 10L, 1L),
  MinChildWeight = seq(1.0, 10.0, 1.0),
  SubSample = seq(0.55, 1.0, 0.05),
  ColSampleByTree = seq(0.55, 1.0, 0.05))

[mbanco]

Hi Adrian, the script works fine, but I would like to know how to use the model parameters with better performance to rerun and generate a forecast.

The computer specs are: Core i7, RAM 16 Gb, Windows 10 64 Bits

[AdrianAntico]

@mbanco

To generate a forecast you just need to switch the parameter TrainOnFull = TRUE and switch GridTune = FALSE.

There are quite a few ways to optimize a forecast with the CARMA models. A few examples are below. You have have a holdout period of data that you can compare the forecast against and update the model info based on the performance of that.

Optimize the ML args
# Loss Function
LossFunction = 'reg:squarederror' can be changed for something else
NTrees = seq(200L, 300L, 10L),
LearningRate = c(0.01,0.02,0.03,0.04),
MaxDepth = seq(4L, 10L, 1L),
MinChildWeight = seq(1.0, 10.0, 1.0),
SubSample = seq(0.55, 1.0, 0.05),
ColSampleByTree = seq(0.55, 1.0, 0.05)

# Alter the variables in the model
TargetTransformation = TRUE,
Methods = c('Asinh', 'Asin', 'Log', 'LogPlus1', 'Sqrt', 'Logit'),
Difference = FALSE,

# Features
Lags = list('weeks' = seq(1L, 10L, 1L), 'months' = seq(1L, 5L, 1L)),
MA_Periods = list('weeks' = seq(5L, 20L, 5L), 'months' = seq(2L, 10L, 2L)),
SD_Periods = NULL,
Skew_Periods = NULL,
Kurt_Periods = NULL,
Quantile_Periods = NULL,
Quantiles_Selected = c('q5','q95'),
XREGS = xregs,
FourierTerms = 4,
CalendarVariables = c('week', 'wom', 'month', 'quarter'),
HolidayVariable = c('USPublicHolidays','EasterGroup', 'ChristmasGroup','OtherEcclesticalFeasts'),
HolidayLookback = NULL,
HolidayLags = 1,
HolidayMovingAverages = 1:2,
TimeTrendVariable = TRUE,