Paper review: All Models are Wrong, but Many are Useful: Learning a Variable’s Importance by Studying an Entire Class of Prediction Models Simultaneously

[youyinnn]

ieee: https://ieeexplore.ieee.org/document/9671521

---

Abstract

The server workloads of large-scale online video games are elastic and on-demand. The workload can range from tens to thousands of server instances in short periods. In fact, a cloud-based video game ecosystem can reach a workload of millions of players every week. Given such a large scale, even a small portion of over-provisioning leads to a significant amount of resource idling and a high cost of waste. It is essential to define an effective forecasting model on the game session workloads given a time span. The effectiveness shall be measured by metrics representing Service Level Objectives (SLOs). In this work, we analyze time series forecasting models using ARIMA, Prophet, and LSTM to predict the number of virtual machines in need of cloud resource monitoring data. In addition, we define service-level metrics for measuring effectiveness based on factors of over/under provision and ratio of resource waste. We analyze models with 16 fleets with an average of 2754 game servers over a four-month-long period of time in the production environment. We observe that our LSTM model is the most accurate in forecasting the demand of virtual machines in terms of RMSE and MAE. Further analysis using metrics of SLOs, we observe that the LSTM model leads to more cases of under-provisioning than ARIMA and Prophet do. The LSTM model forecasts the demand of virtual machines with less over-provision ratio than ARIMA and Prophet do for 14 out of 16 fleets. Using the LSTM model, we further evaluate the forecasting effect across different time spans of a single fleet and across multiple fleets within the same time span.

---

• 18-760.pdf

---

Catch-up knowledge

#### _iid_ Independent and identically distributed #### Standard Empirical Loss 从本质出发（非计算公式），如何理解机器学习中的损失函数、代价函数，经验损失函数，以及期望代价函数？ - Dennis LD 的回答 - 知乎 https://www.zhihu.com/question/344292001/answer/820991495

[youyinnn]

The method of estimating the MR requires loss function, thus requires the actual Y value to get the empirical loss.

It is not applicable to our project.