search

yfzhang114 / OneNet

This is an official PyTorch implementation of the NeurIPS 2023 paper 《OneNet: Enhancing Time Series Forecasting Models under Concept Drift by Online Ensembling》
91 stars 14 forks source link

Do you notice the phenomenon of memory leak in the code of FSNet #3

Open LiuYasuo opened 6 months ago

LiuYasuo commented 6 months ago

I've fully experimented with their code taking up CPU memory issues. It is evident that the CPU memory decreases gradually when the program is running. Fortunately, their process will not be killed because the datasets in their experiment are too small. However, I tested their code on a dataset with 100,000 entries. And the process was killed when we got to one tenth of the way through the test because of memory leak. Long data series are not uncommon in the real world of online learning, and online learning is geared towards applications in the display world. Therefore, we are supposed to take this issue seriously. I have asked the author of the FSNet but got no reply. Considering the in-depth research you have conducted on time series online learning， I hope you could discuss this problem with me. Looking forward to your reply.

yfzhang114 commented 6 months ago

Regarding your concern about the dataset size and memory management, we acknowledge that larger datasets can pose significant challenges in terms of memory utilization. While we have successfully tested our code on datasets with high numbers of time steps and channels, such as the ECL dataset with over 100k time steps and 300+ channels, we recognize that each dataset may have unique characteristics that could affect memory usage differently.

In response to your query about the specific details of the dataset you used, such as the number of channels and the meaning of "100,000 entries," we would appreciate more information to better understand the context of your testing. This will allow us to provide more targeted suggestions for optimizing memory usage and addressing potential memory leaks.

Here are some advice to process large datasets and avoid CPU memory issues:

  1. Batch Processing: Instead of loading the entire dataset into memory at once, process the data in smaller batches. This helps in reducing memory usage and prevents overwhelming the CPU.
  2. Data Compression: If applicable, consider compressing the dataset before loading it into memory. This can significantly reduce memory usage while still allowing for efficient processing.
  3. Data Preprocessing: Prior to loading the data into memory, perform preprocessing steps such as feature selection, normalization, or downsampling to reduce the size of the dataset without losing important information.
  4. Resource Monitoring: Continuously monitor CPU and memory usage during data processing to identify any abnormal spikes or patterns. This can help in detecting memory issues early on and taking appropriate measures to address them.

note that 4 is important, we need to know where the error occurred.

LiuYasuo commented 6 months ago

Thanks for your reply. I realized I hadn't made the point clear. "100,000 entries" means our dataset has over 100k time steps. What's more, in industrial time series forecasting scenarios, we need to make multi-step iterative predictions, which would magnify the problems mentioned above indefinitely and the process would be killed eventually. As a matter of fact, if you have the time, I hope you could run the original code of FSnet again. And you can use the "top" command to check the memory usage of this process while it is running. Then you will find that the memory occupied by this process continues to increase. Looking forward to your early reply.

yfzhang114 commented 6 months ago

It is unreasonable, the ECL dataset also contains more than 100,000 time steps and a large number of channels, works well with the model.