search

HHTseng / video-classification

Tutorial for video classification/ action recognition using 3D CNN/ CNN+RNN on UCF101
937 stars 215 forks source link

RuntimeError: freeze_support() on Windows #16

Open MubarkLa opened 4 years ago

MubarkLa commented 4 years ago

Hello, thanks a lot for your great job which is very excellent and inspires me a lot. Now I am trying to run your code UCF101_CRNN.py on a Windows PC but facing a Runtime problem. I tried weeks to fix this problem but still failed. So I am wondering if you could please kindly help me.

When I was simply running UCF101_CRNN.py, I got this error:

C:\Anaconda3\envs\pytorch1\python.exe D:/LSTM/study/video-classification-master/CRNN/UCF101_CRNN.py C:\Anaconda3\envs\pytorch1\lib\site-packages\sklearn\preprocessing_encoders.py:415: FutureWarning: The handling of integer data will change in version 0.22. Currently, the categories are determined based on the range [0, max(values)], while in the future they will be determined based on the unique values. If you want the future behaviour and silence this warning, you can specify "categories='auto'". In case you used a LabelEncoder before this OneHotEncoder to convert the categories to integers, then you can now use the OneHotEncoder directly. warnings.warn(msg, FutureWarning) C:\Anaconda3\envs\pytorch1\lib\site-packages\sklearn\preprocessing_encoders.py:415: FutureWarning: The handling of integer data will change in version 0.22. Currently, the categories are determined based on the range [0, max(values)], while in the future they will be determined based on the unique values. If you want the future behaviour and silence this warning, you can specify "categories='auto'". In case you used a LabelEncoder before this OneHotEncoder to convert the categories to integers, then you can now use the OneHotEncoder directly. warnings.warn(msg, FutureWarning) Traceback (most recent call last): File "", line 1, in File "C:\Anaconda3\envs\pytorch1\lib\multiprocessing\spawn.py", line 105, in spawn_main Traceback (most recent call last): File "D:/LSTM/study/video-classification-master/CRNN/UCF101_CRNN.py", line 212, in exitcode = _main(fd) File "C:\Anaconda3\envs\pytorch1\lib\multiprocessing\spawn.py", line 114, in _main train_losses, train_scores = train(log_interval, [cnn_encoder, rnn_decoder], device, train_loader, optimizer, epoch) File "D:/LSTM/study/video-classification-master/CRNN/UCF101_CRNN.py", line 56, in train prepare(preparation_data) File "C:\Anaconda3\envs\pytorch1\lib\multiprocessing\spawn.py", line 225, in prepare for batch_idx, (X, y) in enumerate(train_loader): File "C:\Anaconda3\envs\pytorch1\lib\site-packages\torch\utils\data\dataloader.py", line 278, in iter _fixup_main_from_path(data['init_main_from_path']) File "C:\Anaconda3\envs\pytorch1\lib\multiprocessing\spawn.py", line 277, in _fixup_main_from_path return _MultiProcessingDataLoaderIter(self) File "C:\Anaconda3\envs\pytorch1\lib\site-packages\torch\utils\data\dataloader.py", line 682, in init run_name="mp_main__") File "C:\Anaconda3\envs\pytorch1\lib\runpy.py", line 263, in run_path pkg_name=pkg_name, script_name=fname) File "C:\Anaconda3\envs\pytorch1\lib\runpy.py", line 96, in _run_module_code w.start() File "C:\Anaconda3\envs\pytorch1\lib\multiprocessing\process.py", line 105, in start mod_name, mod_spec, pkg_name, script_name) File "C:\Anaconda3\envs\pytorch1\lib\runpy.py", line 85, in _run_code exec(code, run_globals) File "D:\LSTM\study\video-classification-master\CRNN\UCF101_CRNN.py", line 212, in self._popen = self._Popen(self) File "C:\Anaconda3\envs\pytorch1\lib\multiprocessing\context.py", line 223, in _Popen train_losses, train_scores = train(log_interval, [cnn_encoder, rnn_decoder], device, train_loader, optimizer, epoch) File "D:\LSTM\study\video-classification-master\CRNN\UCF101_CRNN.py", line 56, in train return _default_context.get_context().Process._Popen(process_obj) File "C:\Anaconda3\envs\pytorch1\lib\multiprocessing\context.py", line 322, in _Popen for batch_idx, (X, y) in enumerate(train_loader): File "C:\Anaconda3\envs\pytorch1\lib\site-packages\torch\utils\data\dataloader.py", line 278, in iter return Popen(process_obj) File "C:\Anaconda3\envs\pytorch1\lib\multiprocessing\popen_spawn_win32.py", line 65, in init return _MultiProcessingDataLoaderIter(self) File "C:\Anaconda3\envs\pytorch1\lib\site-packages\torch\utils\data\dataloader.py", line 682, in init reduction.dump(process_obj, to_child) File "C:\Anaconda3\envs\pytorch1\lib\multiprocessing\reduction.py", line 60, in dump ForkingPickler(file, protocol).dump(obj) BrokenPipeError: [Errno 32] Broken pipe w.start() File "C:\Anaconda3\envs\pytorch1\lib\multiprocessing\process.py", line 105, in start self._popen = self._Popen(self) File "C:\Anaconda3\envs\pytorch1\lib\multiprocessing\context.py", line 223, in _Popen return _default_context.get_context().Process._Popen(process_obj) File "C:\Anaconda3\envs\pytorch1\lib\multiprocessing\context.py", line 322, in _Popen return Popen(process_obj) File "C:\Anaconda3\envs\pytorch1\lib\multiprocessing\popen_spawn_win32.py", line 33, in init__ prep_data = spawn.get_preparation_data(process_obj._name) File "C:\Anaconda3\envs\pytorch1\lib\multiprocessing\spawn.py", line 143, in get_preparation_data _check_not_importing_main() File "C:\Anaconda3\envs\pytorch1\lib\multiprocessing\spawn.py", line 136, in _check_not_importing_main is not going to be frozen to produce an executable.''') RuntimeError: An attempt has been made to start a new process before the current process has finished its bootstrapping phase.

    This probably means that you are not using fork to start your
    child processes and you have forgotten to use the proper idiom
    in the main module:

        if __name__ == '__main__':
            freeze_support()
            ...

    The "freeze_support()" line can be omitted if the program
    is not going to be frozen to produce an executable.

Process finished with exit code 1

I searched on the Internet and it seemed that the problem is caused by that Windows doesn't not support fork multiprocessing. Then according to https://github.com/pytorch/pytorch/issues/5858, I added these codes at the beginning of UCF101_CRNN.py:

def run(): torch.multiprocessing.freeze_support() print('loop')

if name == 'main': run()

And the modified code becomes into:

import os import numpy as np import torch import torch.nn as nn import torch.nn.functional as F import torchvision.models as models import torchvision.transforms as transforms import torch.utils.data as data import torchvision from torch.autograd import Variable import matplotlib.pyplot as plt from functions import * from sklearn.model_selection import train_test_split from sklearn.preprocessing import OneHotEncoder, LabelEncoder from sklearn.metrics import accuracy_score import pickle

def run(): torch.multiprocessing.freeze_support() print('loop')

if name == 'main': run()

data_path = "D:\ucf\jpegs_256" # define UCF-101 RGB data path action_name_path = './UCF101actions.pkl' save_model_path = "./CRNN_ckpt/"

CNN_fc_hidden1, CNN_fc_hidden2 = 1024, 768 CNN_embed_dim = 512 # latent dim extracted by 2D CNN img_x, img_y = 256, 342 # resize video 2d frame size dropout_p = 0.0 # dropout probability

RNN_hidden_layers = 3 RNN_hidden_nodes = 512 RNN_FC_dim = 256

... all other lines ...

But I still have the same problem:

C:\Anaconda3\envs\pytorch1\python.exe D:/LSTM/study/video-classification-master/CRNN/UCF101_CRNN.py loop C:\Anaconda3\envs\pytorch1\lib\site-packages\sklearn\preprocessing_encoders.py:415: FutureWarning: The handling of integer data will change in version 0.22. Currently, the categories are determined based on the range [0, max(values)], while in the future they will be determined based on the unique values. If you want the future behaviour and silence this warning, you can specify "categories='auto'". In case you used a LabelEncoder before this OneHotEncoder to convert the categories to integers, then you can now use the OneHotEncoder directly. warnings.warn(msg, FutureWarning) C:\Anaconda3\envs\pytorch1\lib\site-packages\sklearn\preprocessing_encoders.py:415: FutureWarning: The handling of integer data will change in version 0.22. Currently, the categories are determined based on the range [0, max(values)], while in the future they will be determined based on the unique values. If you want the future behaviour and silence this warning, you can specify "categories='auto'". In case you used a LabelEncoder before this OneHotEncoder to convert the categories to integers, then you can now use the OneHotEncoder directly. warnings.warn(msg, FutureWarning) Traceback (most recent call last): File "", line 1, in File "C:\Anaconda3\envs\pytorch1\lib\multiprocessing\spawn.py", line 105, in spawn_main Traceback (most recent call last): File "D:/LSTM/study/video-classification-master/CRNN/UCF101_CRNN.py", line 218, in exitcode = _main(fd) File "C:\Anaconda3\envs\pytorch1\lib\multiprocessing\spawn.py", line 114, in _main train_losses, train_scores = train(log_interval, [cnn_encoder, rnn_decoder], device, train_loader, optimizer, epoch) File "D:/LSTM/study/video-classification-master/CRNN/UCF101_CRNN.py", line 62, in train prepare(preparation_data) File "C:\Anaconda3\envs\pytorch1\lib\multiprocessing\spawn.py", line 225, in prepare for batch_idx, (X, y) in enumerate(train_loader): File "C:\Anaconda3\envs\pytorch1\lib\site-packages\torch\utils\data\dataloader.py", line 278, in iter _fixup_main_from_path(data['init_main_from_path']) File "C:\Anaconda3\envs\pytorch1\lib\multiprocessing\spawn.py", line 277, in _fixup_main_from_path return _MultiProcessingDataLoaderIter(self) File "C:\Anaconda3\envs\pytorch1\lib\site-packages\torch\utils\data\dataloader.py", line 682, in init run_name="mp_main__") File "C:\Anaconda3\envs\pytorch1\lib\runpy.py", line 263, in run_path pkg_name=pkg_name, script_name=fname) File "C:\Anaconda3\envs\pytorch1\lib\runpy.py", line 96, in _run_module_code w.start() File "C:\Anaconda3\envs\pytorch1\lib\multiprocessing\process.py", line 105, in start mod_name, mod_spec, pkg_name, script_name) File "C:\Anaconda3\envs\pytorch1\lib\runpy.py", line 85, in _run_code self._popen = self._Popen(self) File "C:\Anaconda3\envs\pytorch1\lib\multiprocessing\context.py", line 223, in _Popen exec(code, run_globals) File "D:\LSTM\study\video-classification-master\CRNN\UCF101_CRNN.py", line 218, in train_losses, train_scores = train(log_interval, [cnn_encoder, rnn_decoder], device, train_loader, optimizer, epoch) return _default_context.get_context().Process._Popen(process_obj) File "D:\LSTM\study\video-classification-master\CRNN\UCF101_CRNN.py", line 62, in train File "C:\Anaconda3\envs\pytorch1\lib\multiprocessing\context.py", line 322, in _Popen for batch_idx, (X, y) in enumerate(train_loader): File "C:\Anaconda3\envs\pytorch1\lib\site-packages\torch\utils\data\dataloader.py", line 278, in iter return Popen(process_obj) File "C:\Anaconda3\envs\pytorch1\lib\multiprocessing\popen_spawn_win32.py", line 65, in init reduction.dump(process_obj, to_child) File "C:\Anaconda3\envs\pytorch1\lib\multiprocessing\reduction.py", line 60, in dump return _MultiProcessingDataLoaderIter(self) File "C:\Anaconda3\envs\pytorch1\lib\site-packages\torch\utils\data\dataloader.py", line 682, in init ForkingPickler(file, protocol).dump(obj) BrokenPipeError: [Errno 32] Broken pipe w.start() File "C:\Anaconda3\envs\pytorch1\lib\multiprocessing\process.py", line 105, in start self._popen = self._Popen(self) File "C:\Anaconda3\envs\pytorch1\lib\multiprocessing\context.py", line 223, in _Popen return _default_context.get_context().Process._Popen(process_obj) File "C:\Anaconda3\envs\pytorch1\lib\multiprocessing\context.py", line 322, in _Popen return Popen(process_obj) File "C:\Anaconda3\envs\pytorch1\lib\multiprocessing\popen_spawn_win32.py", line 33, in init__ prep_data = spawn.get_preparation_data(process_obj._name) File "C:\Anaconda3\envs\pytorch1\lib\multiprocessing\spawn.py", line 143, in get_preparation_data _check_not_importing_main() File "C:\Anaconda3\envs\pytorch1\lib\multiprocessing\spawn.py", line 136, in _check_not_importing_main is not going to be frozen to produce an executable.''') RuntimeError: An attempt has been made to start a new process before the current process has finished its bootstrapping phase.

    This probably means that you are not using fork to start your
    child processes and you have forgotten to use the proper idiom
    in the main module:

        if __name__ == '__main__':
            freeze_support()
            ...

    The "freeze_support()" line can be omitted if the program
    is not going to be frozen to produce an executable.

Process finished with exit code 1

Could you please kindly help me fix this problem?

HHTseng commented 4 years ago

Hi, thank you for reporting a problem running on Windows. Long time ago I seemed to notice there's a difference running Pytorch codes on Windows & Linux. Since now I no longer have a Windows machine, I am afraid that I am not able to reproduce your problem.

May I suggest that you remove (or comment out) the part parallelizing to multiple GPUs? eg. in CRNN/UCF101_CRNN.py, delete all lines between 192 and 195

mampferd commented 4 years ago

Hi, I got the same problem but just in the ResNetCRNN_check_prediction.py file. BUT with the help of a friend I found the solution on github after long time searching the internet:

Windows doesn't use fork, but spawn which needs the code to be changed a bit to not run the code multiple times at once.

After all import commands you need to define the whole code as the main() function _... from functions import *

def main():

set path

data_path = "C:/Users/erik.ackermann/LSTM_Data/jpegs_256"  # define UCF-101 RGB data path
action_name_path = "./UCF101actions.pkl"

... print('video prediction finished!')

if name == 'main': main()_

More explanation can be found in the Windows FAQ to pytorch

The solution was reported by peterjc123 on github #7485

I hope that still helps somebody.