Issue with combine_A_and_B

[sinievanderben]

Hello!

I tried to combine A and B, having two separated sets including train,test and val. But I get this error:

If I try it with --no_multiprocessing it does not produce anything. I was wondering what is going on?

Thanks in advance!

[sinievanderben]

This is the full error

(imaging) C:\Users\sinie\Documents\Utrecht_Universiteit\Stage\pytorch-CycleGAN-and-pix2pix>python datasets/combine_A_and_B.py --fold_A A --fold_B B --fold_AB AB [fold_A] = A [fold_B] = B [fold_AB] = AB [num_imgs] = 1000000 [use_AB] = False [no_multiprocessing] = False split = test, use 22/22 images split = test, number of images = 22 split = train, use 176/176 images split = train, number of images = 176 split = val, use 22/22 images split = val, number of images = 22 [fold_A] = A [fold_B] = B [fold_AB] = AB [num_imgs] = 1000000 [use_AB] = False [no_multiprocessing] = False [fold_A] = A [fold_B] = B [fold_AB] = AB [num_imgs] = 1000000 [use_AB] = False [no_multiprocessing] = False Traceback (most recent call last): File "", line 1, in File "C:\Users\sinie\anaconda3\envs\imaging\lib\multiprocessing\spawn.py", line 116, in spawn_main exitcode = _main(fd, parent_sentinel) File "C:\Users\sinie\anaconda3\envs\imaging\lib\multiprocessing\spawn.py", line 125, in _main prepare(preparation_data) File "C:\Users\sinie\anaconda3\envs\imaging\lib\multiprocessing\spawn.py", line 236, in prepare _fixup_main_from_path(data['init_main_from_path']) File "C:\Users\sinie\anaconda3\envs\imaging\lib\multiprocessing\spawn.py", line 287, in _fixup_main_from_path main_content = runpy.run_path(main_path, File "C:\Users\sinie\anaconda3\envs\imaging\lib\runpy.py", line 265, in run_path Traceback (most recent call last): return _run_module_code(code, init_globals, run_name, File "C:\Users\sinie\anaconda3\envs\imaging\lib\runpy.py", line 97, in _run_module_code File "", line 1, in _run_code(code, mod_globals, init_globals, File "C:\Users\sinie\anaconda3\envs\imaging\lib\runpy.py", line 87, in _run_code File "C:\Users\sinie\anaconda3\envs\imaging\lib\multiprocessing\spawn.py", line 116, in spawn_main exitcode = _main(fd, parent_sentinel) File "C:\Users\sinie\anaconda3\envs\imaging\lib\multiprocessing\spawn.py", line 125, in _main prepare(preparation_data) File "C:\Users\sinie\anaconda3\envs\imaging\lib\multiprocessing\spawn.py", line 236, in prepare [fold_A] = A [fold_B] = B [fold_AB] = AB [num_imgs] = 1000000 [use_AB] = False [no_multiprocessing] = False _fixup_main_from_path(data['init_main_from_path']) File "C:\Users\sinie\anaconda3\envs\imaging\lib\multiprocessing\spawn.py", line 287, in _fixup_main_from_path main_content = runpy.run_path(main_path, File "C:\Users\sinie\anaconda3\envs\imaging\lib\runpy.py", line 265, in run_path return _run_module_code(code, init_globals, run_name, File "C:\Users\sinie\anaconda3\envs\imaging\lib\runpy.py", line 97, in _run_module_code _run_code(code, mod_globals, init_globals, File "C:\Users\sinie\anaconda3\envs\imaging\lib\runpy.py", line 87, in _run_code exec(code, run_globals) File "C:\Users\sinie\Documents\Utrecht_Universiteit\Stage\pytorch-CycleGAN-and-pix2pix\datasets\combine_A_and_B.py", line 30, in exec(code, run_globals) File "C:\Users\sinie\Documents\Utrecht_Universiteit\Stage\pytorch-CycleGAN-and-pix2pix\datasets\combine_A_and_B.py", line 30, in pool=Pool() File "C:\Users\sinie\anaconda3\envs\imaging\lib\multiprocessing\context.py", line 119, in Pool pool=Pool() File "C:\Users\sinie\anaconda3\envs\imaging\lib\multiprocessing\context.py", line 119, in Pool [fold_A] = A return Pool(processes, initializer, initargs, maxtasksperchild, File "C:\Users\sinie\anaconda3\envs\imaging\lib\multiprocessing\pool.py", line 212, in init [fold_B] = B self._repopulate_pool() [fold_AB] = AB return Pool(processes, initializer, initargs, maxtasksperchild, File "C:\Users\sinie\anaconda3\envs\imaging\lib\multiprocessing\pool.py", line 212, in init self._repopulate_pool() File "C:\Users\sinie\anaconda3\envs\imaging\lib\multiprocessing\pool.py", line 303, in _repopulate_pool return self._repopulate_pool_static(self._ctx, self.Process, File "C:\Users\sinie\anaconda3\envs\imaging\lib\multiprocessing\pool.py", line 326, in _repopulate_pool_static [num_imgs] = 1000000 [use_AB] = False w.start() File "C:\Users\sinie\anaconda3\envs\imaging\lib\multiprocessing\process.py", line 121, in start [no_multiprocessing] = False Traceback (most recent call last): File "", line 1, in File "C:\Users\sinie\anaconda3\envs\imaging\lib\multiprocessing\pool.py", line 303, in _repopulate_pool self._popen = self._Popen(self) File "C:\Users\sinie\anaconda3\envs\imaging\lib\multiprocessing\context.py", line 327, in _Popen return Popen(process_obj) File "C:\Users\sinie\anaconda3\envs\imaging\lib\multiprocessing\popen_spawn_win32.py", line 45, in init prep_data = spawn.get_preparation_data(process_obj._name) File "C:\Users\sinie\anaconda3\envs\imaging\lib\multiprocessing\spawn.py", line 154, in get_preparation_data File "C:\Users\sinie\anaconda3\envs\imaging\lib\multiprocessing\spawn.py", line 116, in spawn_main _check_not_importing_main() File "C:\Users\sinie\anaconda3\envs\imaging\lib\multiprocessing\spawn.py", line 134, in _check_not_importing_main return self._repopulate_pool_static(self._ctx, self.Process, File "C:\Users\sinie\anaconda3\envs\imaging\lib\multiprocessing\pool.py", line 326, in _repopulate_pool_static raise RuntimeError(''' exitcode = _main(fd, parent_sentinel) File "C:\Users\sinie\anaconda3\envs\imaging\lib\multiprocessing\spawn.py", line 125, in _main prepare(preparation_data) File "C:\Users\sinie\anaconda3\envs\imaging\lib\multiprocessing\spawn.py", line 236, in prepare _fixup_main_from_path(data['init_main_from_path']) File "C:\Users\sinie\anaconda3\envs\imaging\lib\multiprocessing\spawn.py", line 287, in _fixup_main_from_path main_content = runpy.run_path(main_path, File "C:\Users\sinie\anaconda3\envs\imaging\lib\runpy.py", line 265, in run_path RuntimeError: An attempt has been made to start a new process before the current process has finished its bootstrapping phase.

    This probably means that you are not using fork to start your
    child processes and you have forgotten to use the proper idiom
    in the main module:

        if __name__ == '__main__':
            freeze_support()
            ...

    The "freeze_support()" line can be omitted if the program
    is not going to be frozen to produce an executable.
return _run_module_code(code, init_globals, run_name,

File "C:\Users\sinie\anaconda3\envs\imaging\lib\runpy.py", line 97, in _run_module_code _run_code(code, mod_globals, init_globals, File "C:\Users\sinie\anaconda3\envs\imaging\lib\runpy.py", line 87, in _run_code exec(code, run_globals) File "C:\Users\sinie\Documents\Utrecht_Universiteit\Stage\pytorch-CycleGAN-and-pix2pix\datasets\combine_A_and_B.py", line 30, in pool=Pool() File "C:\Users\sinie\anaconda3\envs\imaging\lib\multiprocessing\context.py", line 119, in Pool return Pool(processes, initializer, initargs, maxtasksperchild, Traceback (most recent call last): File "", line 1, in File "C:\Users\sinie\anaconda3\envs\imaging\lib\multiprocessing\spawn.py", line 116, in spawn_main exitcode = _main(fd, parent_sentinel) File "C:\Users\sinie\anaconda3\envs\imaging\lib\multiprocessing\spawn.py", line 125, in _main w.start() File "C:\Users\sinie\anaconda3\envs\imaging\lib\multiprocessing\process.py", line 121, in start prepare(preparation_data) File "C:\Users\sinie\anaconda3\envs\imaging\lib\multiprocessing\spawn.py", line 236, in prepare self._popen = self._Popen(self) File "C:\Users\sinie\anaconda3\envs\imaging\lib\multiprocessing\context.py", line 327, in _Popen _fixup_main_from_path(data['init_main_from_path']) File "C:\Users\sinie\anaconda3\envs\imaging\lib\multiprocessing\spawn.py", line 287, in _fixup_main_from_path return Popen(process_obj) main_content = runpy.run_path(main_path, File "C:\Users\sinie\anaconda3\envs\imaging\lib\runpy.py", line 265, in run_path return _run_module_code(code, init_globals, run_name, File "C:\Users\sinie\anaconda3\envs\imaging\lib\runpy.py", line 97, in _run_module_code _run_code(code, mod_globals, init_globals, File "C:\Users\sinie\anaconda3\envs\imaging\lib\runpy.py", line 87, in _run_code File "C:\Users\sinie\anaconda3\envs\imaging\lib\multiprocessing\pool.py", line 212, in init File "C:\Users\sinie\anaconda3\envs\imaging\lib\multiprocessing\popen_spawn_win32.py", line 45, in init prep_data = spawn.get_preparation_data(process_obj._name) self._repopulate_pool() File "C:\Users\sinie\anaconda3\envs\imaging\lib\multiprocessing\pool.py", line 303, in _repopulate_pool File "C:\Users\sinie\anaconda3\envs\imaging\lib\multiprocessing\spawn.py", line 154, in get_preparation_data exec(code, run_globals) File "C:\Users\sinie\Documents\Utrecht_Universiteit\Stage\pytorch-CycleGAN-and-pix2pix\datasets\combine_A_and_B.py", line 30, in _check_not_importing_main() File "C:\Users\sinie\anaconda3\envs\imaging\lib\multiprocessing\spawn.py", line 134, in _check_not_importing_main pool=Pool() File "C:\Users\sinie\anaconda3\envs\imaging\lib\multiprocessing\context.py", line 119, in Pool return Pool(processes, initializer, initargs, maxtasksperchild, File "C:\Users\sinie\anaconda3\envs\imaging\lib\multiprocessing\pool.py", line 212, in init self._repopulate_pool() File "C:\Users\sinie\anaconda3\envs\imaging\lib\multiprocessing\pool.py", line 303, in _repopulate_pool return self._repopulate_pool_static(self._ctx, self.Process, File "C:\Users\sinie\anaconda3\envs\imaging\lib\multiprocessing\pool.py", line 326, in _repopulate_pool_static raise RuntimeError(''' RuntimeError: An attempt has been made to start a new process before the current process has finished its bootstrapping phase.

    This probably means that you are not using fork to start your
    child processes and you have forgotten to use the proper idiom
    in the main module:

        if __name__ == '__main__':
            freeze_support()
            ...

    The "freeze_support()" line can be omitted if the program
    is not going to be frozen to produce an executable.
return self._repopulate_pool_static(self._ctx, self.Process,

File "C:\Users\sinie\anaconda3\envs\imaging\lib\multiprocessing\pool.py", line 326, in _repopulate_pool_static w.start() File "C:\Users\sinie\anaconda3\envs\imaging\lib\multiprocessing\process.py", line 121, in start self._popen = self._Popen(self) File "C:\Users\sinie\anaconda3\envs\imaging\lib\multiprocessing\context.py", line 327, in _Popen return Popen(process_obj) File "C:\Users\sinie\anaconda3\envs\imaging\lib\multiprocessing\popen_spawn_win32.py", line 45, in init w.start() File "C:\Users\sinie\anaconda3\envs\imaging\lib\multiprocessing\process.py", line 121, in start prep_data = spawn.get_preparation_data(process_obj._name) File "C:\Users\sinie\anaconda3\envs\imaging\lib\multiprocessing\spawn.py", line 154, in get_preparation_data self._popen = self._Popen(self) File "C:\Users\sinie\anaconda3\envs\imaging\lib\multiprocessing\context.py", line 327, in _Popen _check_not_importing_main() File "C:\Users\sinie\anaconda3\envs\imaging\lib\multiprocessing\spawn.py", line 134, in _check_not_importing_main return Popen(process_obj) File "C:\Users\sinie\anaconda3\envs\imaging\lib\multiprocessing\popen_spawn_win32.py", line 45, in init raise RuntimeError(''' RuntimeError: An attempt has been made to start a new process before the current process has finished its bootstrapping phase.

    This probably means that you are not using fork to start your
    child processes and you have forgotten to use the proper idiom
    in the main module:

        if __name__ == '__main__':
            freeze_support()
            ...

    The "freeze_support()" line can be omitted if the program
    is not going to be frozen to produce an executable.
prep_data = spawn.get_preparation_data(process_obj._name)

File "C:\Users\sinie\anaconda3\envs\imaging\lib\multiprocessing\spawn.py", line 154, in get_preparation_data _check_not_importing_main() File "C:\Users\sinie\anaconda3\envs\imaging\lib\multiprocessing\spawn.py", line 134, in _check_not_importing_main raise RuntimeError(''' RuntimeError: An attempt has been made to start a new process before the current process has finished its bootstrapping phase.

    This probably means that you are not using fork to start your
    child processes and you have forgotten to use the proper idiom
    in the main module:

        if __name__ == '__main__':
            freeze_support()
            ...

    The "freeze_support()" line can be omitted if the program
    is not going to be frozen to produce an executable.

[rickkk856]

I had a similar issue so I needed to look for another 'combine method' The code bellow worked for me

python combine_A_and_B.py --foldA path/to/fold/A --foldB path/to/fold/B --foldAB --foldA path/to/fold/AB

import os
import numpy as np
import cv2
import argparse

parser = argparse.ArgumentParser('create image pairs')
parser.add_argument('--fold_A', dest='fold_A', help='input directory for image A', type=str, default='../dataset/50kshoes_edges')
parser.add_argument('--fold_B', dest='fold_B', help='input directory for image B', type=str, default='../dataset/50kshoes_jpg')
parser.add_argument('--fold_AB', dest='fold_AB', help='output directory', type=str, default='../dataset/test_AB')
parser.add_argument('--num_imgs', dest='num_imgs', help='number of images', type=int, default=1000000)
parser.add_argument('--use_AB', dest='use_AB', help='if true: (0001_A, 0001_B) to (0001_AB)', action='store_true')
args = parser.parse_args()

for arg in vars(args):
    print('[%s] = ' % arg, getattr(args, arg))

splits = os.listdir(args.fold_A)

for sp in splits:
    img_fold_A = os.path.join(args.fold_A, sp)
    img_fold_B = os.path.join(args.fold_B, sp)
    img_list = os.listdir(img_fold_A)
    if args.use_AB:
        img_list = [img_path for img_path in img_list if '_A.' in img_path]

    num_imgs = min(args.num_imgs, len(img_list))
    print('split = %s, use %d/%d images' % (sp, num_imgs, len(img_list)))
    img_fold_AB = os.path.join(args.fold_AB, sp)
    if not os.path.isdir(img_fold_AB):
        os.makedirs(img_fold_AB)
    print('split = %s, number of images = %d' % (sp, num_imgs))
    for n in range(num_imgs):
        name_A = img_list[n]
        path_A = os.path.join(img_fold_A, name_A)
        if args.use_AB:
            name_B = name_A.replace('_A.', '_B.')
        else:
            name_B = name_A
        path_B = os.path.join(img_fold_B, name_B)
        if os.path.isfile(path_A) and os.path.isfile(path_B):
            name_AB = name_A
            if args.use_AB:
                name_AB = name_AB.replace('_A.', '.')  # remove _A
            path_AB = os.path.join(img_fold_AB, name_AB)

            im_A = cv2.imread(path_A, -1) # python2: cv2.CV_LOAD_IMAGE_COLOR; python3: cv2.IMREAD_COLOR
            im_B = cv2.imread(path_B, -1) # python2: cv2.CV_LOAD_IMAGE_COLOR; python3: cv2.IMREAD_COLOR

            #im_A = cv2.imread(path_A, 1) # python2: cv2.CV_LOAD_IMAGE_COLOR; python3: cv2.IMREAD_COLOR
            #im_B = cv2.imread(path_B, 1) # python2: cv2.CV_LOAD_IMAGE_COLOR; python3: cv2.IMREAD_COLOR
            im_AB = np.concatenate([im_A, im_B], 1)
            cv2.imwrite(path_AB, im_AB)

Remember that the data folder structure needs to be:

└── datasets
    └── experiment_name
       ├── foldA
       │    └──train
       │    └──test
       │    └──eval(optional)
       ├── foldB
       │    └──train
       │    └──test
       │    └──eval(optional)
       └──foldAB

After you run the code the train, test and eval folders will be created in foldAB, and the files will be combined there

[854324563y]

@rickkk856 Hello, it didn't work at first, I thought it was because of the route path/to/fold/A did not follow the structure datasets/experiment_name/foldA. I changed the code to "python combine_A_and_B.py --foldA ./experiment_name/foldA --foldB ./experiment_name/foldB --foldAB --foldA ./experiment_name/foldAB" but it turned out I was wrong. Could you please help me work it out?

[854324563y]

@rickkk856 I rewrote a code of my own, and it worked. Thanks anyway!

[Bengejd]

Damn near blew my PC up trying to run combine_A_and_B due to this... ended up spawning 32 Python.exe processes even though PyCharm terminal was terminated, and they would just keep respawning in task manager when I manually ended the tasks until I eventually used a cmd prompt to kill them all.

tasklist | find /i "python.exe" && taskkill /im "python.exe" /F || echo process "python.exe" not running

[ridhachahed]

I had the same issue, just changed the multiprocessing flag to True

[HuachaoGao]

you try make the '--use_AB' is 'default= True', and make the '--no_multiprocessing' is 'default= True', and it worked.

[Dionysus061726]

This issue occurs when you try to run combine_A_and_B with multiprocessing. If there's no need for multiprocessing, change the default of multiprocessing flag to True:

parser.add_argument('--no_multiprocessing', dest='no_multiprocessing', help='If used, chooses single CPU execution instead of parallel execution', action='store_true',default=False)  

↑set the default to True

This may help you with the issue.