search

j-bac / elpigraph-python

Python implementation of the ElPiGraph algorithm with cpu and gpu support
https://elpigraph-python.readthedocs.io
GNU General Public License v3.0
15 stars 10 forks source link

compute principal graph calculation struck at node 43 while using multiple cores(n_cores>1) #5

Open kunamneni117 opened 4 years ago

kunamneni117 commented 4 years ago

Hi, While using python implementation of elpigraph leveraging n_cores>1,the computeprincipalgraph calculation struck at Node 43 .Below attached the code output for reference.

Computing EPG with 60 nodes on 199 points and 3 dimensions Creating a Pool with 2 processes Nodes = 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 I have used the tree data which is in the same repository for doing sanity checks.

In the library I have removed the commentated part multiprocessing library for using multi cores.

j-bac commented 4 years ago

Thanks for the report! Using the example tree data, which has 492 points, I find no problem when uncommenting import multiprocessing as mp in BaseElpi.py and grammar_operations.py. You seem to have used a different dataset with 199 points and I can't reproduce your issue.

I disabled the multi-cpu option for now because it created some issues when used in clusters. If you wish to try it yourself, to see any speedup you need a dataset with ~10**4 points minimum. A better option to speed up your code is to try the GPU version (import elpigraphgpu) using your own GPU or a free one on google Colaboratory.

kunamneni117 commented 4 years ago

Thanks for the prompt response.I have tried again with uncommenting the import multiprocessing as mp in BaseElpi.py and grammar_operations.py. I have used the tree data set. Earlier i have used circle data with header on.(199 data points in the earlier comment) Generating the initial configuration Creating a chain in the 1st PC with 2 nodes 90% of the points have been used as initial conditions. Resetting. Constructing tree 1 of 1 / Subset 1 of 1 Performing PCA Using standard PCA 3 dimensions are being used 100.0 % of the original variance has been retained The elastic matrix is being used. Edge configuration will be ignored Computing EPG with 60 nodes on 492 points and 3 dimensions Creating a Pool with 4 processes Nodes = 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43

If you can provide some tips how to debug the issue i can provide more info about the issue.

I am doing checks on multi core approach b/w python and elpigraph implementation. I am going to use the above elpigraph implementation lot of data(more than 1 million).So i am trying to speed up the process with multi core approach and GPU approach.

The above issue(Multi core issue) can be due to some issue in environment setup? any chance?

j-bac commented 4 years ago

Yes, if you provide more info I am happy to help - however I am quite puzzled what could be causing this, on my laptop with Ubuntu 18.04.4 LTS I have no issue.

In any case the multi-core version is still work in progress and I strongly recommend you do not use it but instead try the GPU version. For 1 million+ datapoints it is the best solution. Using it requires the cupy package, after that using it is as easy as "import elpigraphgpu" instead of "import elpigraph".

Here are example timings: elpirun

kunamneni117 commented 4 years ago

I tried elpigraph in Ubuntu 18.04 LTS and surprisingly it is not struck at node 43 while doing principal graph computation as u called out.I tried with windows again and computation of principal graph struck a Node 43.I have used the tree data while checking. I am struck at how to debug the issue further so that i can solve/report the issue.If can u provide some info where to start for debugging the issue that may be useful for me to proceed further.

j-bac commented 4 years ago

I found this link which might be useful

kunamneni117 commented 4 years ago

I found the python implementation using multi cores works for small data sets without any issue but when using a medium data(50k data points) hangs while doing computation in Ubuntu 18.04 LTS OS.

Used Supermarket data for testing where used the Order.ID,Order.Date,Ship.Date,Ship.Mode,Customer.ID,Customer.Name,Segment,City,State,Country,Market,Region,Product.ID,Category,Sub.Category,Product.Name,Sales,Quantity,Discount,Profit,Shipping.Cost,Order.Priority columns where some of the columns are Label Encoded.

I have forked your R implementation of elpigraph and its working fine with the above mentioned data in single core and multi core approach.

j-bac commented 4 years ago

I hope to set up a new version for multi-core Python soon and will look into this if I access a windows machine. In the meantime if you wish to use a multi-core and not GPU implementation it is indeed best to use the R version. Glad to hear it works!

kunamneni117 commented 4 years ago

I used tree data for checking multi core issue,got some interesting bugs..At Node 43 it is calling again the total code. USED WINDOWS MACHINE Sample Code

import warnings warnings.filterwarnings('ignore') import os import pandas as pd from elpigraph import computeElasticPrincipalCurve file_location=r'C:\Users\tonystark\Documents\folder\elpigraph\testing\data' file_lists=[] for r,d,f in os.walk(file_location): for file in f: if '.csv' in file: file_lists.append(os.path.join(r, file)) file_name=file_lists[2] print("Generating ElPiGraph output for file -",file_name + os.linesep) first_output = pd.read_csv(file_name,header=None,na_values=["NA","na","NULL","null",""," "]) print("Computing Elastic Principal Curve for file-",file_name + os.linesep) computeElasticPrincipalCurve(first_output.astype('float64').values, 60,n_cores=4)

C:\Users\tonystark\AppData\Local\conda\conda\envs\myenv\python.exe "C:/Users/tonystark/Documents/folder/elpigraph/testing/mutliprocessing check.py" Generating ElPiGraph output for file - C:\Users\tonystark\Documents\folder\elpigraph\testing\data\tree_data.csv

Computing Elastic Principal Curve for file- C:\Users\tonystark\Documents\folder\elpigraph\testing\data\tree_data.csv

Generating the initial configuration Creating a chain in the 1st PC with 2 nodes 90% of the points have been used as initial conditions. Resetting. Constructing tree 1 of 1 / Subset 1 of 1 Performing PCA Using standard PCA 3 dimensions are being used 100.0 % of the original variance has been retained The elastic matrix is being used. Edge configuration will be ignored Computing EPG with 60 nodes on 492 points and 3 dimensions Creating a Pool with 4 processes Nodes = 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 Generating ElPiGraph output for file - C:\Users\tonystark\Documents\folder\elpigraph\testing\data\tree_data.csv

Computing Elastic Principal Curve for file- C:\Users\tonystark\Documents\folder\elpigraph\testing\data\tree_data.csv

Generating the initial configuration Creating a chain in the 1st PC with 2 nodes Generating ElPiGraph output for file - C:\Users\tonystark\Documents\folder\elpigraph\testing\data\tree_data.csv

Computing Elastic Principal Curve for file- C:\Users\tonystark\Documents\folder\elpigraph\testing\data\tree_data.csv

Generating the initial configuration Creating a chain in the 1st PC with 2 nodes Generating ElPiGraph output for file - C:\Users\tonystark\Documents\folder\elpigraph\testing\data\tree_data.csv

Computing Elastic Principal Curve for file- C:\Users\tonystark\Documents\folder\elpigraph\testing\data\tree_data.csv

Generating the initial configuration Creating a chain in the 1st PC with 2 nodes Generating ElPiGraph output for file - C:\Users\tonystark\Documents\folder\elpigraph\testing\data\tree_data.csv

Computing Elastic Principal Curve for file- C:\Users\tonystark\Documents\folder\elpigraph\testing\data\tree_data.csv

Generating the initial configuration Creating a chain in the 1st PC with 2 nodes Traceback (most recent call last): File "", line 1, in File "C:\Users\tonystark\AppData\Local\conda\conda\envs\myenv\lib\multiprocessing\spawn.py", line 105, in spawn_main exitcode = _main(fd) File "C:\Users\tonystark\AppData\Local\conda\conda\envs\myenv\lib\multiprocessing\spawn.py", line 114, in _main prepare(preparation_data) File "C:\Users\tonystark\AppData\Local\conda\conda\envs\myenv\lib\multiprocessing\spawn.py", line 225, in prepare _fixup_main_from_path(data['init_main_from_path']) File "C:\Users\tonystark\AppData\Local\conda\conda\envs\myenv\lib\multiprocessing\spawn.py", line 277, in _fixup_main_from_path run_name="mp_main__") File "C:\Users\tonystark\AppData\Local\conda\conda\envs\myenv\lib\runpy.py", line 263, in run_path Traceback (most recent call last): File "", line 1, in pkg_name=pkg_name, script_name=fname) File "C:\Users\tonystark\AppData\Local\conda\conda\envs\myenv\lib\runpy.py", line 96, in _run_module_code mod_name, mod_spec, pkg_name, script_name) File "C:\Users\tonystark\AppData\Local\conda\conda\envs\myenv\lib\runpy.py", line 85, in _run_code exec(code, run_globals) File "C:\Users\tonystark\Documents\folder\elpigraph\testing\mutliprocessing check.py", line 24, in computeElasticPrincipalCurve(first_output.astype('float64').values, 60,n_cores=4)
File "C:\Users\tonystark\AppData\Local\conda\conda\envs\myenv\lib\site-packages\elpigraph_topologies.py", line 597, in computeElasticPrincipalCurve MaxSteps = MaxSteps File "C:\Users\tonystark\AppData\Local\conda\conda\envs\myenv\lib\site-packages\elpigraph_BaseElPiWrapper.py", line 284, in computeElasticPrincipalGraphWithGrammars DisplayWarnings=DisplayWarnings File "C:\Users\tonystark\AppData\Local\conda\conda\envs\myenv\lib\site-packages\elpigraph\src\BaseElPi.py", line 585, in computeElasticPrincipalGraph n_cores=n_cores,MinParOp=MinParOp) File "C:\Users\tonystark\AppData\Local\conda\conda\envs\myenv\lib\site-packages\elpigraph\src\BaseElPi.py", line 184, in ElPrincGraph pool = mp.Pool(n_cores) File "C:\Users\tonystark\AppData\Local\conda\conda\envs\myenv\lib\multiprocessing\context.py", line 119, in Pool Traceback (most recent call last): File "", line 1, in context=self.get_context()) File "C:\Users\tonystark\AppData\Local\conda\conda\envs\myenv\lib\multiprocessing\pool.py", line 177, in init File "C:\Users\tonystark\AppData\Local\conda\conda\envs\myenv\lib\multiprocessing\spawn.py", line 105, in spawn_main self._repopulate_pool() File "C:\Users\tonystark\AppData\Local\conda\conda\envs\myenv\lib\multiprocessing\pool.py", line 238, in _repopulate_pool self._wrap_exception) File "C:\Users\tonystark\AppData\Local\conda\conda\envs\myenv\lib\multiprocessing\pool.py", line 257, in _repopulate_pool_static w.start() File "C:\Users\tonystark\AppData\Local\conda\conda\envs\myenv\lib\multiprocessing\process.py", line 112, in start exitcode = _main(fd) File "C:\Users\tonystark\AppData\Local\conda\conda\envs\myenv\lib\multiprocessing\spawn.py", line 114, in _main prepare(preparation_data) File "C:\Users\tonystark\AppData\Local\conda\conda\envs\myenv\lib\multiprocessing\spawn.py", line 225, in prepare _fixup_main_from_path(data['init_main_from_path']) File "C:\Users\tonystark\AppData\Local\conda\conda\envs\myenv\lib\multiprocessing\spawn.py", line 277, in _fixup_main_from_path run_name="__mp_main") File "C:\Users\tonystark\AppData\Local\conda\conda\envs\myenv\lib\runpy.py", line 263, in run_path pkg_name=pkg_name, script_name=fname) File "C:\Users\tonystark\AppData\Local\conda\conda\envs\myenv\lib\runpy.py", line 96, in _run_module_code mod_name, mod_spec, pkg_name, script_name) File "C:\Users\tonystark\AppData\Local\conda\conda\envs\myenv\lib\runpy.py", line 85, in _run_code exec(code, run_globals) File "C:\Users\tonystark\Documents\folder\elpigraph\testing\mutliprocessing check.py", line 24, in self._popen = self._Popen(self) File "C:\Users\tonystark\AppData\Local\conda\conda\envs\myenv\lib\multiprocessing\context.py", line 322, in _Popen return Popen(process_obj) File "C:\Users\tonystark\AppData\Local\conda\conda\envs\myenv\lib\multiprocessing\popen_spawn_win32.py", line 33, in init computeElasticPrincipalCurve(first_output.astype('float64').values, 60,n_cores=4)
File "C:\Users\tonystark\AppData\Local\conda\conda\envs\myenv\lib\site-packages\elpigraph_topologies.py", line 597, in computeElasticPrincipalCurve MaxSteps = MaxSteps File "C:\Users\tonystark\AppData\Local\conda\conda\envs\myenv\lib\site-packages\elpigraph_BaseElPiWrapper.py", line 284, in computeElasticPrincipalGraphWithGrammars DisplayWarnings=DisplayWarnings File "C:\Users\tonystark\AppData\Local\conda\conda\envs\myenv\lib\site-packages\elpigraph\src\BaseElPi.py", line 585, in computeElasticPrincipalGraph prep_data = spawn.get_preparation_data(process_obj._name) File "C:\Users\tonystark\AppData\Local\conda\conda\envs\myenv\lib\multiprocessing\spawn.py", line 143, in get_preparation_data _check_not_importing_main() File "C:\Users\tonystark\AppData\Local\conda\conda\envs\myenv\lib\multiprocessing\spawn.py", line 136, in _check_not_importing_main is not going to be frozen to produce an executable.''') RuntimeError: An attempt has been made to start a new process before the current process has finished its bootstrapping phase.

    This probably means that you are not using fork to start your
    child processes and you have forgotten to use the proper idiom
    in the main module:

        if __name__ == '__main__':
            freeze_support()
            ...

    The "freeze_support()" line can be omitted if the program
    is not going to be frozen to produce an executable.
n_cores=n_cores,MinParOp=MinParOp)

File "C:\Users\tonystark\AppData\Local\conda\conda\envs\myenv\lib\site-packages\elpigraph\src\BaseElPi.py", line 184, in ElPrincGraph pool = mp.Pool(n_cores) File "C:\Users\tonystark\AppData\Local\conda\conda\envs\myenv\lib\multiprocessing\context.py", line 119, in Pool context=self.get_context()) File "C:\Users\tonystark\AppData\Local\conda\conda\envs\myenv\lib\multiprocessing\pool.py", line 177, in init self._repopulate_pool() File "C:\Users\tonystark\AppData\Local\conda\conda\envs\myenv\lib\multiprocessing\pool.py", line 238, in _repopulate_pool self._wrap_exception) File "C:\Users\tonystark\AppData\Local\conda\conda\envs\myenv\lib\multiprocessing\pool.py", line 257, in _repopulate_pool_static w.start() File "C:\Users\tonystark\AppData\Local\conda\conda\envs\myenv\lib\multiprocessing\process.py", line 112, in start self._popen = self._Popen(self) File "C:\Users\tonystark\AppData\Local\conda\conda\envs\myenv\lib\multiprocessing\context.py", line 322, in _Popen return Popen(process_obj) File "C:\Users\tonystark\AppData\Local\conda\conda\envs\myenv\lib\multiprocessing\popen_spawn_win32.py", line 33, in init File "C:\Users\tonystark\AppData\Local\conda\conda\envs\myenv\lib\multiprocessing\spawn.py", line 105, in spawn_main prep_data = spawn.get_preparation_data(process_obj._name) File "C:\Users\tonystark\AppData\Local\conda\conda\envs\myenv\lib\multiprocessing\spawn.py", line 143, in get_preparation_data _check_not_importing_main() File "C:\Users\tonystark\AppData\Local\conda\conda\envs\myenv\lib\multiprocessing\spawn.py", line 136, in _check_not_importing_main is not going to be frozen to produce an executable.''') RuntimeError: An attempt has been made to start a new process before the current process has finished its bootstrapping phase.

    This probably means that you are not using fork to start your
    child processes and you have forgotten to use the proper idiom
    in the main module:

        if __name__ == '__main__':
            freeze_support()
            ...

    The "freeze_support()" line can be omitted if the program
    is not going to be frozen to produce an executable.
exitcode = _main(fd)

File "C:\Users\tonystark\AppData\Local\conda\conda\envs\myenv\lib\multiprocessing\spawn.py", line 114, in _main prepare(preparation_data) File "C:\Users\tonystark\AppData\Local\conda\conda\envs\myenv\lib\multiprocessing\spawn.py", line 225, in prepare _fixup_main_from_path(data['init_main_from_path']) File "C:\Users\tonystark\AppData\Local\conda\conda\envs\myenv\lib\multiprocessing\spawn.py", line 277, in _fixup_main_from_path run_name="mp_main__") File "C:\Users\tonystark\AppData\Local\conda\conda\envs\myenv\lib\runpy.py", line 263, in run_path pkg_name=pkg_name, script_name=fname) File "C:\Users\tonystark\AppData\Local\conda\conda\envs\myenv\lib\runpy.py", line 96, in _run_module_code mod_name, mod_spec, pkg_name, script_name) File "C:\Users\tonystark\AppData\Local\conda\conda\envs\myenv\lib\runpy.py", line 85, in _run_code exec(code, run_globals) File "C:\Users\tonystark\Documents\folder\elpigraph\testing\mutliprocessing check.py", line 24, in computeElasticPrincipalCurve(first_output.astype('float64').values, 60,n_cores=4)
File "C:\Users\tonystark\AppData\Local\conda\conda\envs\myenv\lib\site-packages\elpigraph_topologies.py", line 597, in computeElasticPrincipalCurve MaxSteps = MaxSteps File "C:\Users\tonystark\AppData\Local\conda\conda\envs\myenv\lib\site-packages\elpigraph_BaseElPiWrapper.py", line 284, in computeElasticPrincipalGraphWithGrammars DisplayWarnings=DisplayWarnings File "C:\Users\tonystark\AppData\Local\conda\conda\envs\myenv\lib\site-packages\elpigraph\src\BaseElPi.py", line 585, in computeElasticPrincipalGraph n_cores=n_cores,MinParOp=MinParOp) File "C:\Users\tonystark\AppData\Local\conda\conda\envs\myenv\lib\site-packages\elpigraph\src\BaseElPi.py", line 184, in ElPrincGraph pool = mp.Pool(n_cores) File "C:\Users\tonystark\AppData\Local\conda\conda\envs\myenv\lib\multiprocessing\context.py", line 119, in Pool context=self.get_context()) File "C:\Users\tonystark\AppData\Local\conda\conda\envs\myenv\lib\multiprocessing\pool.py", line 177, in init__ self._repopulate_pool() File "C:\Users\tonystark\AppData\Local\conda\conda\envs\myenv\lib\multiprocessing\pool.py", line 238, in _repopulate_pool self._wrap_exception) File "C:\Users\tonystark\AppData\Local\conda\conda\envs\myenv\lib\multiprocessing\pool.py", line 257, in _repopulate_pool_static w.start() File "C:\Users\tonystark\AppData\Local\conda\conda\envs\myenv\lib\multiprocessing\process.py", line 112, in start self._popen = self._Popen(self) File "C:\Users\tonystark\AppData\Local\conda\conda\envs\myenv\lib\multiprocessing\context.py", line 322, in _Popen return Popen(process_obj) File "C:\Users\tonystark\AppData\Local\conda\conda\envs\myenv\lib\multiprocessing\popen_spawn_win32.py", line 33, in init prep_data = spawn.get_preparation_data(process_obj._name) File "C:\Users\tonystark\AppData\Local\conda\conda\envs\myenv\lib\multiprocessing\spawn.py", line 143, in get_preparation_data _check_not_importing_main() File "C:\Users\tonystark\AppData\Local\conda\conda\envs\myenv\lib\multiprocessing\spawn.py", line 136, in _check_not_importing_main is not going to be frozen to produce an executable.''') RuntimeError: An attempt has been made to start a new process before the current process has finished its bootstrapping phase.

    This probably means that you are not using fork to start your
    child processes and you have forgotten to use the proper idiom
    in the main module:

        if __name__ == '__main__':
            freeze_support()
            ...

    The "freeze_support()" line can be omitted if the program
    is not going to be frozen to produce an executable.

90% of the points have been used as initial conditions. Resetting. Constructing tree 1 of 1 / Subset 1 of 1 Performing PCA Using standard PCA 3 dimensions are being used 100.0 % of the original variance has been retained The elastic matrix is being used. Edge configuration will be ignored Computing EPG with 60 nodes on 492 points and 3 dimensions Creating a Pool with 4 processes 90% of the points have been used as initial conditions. Resetting. 90% of the points have been used as initial conditions. Resetting. Constructing tree 1 of 1 / Subset 1 of 1 Performing PCA Using standard PCA 3 dimensions are being used 100.0 % of the original variance has been retained The elastic matrix is being used. Edge configuration will be ignored Computing EPG with 60 nodes on 492 points and 3 dimensions Creating a Pool with 4 processes Constructing tree 1 of 1 / Subset 1 of 1 Performing PCA Using standard PCA 3 dimensions are being used 100.0 % of the original variance has been retained The elastic matrix is being used. Edge configuration will be ignored Computing EPG with 60 nodes on 492 points and 3 dimensions Creating a Pool with 4 processes 90% of the points have been used as initial conditions. Resetting. Constructing tree 1 of 1 / Subset 1 of 1 Performing PCA Using standard PCA 3 dimensions are being used 100.0 % of the original variance has been retained The elastic matrix is being used. Edge configuration will be ignored Computing EPG with 60 nodes on 492 points and 3 dimensions Creating a Pool with 4 processes Traceback (most recent call last): File "", line 1, in File "C:\Users\tonystark\AppData\Local\conda\conda\envs\myenv\lib\multiprocessing\spawn.py", line 105, in spawn_main exitcode = _main(fd) File "C:\Users\tonystark\AppData\Local\conda\conda\envs\myenv\lib\multiprocessing\spawn.py", line 114, in _main prepare(preparation_data) File "C:\Users\tonystark\AppData\Local\conda\conda\envs\myenv\lib\multiprocessing\spawn.py", line 225, in prepare _fixup_main_from_path(data['init_main_from_path']) File "C:\Users\tonystark\AppData\Local\conda\conda\envs\myenv\lib\multiprocessing\spawn.py", line 277, in _fixup_main_from_path run_name="mp_main__") File "C:\Users\tonystark\AppData\Local\conda\conda\envs\myenv\lib\runpy.py", line 263, in run_path pkg_name=pkg_name, script_name=fname) File "C:\Users\tonystark\AppData\Local\conda\conda\envs\myenv\lib\runpy.py", line 96, in _run_module_code mod_name, mod_spec, pkg_name, script_name) File "C:\Users\tonystark\AppData\Local\conda\conda\envs\myenv\lib\runpy.py", line 85, in _run_code exec(code, run_globals) File "C:\Users\tonystark\Documents\folder\elpigraph\testing\mutliprocessing check.py", line 24, in computeElasticPrincipalCurve(first_output.astype('float64').values, 60,n_cores=4)
File "C:\Users\tonystark\AppData\Local\conda\conda\envs\myenv\lib\site-packages\elpigraph_topologies.py", line 597, in computeElasticPrincipalCurve MaxSteps = MaxSteps File "C:\Users\tonystark\AppData\Local\conda\conda\envs\myenv\lib\site-packages\elpigraph_BaseElPiWrapper.py", line 284, in computeElasticPrincipalGraphWithGrammars DisplayWarnings=DisplayWarnings File "C:\Users\tonystark\AppData\Local\conda\conda\envs\myenv\lib\site-packages\elpigraph\src\BaseElPi.py", line 585, in computeElasticPrincipalGraph n_cores=n_cores,MinParOp=MinParOp) File "C:\Users\tonystark\AppData\Local\conda\conda\envs\myenv\lib\site-packages\elpigraph\src\BaseElPi.py", line 184, in ElPrincGraph pool = mp.Pool(n_cores) File "C:\Users\tonystark\AppData\Local\conda\conda\envs\myenv\lib\multiprocessing\context.py", line 119, in Pool context=self.get_context()) File "C:\Users\tonystark\AppData\Local\conda\conda\envs\myenv\lib\multiprocessing\pool.py", line 177, in init__ self._repopulate_pool() File "C:\Users\tonystark\AppData\Local\conda\conda\envs\myenv\lib\multiprocessing\pool.py", line 238, in _repopulate_pool self._wrap_exception) File "C:\Users\tonystark\AppData\Local\conda\conda\envs\myenv\lib\multiprocessing\pool.py", line 257, in _repopulate_pool_static w.start() File "C:\Users\tonystark\AppData\Local\conda\conda\envs\myenv\lib\multiprocessing\process.py", line 112, in start self._popen = self._Popen(self) File "C:\Users\tonystark\AppData\Local\conda\conda\envs\myenv\lib\multiprocessing\context.py", line 322, in _Popen return Popen(process_obj) File "C:\Users\tonystark\AppData\Local\conda\conda\envs\myenv\lib\multiprocessing\popen_spawn_win32.py", line 33, in init prep_data = spawn.get_preparation_data(process_obj._name) File "C:\Users\tonystark\AppData\Local\conda\conda\envs\myenv\lib\multiprocessing\spawn.py", line 143, in get_preparation_data _check_not_importing_main() File "C:\Users\tonystark\AppData\Local\conda\conda\envs\myenv\lib\multiprocessing\spawn.py", line 136, in _check_not_importing_main is not going to be frozen to produce an executable.''') RuntimeError: An attempt has been made to start a new process before the current process has finished its bootstrapping phase.

    This probably means that you are not using fork to start your
    child processes and you have forgotten to use the proper idiom
    in the main module:

        if __name__ == '__main__':
            freeze_support()
            ...

    The "freeze_support()" line can be omitted if the program
    is not going to be frozen to produce an executable.

Generating ElPiGraph output for file - C:\Users\tonystark\Documents\folder\elpigraph\testing\data\tree_data.csv

Computing Elastic Principal Curve for file- C:\Users\tonystark\Documents\folder\elpigraph\testing\data\tree_data.csv

Generating the initial configuration Creating a chain in the 1st PC with 2 nodes Generating ElPiGraph output for file - C:\Users\tonystark\Documents\folder\elpigraph\testing\data\tree_data.csv

Computing Elastic Principal Curve for file- C:\Users\tonystark\Documents\folder\elpigraph\testing\data\tree_data.csv

Generating the initial configuration Creating a chain in the 1st PC with 2 nodes Generating ElPiGraph output for file - C:\Users\tonystark\Documents\folder\elpigraph\testing\data\tree_data.csv

Computing Elastic Principal Curve for file- C:\Users\tonystark\Documents\folder\elpigraph\testing\data\tree_data.csv

Generating the initial configuration Creating a chain in the 1st PC with 2 nodes Generating ElPiGraph output for file - C:\Users\tonystark\Documents\folder\elpigraph\testing\data\tree_data.csv

Computing Elastic Principal Curve for file- C:\Users\tonystark\Documents\folder\elpigraph\testing\data\tree_data.csv

Generating the initial configuration Creating a chain in the 1st PC with 2 nodes 90% of the points have been used as initial conditions. Resetting. 90% of the points have been used as initial conditions. Resetting. Constructing tree 1 of 1 / Subset 1 of 1 Performing PCA Using standard PCA 3 dimensions are being used 100.0 % of the original variance has been retained The elastic matrix is being used. Edge configuration will be ignored Computing EPG with 60 nodes on 492 points and 3 dimensions Traceback (most recent call last): File "", line 1, in Creating a Pool with 4 processes Constructing tree 1 of 1 / Subset 1 of 1 Performing PCA Using standard PCA 3 dimensions are being used 100.0 % of the original variance has been retained The elastic matrix is being used. Edge configuration will be ignored Computing EPG with 60 nodes on 492 points and 3 dimensions Creating a Pool with 4 processes Traceback (most recent call last): File "", line 1, in File "C:\Users\tonystark\AppData\Local\conda\conda\envs\myenv\lib\multiprocessing\spawn.py", line 105, in spawn_main exitcode = _main(fd) File "C:\Users\tonystark\AppData\Local\conda\conda\envs\myenv\lib\multiprocessing\spawn.py", line 114, in _main prepare(preparation_data) File "C:\Users\tonystark\AppData\Local\conda\conda\envs\myenv\lib\multiprocessing\spawn.py", line 225, in prepare _fixup_main_from_path(data['init_main_from_path']) File "C:\Users\tonystark\AppData\Local\conda\conda\envs\myenv\lib\multiprocessing\spawn.py", line 277, in _fixup_main_from_path run_name="mp_main") File "C:\Users\tonystark\AppData\Local\conda\conda\envs\myenv\lib\runpy.py", line 263, in run_path File "C:\Users\tonystark\AppData\Local\conda\conda\envs\myenv\lib\multiprocessing\spawn.py", line 105, in spawn_main pkg_name=pkg_name, script_name=fname) File "C:\Users\tonystark\AppData\Local\conda\conda\envs\myenv\lib\runpy.py", line 96, in _run_module_code mod_name, mod_spec, pkg_name, script_name) File "C:\Users\tonystark\AppData\Local\conda\conda\envs\myenv\lib\runpy.py", line 85, in _run_code exec(code, run_globals) File "C:\Users\tonystark\Documents\folder\elpigraph\testing\mutliprocessing check.py", line 24, in exitcode = _main(fd) File "C:\Users\tonystark\AppData\Local\conda\conda\envs\myenv\lib\multiprocessing\spawn.py", line 114, in _main prepare(preparation_data) File "C:\Users\tonystark\AppData\Local\conda\conda\envs\myenv\lib\multiprocessing\spawn.py", line 225, in prepare _fixup_main_from_path(data['init_main_from_path']) File "C:\Users\tonystark\AppData\Local\conda\conda\envs\myenv\lib\multiprocessing\spawn.py", line 277, in _fixup_main_from_path run_name="mp_main") File "C:\Users\tonystark\AppData\Local\conda\conda\envs\myenv\lib\runpy.py", line 263, in run_path computeElasticPrincipalCurve(first_output.astype('float64').values, 60,n_cores=4)
File "C:\Users\tonystark\AppData\Local\conda\conda\envs\myenv\lib\site-packages\elpigraph_topologies.py", line 597, in computeElasticPrincipalCurve MaxSteps = MaxSteps File "C:\Users\tonystark\AppData\Local\conda\conda\envs\myenv\lib\site-packages\elpigraph_BaseElPiWrapper.py", line 284, in computeElasticPrincipalGraphWithGrammars DisplayWarnings=DisplayWarnings File "C:\Users\tonystark\AppData\Local\conda\conda\envs\myenv\lib\site-packages\elpigraph\src\BaseElPi.py", line 585, in computeElasticPrincipalGraph pkg_name=pkg_name, script_name=fname) File "C:\Users\tonystark\AppData\Local\conda\conda\envs\myenv\lib\runpy.py", line 96, in _run_module_code mod_name, mod_spec, pkg_name, script_name) File "C:\Users\tonystark\AppData\Local\conda\conda\envs\myenv\lib\runpy.py", line 85, in _run_code exec(code, run_globals) File "C:\Users\tonystark\Documents\folder\elpigraph\testing\mutliprocessing check.py", line 24, in computeElasticPrincipalCurve(first_output.astype('float64').values, 60,n_cores=4)
File "C:\Users\tonystark\AppData\Local\conda\conda\envs\myenv\lib\site-packages\elpigraph_topologies.py", line 597, in computeElasticPrincipalCurve MaxSteps = MaxSteps File "C:\Users\tonystark\AppData\Local\conda\conda\envs\myenv\lib\site-packages\elpigraph_BaseElPiWrapper.py", line 284, in computeElasticPrincipalGraphWithGrammars DisplayWarnings=DisplayWarnings File "C:\Users\tonystark\AppData\Local\conda\conda\envs\myenv\lib\site-packages\elpigraph\src\BaseElPi.py", line 585, in computeElasticPrincipalGraph n_cores=n_cores,MinParOp=MinParOp) File "C:\Users\tonystark\AppData\Local\conda\conda\envs\myenv\lib\site-packages\elpigraph\src\BaseElPi.py", line 184, in ElPrincGraph pool = mp.Pool(n_cores) File "C:\Users\tonystark\AppData\Local\conda\conda\envs\myenv\lib\multiprocessing\context.py", line 119, in Pool n_cores=n_cores,MinParOp=MinParOp) File "C:\Users\tonystark\AppData\Local\conda\conda\envs\myenv\lib\site-packages\elpigraph\src\BaseElPi.py", line 184, in ElPrincGraph pool = mp.Pool(n_cores) File "C:\Users\tonystark\AppData\Local\conda\conda\envs\myenv\lib\multiprocessing\context.py", line 119, in Pool context=self.get_context()) File "C:\Users\tonystark\AppData\Local\conda\conda\envs\myenv\lib\multiprocessing\pool.py", line 177, in init context=self.get_context()) File "C:\Users\tonystark\AppData\Local\conda\conda\envs\myenv\lib\multiprocessing\pool.py", line 177, in init self._repopulate_pool() File "C:\Users\tonystark\AppData\Local\conda\conda\envs\myenv\lib\multiprocessing\pool.py", line 238, in _repopulate_pool self._wrap_exception) File "C:\Users\tonystark\AppData\Local\conda\conda\envs\myenv\lib\multiprocessing\pool.py", line 257, in _repopulate_pool_static w.start() File "C:\Users\tonystark\AppData\Local\conda\conda\envs\myenv\lib\multiprocessing\process.py", line 112, in start self._repopulate_pool() File "C:\Users\tonystark\AppData\Local\conda\conda\envs\myenv\lib\multiprocessing\pool.py", line 238, in _repopulate_pool self._wrap_exception) File "C:\Users\tonystark\AppData\Local\conda\conda\envs\myenv\lib\multiprocessing\pool.py", line 257, in _repopulate_pool_static w.start() File "C:\Users\tonystark\AppData\Local\conda\conda\envs\myenv\lib\multiprocessing\process.py", line 112, in start self._popen = self._Popen(self) File "C:\Users\tonystark\AppData\Local\conda\conda\envs\myenv\lib\multiprocessing\context.py", line 322, in _Popen return Popen(process_obj) File "C:\Users\tonystark\AppData\Local\conda\conda\envs\myenv\lib\multiprocessing\popen_spawn_win32.py", line 33, in init self._popen = self._Popen(self) File "C:\Users\tonystark\AppData\Local\conda\conda\envs\myenv\lib\multiprocessing\context.py", line 322, in _Popen return Popen(process_obj) File "C:\Users\tonystark\AppData\Local\conda\conda\envs\myenv\lib\multiprocessing\popen_spawn_win32.py", line 33, in init prep_data = spawn.get_preparation_data(process_obj._name) File "C:\Users\tonystark\AppData\Local\conda\conda\envs\myenv\lib\multiprocessing\spawn.py", line 143, in get_preparation_data _check_not_importing_main() File "C:\Users\tonystark\AppData\Local\conda\conda\envs\myenv\lib\multiprocessing\spawn.py", line 136, in _check_not_importing_main is not going to be frozen to produce an executable.''') RuntimeError: An attempt has been made to start a new process before the current process has finished its bootstrapping phase.

    This probably means that you are not using fork to start your
    child processes and you have forgotten to use the proper idiom
    in the main module:

        if __name__ == '__main__':
            freeze_support()
            ...

    The "freeze_support()" line can be omitted if the program
    is not going to be frozen to produce an executable.
prep_data = spawn.get_preparation_data(process_obj._name)

File "C:\Users\tonystark\AppData\Local\conda\conda\envs\myenv\lib\multiprocessing\spawn.py", line 143, in get_preparation_data _check_not_importing_main() File "C:\Users\tonystark\AppData\Local\conda\conda\envs\myenv\lib\multiprocessing\spawn.py", line 136, in _check_not_importing_main is not going to be frozen to produce an executable.''') RuntimeError: An attempt has been made to start a new process before the current process has finished its bootstrapping phase.

    This probably means that you are not using fork to start your
    child processes and you have forgotten to use the proper idiom
    in the main module:

        if __name__ == '__main__':
            freeze_support()
            ...

    The "freeze_support()" line can be omitted if the program
    is not going to be frozen to produce an executable.

90% of the points have been used as initial conditions. Resetting. Constructing tree 1 of 1 / Subset 1 of 1 Performing PCA Using standard PCA 3 dimensions are being used 100.0 % of the original variance has been retained The elastic matrix is being used. Edge configuration will be ignored Computing EPG with 60 nodes on 492 points and 3 dimensions Creating a Pool with 4 processes Traceback (most recent call last): File "", line 1, in File "C:\Users\tonystark\AppData\Local\conda\conda\envs\myenv\lib\multiprocessing\spawn.py", line 105, in spawn_main exitcode = _main(fd) File "C:\Users\tonystark\AppData\Local\conda\conda\envs\myenv\lib\multiprocessing\spawn.py", line 114, in _main prepare(preparation_data) File "C:\Users\tonystark\AppData\Local\conda\conda\envs\myenv\lib\multiprocessing\spawn.py", line 225, in prepare _fixup_main_from_path(data['init_main_from_path']) File "C:\Users\tonystark\AppData\Local\conda\conda\envs\myenv\lib\multiprocessing\spawn.py", line 277, in _fixup_main_from_path run_name="mp_main__") File "C:\Users\tonystark\AppData\Local\conda\conda\envs\myenv\lib\runpy.py", line 263, in run_path pkg_name=pkg_name, script_name=fname) File "C:\Users\tonystark\AppData\Local\conda\conda\envs\myenv\lib\runpy.py", line 96, in _run_module_code mod_name, mod_spec, pkg_name, script_name) File "C:\Users\tonystark\AppData\Local\conda\conda\envs\myenv\lib\runpy.py", line 85, in _run_code exec(code, run_globals) File "C:\Users\tonystark\Documents\folder\elpigraph\testing\mutliprocessing check.py", line 24, in computeElasticPrincipalCurve(first_output.astype('float64').values, 60,n_cores=4)
File "C:\Users\tonystark\AppData\Local\conda\conda\envs\myenv\lib\site-packages\elpigraph_topologies.py", line 597, in computeElasticPrincipalCurve MaxSteps = MaxSteps File "C:\Users\tonystark\AppData\Local\conda\conda\envs\myenv\lib\site-packages\elpigraph_BaseElPiWrapper.py", line 284, in computeElasticPrincipalGraphWithGrammars DisplayWarnings=DisplayWarnings File "C:\Users\tonystark\AppData\Local\conda\conda\envs\myenv\lib\site-packages\elpigraph\src\BaseElPi.py", line 585, in computeElasticPrincipalGraph n_cores=n_cores,MinParOp=MinParOp) File "C:\Users\tonystark\AppData\Local\conda\conda\envs\myenv\lib\site-packages\elpigraph\src\BaseElPi.py", line 184, in ElPrincGraph pool = mp.Pool(n_cores) File "C:\Users\tonystark\AppData\Local\conda\conda\envs\myenv\lib\multiprocessing\context.py", line 119, in Pool context=self.get_context()) File "C:\Users\tonystark\AppData\Local\conda\conda\envs\myenv\lib\multiprocessing\pool.py", line 177, in init__ self._repopulate_pool() File "C:\Users\tonystark\AppData\Local\conda\conda\envs\myenv\lib\multiprocessing\pool.py", line 238, in _repopulate_pool self._wrap_exception) File "C:\Users\tonystark\AppData\Local\conda\conda\envs\myenv\lib\multiprocessing\pool.py", line 257, in _repopulate_pool_static w.start() File "C:\Users\tonystark\AppData\Local\conda\conda\envs\myenv\lib\multiprocessing\process.py", line 112, in start self._popen = self._Popen(self) File "C:\Users\tonystark\AppData\Local\conda\conda\envs\myenv\lib\multiprocessing\context.py", line 322, in _Popen return Popen(process_obj) File "C:\Users\tonystark\AppData\Local\conda\conda\envs\myenv\lib\multiprocessing\popen_spawn_win32.py", line 33, in init prep_data = spawn.get_preparation_data(process_obj._name) File "C:\Users\tonystark\AppData\Local\conda\conda\envs\myenv\lib\multiprocessing\spawn.py", line 143, in get_preparation_data _check_not_importing_main() File "C:\Users\tonystark\AppData\Local\conda\conda\envs\myenv\lib\multiprocessing\spawn.py", line 136, in _check_not_importing_main is not going to be frozen to produce an executable.''') RuntimeError: An attempt has been made to start a new process before the current process has finished its bootstrapping phase.

    This probably means that you are not using fork to start your
    child processes and you have forgotten to use the proper idiom
    in the main module:

        if __name__ == '__main__':
            freeze_support()
            ...

    The "freeze_support()" line can be omitted if the program
    is not going to be frozen to produce an executable.

90% of the points have been used as initial conditions. Resetting. Constructing tree 1 of 1 / Subset 1 of 1 Performing PCA Using standard PCA 3 dimensions are being used 100.0 % of the original variance has been retained The elastic matrix is being used. Edge configuration will be ignored Computing EPG with 60 nodes on 492 points and 3 dimensions Creating a Pool with 4 processes Traceback (most recent call last): File "", line 1, in File "C:\Users\tonystark\AppData\Local\conda\conda\envs\myenv\lib\multiprocessing\spawn.py", line 105, in spawn_main exitcode = _main(fd) File "C:\Users\tonystark\AppData\Local\conda\conda\envs\myenv\lib\multiprocessing\spawn.py", line 114, in _main prepare(preparation_data) File "C:\Users\tonystark\AppData\Local\conda\conda\envs\myenv\lib\multiprocessing\spawn.py", line 225, in prepare _fixup_main_from_path(data['init_main_from_path']) File "C:\Users\tonystark\AppData\Local\conda\conda\envs\myenv\lib\multiprocessing\spawn.py", line 277, in _fixup_main_from_path run_name="mp_main__") File "C:\Users\tonystark\AppData\Local\conda\conda\envs\myenv\lib\runpy.py", line 263, in run_path pkg_name=pkg_name, script_name=fname) File "C:\Users\tonystark\AppData\Local\conda\conda\envs\myenv\lib\runpy.py", line 96, in _run_module_code mod_name, mod_spec, pkg_name, script_name) File "C:\Users\tonystark\AppData\Local\conda\conda\envs\myenv\lib\runpy.py", line 85, in _run_code exec(code, run_globals) File "C:\Users\tonystark\Documents\folder\elpigraph\testing\mutliprocessing check.py", line 24, in computeElasticPrincipalCurve(first_output.astype('float64').values, 60,n_cores=4)
File "C:\Users\tonystark\AppData\Local\conda\conda\envs\myenv\lib\site-packages\elpigraph_topologies.py", line 597, in computeElasticPrincipalCurve MaxSteps = MaxSteps File "C:\Users\tonystark\AppData\Local\conda\conda\envs\myenv\lib\site-packages\elpigraph_BaseElPiWrapper.py", line 284, in computeElasticPrincipalGraphWithGrammars DisplayWarnings=DisplayWarnings File "C:\Users\tonystark\AppData\Local\conda\conda\envs\myenv\lib\site-packages\elpigraph\src\BaseElPi.py", line 585, in computeElasticPrincipalGraph n_cores=n_cores,MinParOp=MinParOp) File "C:\Users\tonystark\AppData\Local\conda\conda\envs\myenv\lib\site-packages\elpigraph\src\BaseElPi.py", line 184, in ElPrincGraph pool = mp.Pool(n_cores) File "C:\Users\tonystark\AppData\Local\conda\conda\envs\myenv\lib\multiprocessing\context.py", line 119, in Pool context=self.get_context()) File "C:\Users\tonystark\AppData\Local\conda\conda\envs\myenv\lib\multiprocessing\pool.py", line 177, in init__ self._repopulate_pool() File "C:\Users\tonystark\AppData\Local\conda\conda\envs\myenv\lib\multiprocessing\pool.py", line 238, in _repopulate_pool self._wrap_exception) File "C:\Users\tonystark\AppData\Local\conda\conda\envs\myenv\lib\multiprocessing\pool.py", line 257, in _repopulate_pool_static w.start() File "C:\Users\tonystark\AppData\Local\conda\conda\envs\myenv\lib\multiprocessing\process.py", line 112, in start self._popen = self._Popen(self) File "C:\Users\tonystark\AppData\Local\conda\conda\envs\myenv\lib\multiprocessing\context.py", line 322, in _Popen return Popen(process_obj) File "C:\Users\tonystark\AppData\Local\conda\conda\envs\myenv\lib\multiprocessing\popen_spawn_win32.py", line 33, in init prep_data = spawn.get_preparation_data(process_obj._name) File "C:\Users\tonystark\AppData\Local\conda\conda\envs\myenv\lib\multiprocessing\spawn.py", line 143, in get_preparation_data _check_not_importing_main() File "C:\Users\tonystark\AppData\Local\conda\conda\envs\myenv\lib\multiprocessing\spawn.py", line 136, in _check_not_importing_main is not going to be frozen to produce an executable.''') RuntimeError: An attempt has been made to start a new process before the current process has finished its bootstrapping phase.

    This probably means that you are not using fork to start your
    child processes and you have forgotten to use the proper idiom
    in the main module:

        if __name__ == '__main__':
            freeze_support()
            ...

    The "freeze_support()" line can be omitted if the program
    is not going to be frozen to produce an executable.
kunamneni117 commented 4 years ago

I think link give some light for the current issue.