Refefer / fastxml

FastXML / PFastXML / PFastreXML - Implementation of Extreme Multi-label Classification
Other
149 stars 47 forks source link

TypeError: can't pickle fastxml.splitter.Splitter objects #12

Closed DataAndCodeMania closed 5 years ago

DataAndCodeMania commented 6 years ago

Hi,

X is a list of csr_matrix. y is a list of lists (as shown below)

from scipy import sparse
X = np.random.rand(10, 10)
y = [[1], [2], [3], [1], [2], [3], [1], [2], [3], [1]]
XList = []
for i in range(10):
    XList.append(sparse.csr_matrix(X[i]))

X = XList   
trainer = Trainer(n_trees=32, n_jobs=-1)
trainer.fit(X, y)

trainer.fit(X, y) Traceback (most recent call last): File "", line 1, in File "C:\ProgramData\Anaconda3\lib\site-packages\fastxml-2.0.0-py3.6-win-amd64.egg\fastxml\trainer.py", line 465, in fit self.roots = self._build_roots(X, y, weights) File "C:\ProgramData\Anaconda3\lib\site-packages\fastxml-2.0.0-py3.6-win-amd64.egg\fastxml\trainer.py", line 407, in _build_roots procs.append(f(X, y, next(idxs), rs, splitter)) File "C:\ProgramData\Anaconda3\lib\site-packages\fastxml-2.0.0-py3.6-win-amd64.egg\fastxml\proc.py", line 50, in f2 p.start() File "C:\ProgramData\Anaconda3\lib\multiprocessing\process.py", line 105, in start self._popen = self._Popen(self) File "C:\ProgramData\Anaconda3\lib\multiprocessing\context.py", line 223, in _Popen return _default_context.get_context().Process._Popen(process_obj) File "C:\ProgramData\Anaconda3\lib\multiprocessing\context.py", line 322, in _Popen return Popen(process_obj) File "C:\ProgramData\Anaconda3\lib\multiprocessing\popen_spawn_win32.py", line 65, in init reduction.dump(process_obj, to_child) File "C:\ProgramData\Anaconda3\lib\multiprocessing\reduction.py", line 60, in dump ForkingPickler(file, protocol).dump(obj) TypeError: can't pickle fastxml.splitter.Splitter objects

Refefer commented 6 years ago

Interesting; I haven't seen this before. I wonder if this is due to how windows does forking (or lack there of).

Two asks since I don't have access to a windows machine (I only run linux):

  1. Can you try with python2 on your windows machine?
  2. Would you be willing to help debug a windows build if it's determined that that's the issue?