search

mars-project / mars

Mars is a tensor-based unified framework for large-scale data computation which scales numpy, pandas, scikit-learn and Python functions.
https://mars-project.readthedocs.io
Apache License 2.0
2.68k stars 325 forks source link

MARS sample with Distributed Flaml/AutoML? #3273

Closed wil70 closed 1 year ago

wil70 commented 1 year ago

Hello Hello, I'm trying to run a distributed AutoML with MARS + LightGBM for huge (TB) csv input file. I can't find how to do it any idea? TY! https://microsoft.github.io/FLAML/

qinxuye commented 1 year ago

So you are using FLAML to do automl, and your data is ~TB, do you want a demo or sth to do this?

qinxuye commented 1 year ago

Seems you have a duplicated issue opened in FLAML, let's have a discussion there. Will close this issue first.

wthomas-og commented 1 year ago

Thanks @sonichi - your answer help as I'm learning Python and Mars (come from c, c++, c#, java) https://github.com/microsoft/FLAML/issues/758#issuecomment-1274078507

Hi @qinxuye , yes I have a bunch of data I would like to train on. Note: I do not know what sth mean or the sentence when you say "do you want a demo or sth to do this?" (I'm totally new so I'm probably missing context - TY!)

Question, I can train huge amount of data with the ligthGBM in c++, but

  1. I could use an parameter tuning tool like Flaml. IS there a way to do this? or can I reuse the parameters found on a sub-set in the c++ version? I know Python call the c/c++ version, but
  2. I wasn't able to do handle huge dataset yet with Python (and Panda), I try with DAS but that didn't work, and now I'm trying with Mars on 1 machine then I will try on several machines. How do I handle those huge csv file and compute in parallel with Python? I will look at @sonichi

TY

qinxuye commented 1 year ago

Did you run Mars + LGM successfully? @wthomas-og

wil70 commented 1 year ago

Hello, it fails to import somehow, it says

Cell In [3], line 4
      1 from sklearn.datasets import load_boston
      2 boston = load_boston()
----> 4 import mars.dataframe as md
      5 data = md.DataFrame(boston.data, columns=boston.feature_names)
      7 data.head().execute()

File c:\Users\Wilhelm\AppData\Local\Programs\Python\Python310\lib\site-packages\mars\dataframe\__init__.py:39
     33 from .datasource.read_parquet import read_parquet
     34 from .datasource.read_raydataset import (
     35     read_raydataset,
     36     read_ray_dataset,
     37     read_ray_mldataset,
     38 )
---> 39 from .datasource.date_range import date_range
     40 from .fetch import DataFrameFetch, DataFrameFetchShuffle
     41 from .merge import concat, merge

File c:\Users\Wilhelm\AppData\Local\Programs\Python\Python310\lib\site-packages\mars\dataframe\datasource\date_range.py:303
    291         ts = ts.tz_localize(**localize_args)
    292     return ts
    295 def date_range(
    296     start=None,
...
    372 if item in {'__name__', '__qualname__'}:
    373     return self._name
--> 375 raise AttributeError(item)

AttributeError: no_default

I have

C:\Users\W> pip install pymars
WARNING: Ignoring invalid distribution -ymars (c:\users\wilhelm\appdata\local\programs\python\python310\lib\site-packages)
WARNING: Ignoring invalid distribution -ymars (c:\users\wilhelm\appdata\local\programs\python\python310\lib\site-packages)
Requirement already satisfied: pymars in c:\users\wilhelm\appdata\local\programs\python\python310\lib\site-packages (0.9.0)
Requirement already satisfied: tornado>=6.0 in c:\users\wilhelm\appdata\local\programs\python\python310\lib\site-packages (from pymars) (6.1)
Requirement already satisfied: cloudpickle>=1.5.0 in c:\users\wilhelm\appdata\local\programs\python\python310\lib\site-packages (from pymars) (2.2.0)
Requirement already satisfied: defusedxml>=0.5.0 in c:\users\wilhelm\appdata\local\programs\python\python310\lib\site-packages (from pymars) (0.7.1)
Requirement already satisfied: pandas>=1.0.0 in c:\users\wilhelm\appdata\local\programs\python\python310\lib\site-packages (from pymars) (1.5.0)
Requirement already satisfied: tqdm>=4.1.0 in c:\users\wilhelm\appdata\local\programs\python\python310\lib\site-packages (from pymars) (4.64.1)
Requirement already satisfied: scipy>=1.0.0 in c:\users\wilhelm\appdata\local\programs\python\python310\lib\site-packages (from pymars) (1.9.2)
Requirement already satisfied: psutil>=4.0.0 in c:\users\wilhelm\appdata\roaming\python\python310\site-packages (from pymars) (5.9.2)
Requirement already satisfied: sqlalchemy>=1.2.0 in c:\users\wilhelm\appdata\local\programs\python\python310\lib\site-packages (from pymars) (1.4.41)
Requirement already satisfied: numpy>=1.14.0 in c:\users\wilhelm\appdata\local\programs\python\python310\lib\site-packages (from pymars) (1.23.3)
Requirement already satisfied: numexpr>=2.6.4 in c:\users\wilhelm\appdata\local\programs\python\python310\lib\site-packages (from pymars) (2.8.3)
Requirement already satisfied: scikit-learn>=0.20 in c:\users\wilhelm\appdata\local\programs\python\python310\lib\site-packages (from pymars) (1.1.2)
Requirement already satisfied: pyyaml>=5.1 in c:\users\wilhelm\appdata\local\programs\python\python310\lib\site-packages (from pymars) (6.0)
Requirement already satisfied: packaging in c:\users\wilhelm\appdata\roaming\python\python310\site-packages (from numexpr>=2.6.4->pymars) (21.3)
Requirement already satisfied: pytz>=2020.1 in c:\users\wilhelm\appdata\local\programs\python\python310\lib\site-packages (from pandas>=1.0.0->pymars) (2022.4)
Requirement already satisfied: python-dateutil>=2.8.1 in c:\users\wilhelm\appdata\roaming\python\python310\site-packages (from pandas>=1.0.0->pymars) (2.8.2)
Requirement already satisfied: threadpoolctl>=2.0.0 in c:\users\wilhelm\appdata\local\programs\python\python310\lib\site-packages (from scikit-learn>=0.20->pymars) (3.1.0)
Requirement already satisfied: joblib>=1.0.0 in c:\users\wilhelm\appdata\local\programs\python\python310\lib\site-packages (from scikit-learn>=0.20->pymars) (1.2.0)
Requirement already satisfied: greenlet!=0.4.17 in c:\users\wilhelm\appdata\local\programs\python\python310\lib\site-packages (from sqlalchemy>=1.2.0->pymars) (1.1.3.post0)
Requirement already satisfied: colorama in c:\users\wilhelm\appdata\roaming\python\python310\site-packages (from tqdm>=4.1.0->pymars) (0.4.5)
Requirement already satisfied: six>=1.5 in c:\users\wilhelm\appdata\roaming\python\python310\site-packages (from python-dateutil>=2.8.1->pandas>=1.0.0->pymars) (1.16.0)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in c:\users\wilhelm\appdata\roaming\python\python310\site-packages (from packaging->numexpr>=2.6.4->pymars) (3.0.9)
WARNING: Ignoring invalid distribution -ymars (c:\users\wilhelm\appdata\local\programs\python\python310\lib\site-packages)
WARNING: Ignoring invalid distribution -ymars (c:\users\wilhelm\appdata\local\programs\python\python310\lib\site-packages)
WARNING: Ignoring invalid distribution -ymars (c:\users\wilhelm\appdata\local\programs\python\python310\lib\site-packages)
WARNING: Ignoring invalid distribution -ymars (c:\users\wilhelm\appdata\local\programs\python\python310\lib\site-packages)
qinxuye commented 1 year ago

Could you please try pandas < 1.5.0 via pip install 'pandas<1.5.0'? It looks like the failure is caused by new version of pandas.

wil70 commented 1 year ago

Thanks @qinxuye 

from pickle import FALSE, TRUE
import lightgbm as lgb
import pandas as pd
import numpy as np

import mars.dataframe as md
from mars.dataframe.datasource import read_csv as md

# Loading the data:
data = md.read_csv('huge.csv')
data[data==np.nan] = 1000000000
data[data>1000000]=1000000
data[data<-1000000]=-1000000

# define input and output feature
x = data.drop([data.columns[4204],data.columns[4205],data.columns[4206],data.columns[4207]],axis=1).execute()
y = data[data.columns[4207]]

# train test split
x_train,x_test,y_train,y_test = train_test_split(x,y,test_size=0.33,random_state=42)

# might need to rewrite this....
from flaml import AutoML
automl = AutoML()
automl.fit(X_train=x_train, y_train=y_train, time_budget=60*3, estimator_list=['lgbm'], task='regression') #'regression classification

It fails at the dataframe.drop https://docs.pymars.org/en/latest/reference/dataframe/generated/mars.dataframe.DataFrame.drop.html#mars.dataframe.DataFrame.drop

c:\Users\W\AppData\Local\Programs\Python\Python310\lib\site-packages\mars\deploy\oscar\session.py:2064: UserWarning: 
No session found, local session will be created in background, it may take a while before execution. If you want to new a local session by yourself, run code below:

import mars

mars.new_session()

  warnings.warn(warning_msg)
Output exceeds the [size limit](command:workbench.action.openSettings?[). Open the full output data [in a text editor](command:workbench.action.openLargeOutput?74cdc4b1-7c8a-4e64-84e9-1ba33fe107a6)
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
Cell In [4], line 29
     19 data[data<-1000000]=-1000000
     21 #data.head()
     22 #data = data.dropna()
     23 #data = data.apply(lambda x: x.fillna(1000000))
   (...)
     27 
     28 # define input and output feature
---> 29 x = data.drop([data.columns[4204],data.columns[4205],data.columns[4206],data.columns[4207]],axis=1).execute()
     30 y = data[data.columns[4207]]

File c:\Users\W\AppData\Local\Programs\Python\Python310\lib\site-packages\mars\dataframe\base\drop.py:349, in df_drop(df, labels, axis, index, columns, level, inplace, errors)
    218 def df_drop(
    219     df,
    220     labels=None,
   (...)
    226     errors="raise",
    227 ):
    228     """
    229     Drop specified labels from rows or columns.
    230 
   (...)
    347             weight  1.0     0.8
...
--> 392     func = self._FuncPtr((name_or_ordinal, self))
    393     if not isinstance(name_or_ordinal, int):
    394         func.__name__ = name_or_ordinal

AttributeError: function 'nvmlInit_v2' not found
wil70 commented 1 year ago

I'm working with a huge csv file of 1/2 TB

qinxuye commented 1 year ago

Could you paste some sample data so I can have some test?

wil70 commented 1 year ago

Here you go, I put 25 rows as sample test_tmp.csv TY

qinxuye commented 1 year ago

Could you try to modify the code into

x = data.drop([data.dtypes.index[4204],data.dtypes.index[4205],data.dtypes.index[4206],data.dtypes.index[4207]],axis=1).execute()

which basically changed data.columns into data.dtypes.index.

to see if it can work.

wil70 commented 1 year ago

Thanks @qinxuye Here you go


AppData\Local\Programs\Python\Python310\lib\site-packages\mars\deploy\oscar\session.py:2064: UserWarning: 
No session found, local session will be created in background, it may take a while before execution. If you want to new a local session by yourself, run code below:

import mars

mars.new_session()

  warnings.warn(warning_msg)
Output exceeds the [size limit](command:workbench.action.openSettings?[). Open the full output data [in a text editor](command:workbench.action.openLargeOutput?b855ad54-0e51-439f-bcb2-ba090384e725)
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
Cell In [10], line 30
     20 data[data<-1000000]=-1000000
     22 #data.head()
     23 #data = data.dropna()
     24 #data = data.apply(lambda x: x.fillna(1000000))
   (...)
     28 #mars.new_session()
     29 # define input and output feature
---> 30 x = data.drop([data.dtypes.index[4204],data.dtypes.index[4205],data.dtypes.index[4206],data.dtypes.index[4207]],axis=1).execute() #which basically changed data.columns into data.dtypes.index.
     31 #x = data.drop([data.columns[4204],data.columns[4205],data.columns[4206],data.columns[4207]],axis=1).execute()
     32 y = data[data.columns[4207]]

File c:\Users\Wilhelm\AppData\Local\Programs\Python\Python310\lib\site-packages\mars\core\entity\tileables.py:462, in HasShapeTileable.execute(self, session, **kw)
    461 def execute(self, session=None, **kw):
--> 462     result = self.data.execute(session=session, **kw)
    463     if isinstance(result, TILEABLE_TYPE):
    464         return self

File c:\Users\Wilhelm\AppData\Local\Programs\Python\Python310\lib\site-packages\mars\core\entity\executable.py:144, in _ExecutableMixin.execute(self, session, **kw)
    141 from ...deploy.oscar.session import execute
    143 session = _get_session(self, session)
--> 144 return execute(self, session=session, **kw)
...
--> 392     func = self._FuncPtr((name_or_ordinal, self))
    393     if not isinstance(name_or_ordinal, int):
    394         func.__name__ = name_or_ordinal

AttributeError: function 'nvmlInit_v2' not found
qinxuye commented 1 year ago

@wil70 Could you please paste the entire error stack, seems the issue is caused by GPU stuff.

wil70 commented 1 year ago

Thanks @qinxuye Strange, I'm not using a GPU

Here you go TY! WIl

{
    "name": "AttributeError",
    "message": "function 'nvmlInit_v2' not found",
    "stack": "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m\n\u001b[1;31mAttributeError\u001b[0m                            Traceback (most recent call last)\nCell \u001b[1;32mIn [8], line 20\u001b[0m\n\u001b[0;32m     15\u001b[0m data[data\u001b[39m<\u001b[39m\u001b[39m-\u001b[39m\u001b[39m1000000\u001b[39m]\u001b[39m=\u001b[39m\u001b[39m-\u001b[39m\u001b[39m1000000\u001b[39m\n\u001b[0;32m     17\u001b[0m \u001b[39m# define input and output feature\u001b[39;00m\n\u001b[0;32m     18\u001b[0m \u001b[39m#x = data.drop([data.columns[4204],data.columns[4205],data.columns[4206],data.columns[4207]],axis=1).execute()\u001b[39;00m\n\u001b[0;32m     19\u001b[0m \u001b[39m#y = data[data.columns[4207]]\u001b[39;00m\n\u001b[1;32m---> 20\u001b[0m x \u001b[39m=\u001b[39m data\u001b[39m.\u001b[39;49mdrop([data\u001b[39m.\u001b[39;49mdtypes\u001b[39m.\u001b[39;49mindex[\u001b[39m4204\u001b[39;49m],data\u001b[39m.\u001b[39;49mdtypes\u001b[39m.\u001b[39;49mindex[\u001b[39m4205\u001b[39;49m],data\u001b[39m.\u001b[39;49mdtypes\u001b[39m.\u001b[39;49mindex[\u001b[39m4206\u001b[39;49m],data\u001b[39m.\u001b[39;49mdtypes\u001b[39m.\u001b[39;49mindex[\u001b[39m4207\u001b[39;49m]],axis\u001b[39m=\u001b[39;49m\u001b[39m1\u001b[39;49m)\u001b[39m.\u001b[39;49mexecute()\n\u001b[0;32m     21\u001b[0m y \u001b[39m=\u001b[39m data[data\u001b[39m.\u001b[39mdtypes\u001b[39m.\u001b[39mindex[\u001b[39m4207\u001b[39m]]\n\u001b[0;32m     23\u001b[0m \u001b[39m# train test split\u001b[39;00m\n\nFile \u001b[1;32mc:\\Users\\Wilhelm\\AppData\\Local\\Programs\\Python\\Python310\\lib\\site-packages\\mars\\core\\entity\\tileables.py:462\u001b[0m, in \u001b[0;36mHasShapeTileable.execute\u001b[1;34m(self, session, **kw)\u001b[0m\n\u001b[0;32m    461\u001b[0m \u001b[39mdef\u001b[39;00m \u001b[39mexecute\u001b[39m(\u001b[39mself\u001b[39m, session\u001b[39m=\u001b[39m\u001b[39mNone\u001b[39;00m, \u001b[39m*\u001b[39m\u001b[39m*\u001b[39mkw):\n\u001b[1;32m--> 462\u001b[0m     result \u001b[39m=\u001b[39m \u001b[39mself\u001b[39m\u001b[39m.\u001b[39mdata\u001b[39m.\u001b[39mexecute(session\u001b[39m=\u001b[39msession, \u001b[39m*\u001b[39m\u001b[39m*\u001b[39mkw)\n\u001b[0;32m    463\u001b[0m     \u001b[39mif\u001b[39;00m \u001b[39misinstance\u001b[39m(result, TILEABLE_TYPE):\n\u001b[0;32m    464\u001b[0m         \u001b[39mreturn\u001b[39;00m \u001b[39mself\u001b[39m\n\nFile \u001b[1;32mc:\\Users\\Wilhelm\\AppData\\Local\\Programs\\Python\\Python310\\lib\\site-packages\\mars\\core\\entity\\executable.py:144\u001b[0m, in \u001b[0;36m_ExecutableMixin.execute\u001b[1;34m(self, session, **kw)\u001b[0m\n\u001b[0;32m    141\u001b[0m \u001b[39mfrom\u001b[39;00m \u001b[39m.\u001b[39;00m\u001b[39m.\u001b[39;00m\u001b[39m.\u001b[39;00m\u001b[39mdeploy\u001b[39;00m\u001b[39m.\u001b[39;00m\u001b[39moscar\u001b[39;00m\u001b[39m.\u001b[39;00m\u001b[39msession\u001b[39;00m \u001b[39mimport\u001b[39;00m execute\n\u001b[0;32m    143\u001b[0m session \u001b[39m=\u001b[39m _get_session(\u001b[39mself\u001b[39m, session)\n\u001b[1;32m--> 144\u001b[0m \u001b[39mreturn\u001b[39;00m execute(\u001b[39mself\u001b[39m, session\u001b[39m=\u001b[39msession, \u001b[39m*\u001b[39m\u001b[39m*\u001b[39mkw)\n\nFile \u001b[1;32mc:\\Users\\Wilhelm\\AppData\\Local\\Programs\\Python\\Python310\\lib\\site-packages\\mars\\deploy\\oscar\\session.py:1888\u001b[0m, in \u001b[0;36mexecute\u001b[1;34m(tileable, session, wait, new_session_kwargs, show_progress, progress_update_interval, *tileables, **kwargs)\u001b[0m\n\u001b[0;32m   1886\u001b[0m     tileable, tileables \u001b[39m=\u001b[39m tileable[\u001b[39m0\u001b[39m], tileable[\u001b[39m1\u001b[39m:]\n\u001b[0;32m   1887\u001b[0m \u001b[39mif\u001b[39;00m session \u001b[39mis\u001b[39;00m \u001b[39mNone\u001b[39;00m:\n\u001b[1;32m-> 1888\u001b[0m     session \u001b[39m=\u001b[39m get_default_or_create(\u001b[39m*\u001b[39m\u001b[39m*\u001b[39m(new_session_kwargs \u001b[39mor\u001b[39;00m \u001b[39mdict\u001b[39m()))\n\u001b[0;32m   1889\u001b[0m session \u001b[39m=\u001b[39m _ensure_sync(session)\n\u001b[0;32m   1890\u001b[0m \u001b[39mreturn\u001b[39;00m session\u001b[39m.\u001b[39mexecute(\n\u001b[0;32m   1891\u001b[0m     tileable,\n\u001b[0;32m   1892\u001b[0m     \u001b[39m*\u001b[39mtileables,\n\u001b[1;32m   (...)\u001b[0m\n\u001b[0;32m   1896\u001b[0m     \u001b[39m*\u001b[39m\u001b[39m*\u001b[39mkwargs,\n\u001b[0;32m   1897\u001b[0m )\n\nFile \u001b[1;32mc:\\Users\\Wilhelm\\AppData\\Local\\Programs\\Python\\Python310\\lib\\site-packages\\mars\\deploy\\oscar\\session.py:2065\u001b[0m, in \u001b[0;36mget_default_or_create\u001b[1;34m(**kwargs)\u001b[0m\n\u001b[0;32m   2062\u001b[0m     \u001b[39mif\u001b[39;00m session \u001b[39mis\u001b[39;00m \u001b[39mNone\u001b[39;00m:\n\u001b[0;32m   2063\u001b[0m         \u001b[39m# no session attached, try to create one\u001b[39;00m\n\u001b[0;32m   2064\u001b[0m         warnings\u001b[39m.\u001b[39mwarn(warning_msg)\n\u001b[1;32m-> 2065\u001b[0m         session \u001b[39m=\u001b[39m new_session(\u001b[39m\"\u001b[39m\u001b[39m127.0.0.1\u001b[39m\u001b[39m\"\u001b[39m, init_local\u001b[39m=\u001b[39m\u001b[39mTrue\u001b[39;00m, \u001b[39m*\u001b[39m\u001b[39m*\u001b[39mkwargs)\n\u001b[0;32m   2066\u001b[0m         session\u001b[39m.\u001b[39mas_default()\n\u001b[0;32m   2067\u001b[0m \u001b[39mif\u001b[39;00m \u001b[39misinstance\u001b[39m(session, _IsolatedSession):\n\nFile \u001b[1;32mc:\\Users\\Wilhelm\\AppData\\Local\\Programs\\Python\\Python310\\lib\\site-packages\\mars\\deploy\\oscar\\session.py:2039\u001b[0m, in \u001b[0;36mnew_session\u001b[1;34m(address, session_id, backend, default, new, **kwargs)\u001b[0m\n\u001b[0;32m   2036\u001b[0m \u001b[39mif\u001b[39;00m session_id \u001b[39mis\u001b[39;00m \u001b[39mNone\u001b[39;00m:\n\u001b[0;32m   2037\u001b[0m     session_id \u001b[39m=\u001b[39m _new_session_id()\n\u001b[1;32m-> 2039\u001b[0m session \u001b[39m=\u001b[39m SyncSession\u001b[39m.\u001b[39minit(\n\u001b[0;32m   2040\u001b[0m     address, session_id\u001b[39m=\u001b[39msession_id, backend\u001b[39m=\u001b[39mbackend, new\u001b[39m=\u001b[39mnew, \u001b[39m*\u001b[39m\u001b[39m*\u001b[39mkwargs\n\u001b[0;32m   2041\u001b[0m )\n\u001b[0;32m   2042\u001b[0m \u001b[39mif\u001b[39;00m default:\n\u001b[0;32m   2043\u001b[0m     session\u001b[39m.\u001b[39mas_default()\n\nFile \u001b[1;32mc:\\Users\\Wilhelm\\AppData\\Local\\Programs\\Python\\Python310\\lib\\site-packages\\mars\\deploy\\oscar\\session.py:1636\u001b[0m, in \u001b[0;36mSyncSession.init\u001b[1;34m(cls, address, session_id, backend, new, **kwargs)\u001b[0m\n\u001b[0;32m   1634\u001b[0m coro \u001b[39m=\u001b[39m _IsolatedSession\u001b[39m.\u001b[39minit(address, session_id, backend, new\u001b[39m=\u001b[39mnew, \u001b[39m*\u001b[39m\u001b[39m*\u001b[39mkwargs)\n\u001b[0;32m   1635\u001b[0m fut \u001b[39m=\u001b[39m asyncio\u001b[39m.\u001b[39mrun_coroutine_threadsafe(coro, isolation\u001b[39m.\u001b[39mloop)\n\u001b[1;32m-> 1636\u001b[0m isolated_session \u001b[39m=\u001b[39m fut\u001b[39m.\u001b[39;49mresult()\n\u001b[0;32m   1637\u001b[0m \u001b[39mreturn\u001b[39;00m SyncSession(address, session_id, isolated_session, isolation)\n\nFile \u001b[1;32mc:\\Users\\Wilhelm\\AppData\\Local\\Programs\\Python\\Python310\\lib\\concurrent\\futures\\_base.py:458\u001b[0m, in \u001b[0;36mFuture.result\u001b[1;34m(self, timeout)\u001b[0m\n\u001b[0;32m    456\u001b[0m     \u001b[39mraise\u001b[39;00m CancelledError()\n\u001b[0;32m    457\u001b[0m \u001b[39melif\u001b[39;00m \u001b[39mself\u001b[39m\u001b[39m.\u001b[39m_state \u001b[39m==\u001b[39m FINISHED:\n\u001b[1;32m--> 458\u001b[0m     \u001b[39mreturn\u001b[39;00m \u001b[39mself\u001b[39;49m\u001b[39m.\u001b[39;49m__get_result()\n\u001b[0;32m    459\u001b[0m \u001b[39melse\u001b[39;00m:\n\u001b[0;32m    460\u001b[0m     \u001b[39mraise\u001b[39;00m \u001b[39mTimeoutError\u001b[39;00m()\n\nFile \u001b[1;32mc:\\Users\\Wilhelm\\AppData\\Local\\Programs\\Python\\Python310\\lib\\concurrent\\futures\\_base.py:403\u001b[0m, in \u001b[0;36mFuture.__get_result\u001b[1;34m(self)\u001b[0m\n\u001b[0;32m    401\u001b[0m \u001b[39mif\u001b[39;00m \u001b[39mself\u001b[39m\u001b[39m.\u001b[39m_exception:\n\u001b[0;32m    402\u001b[0m     \u001b[39mtry\u001b[39;00m:\n\u001b[1;32m--> 403\u001b[0m         \u001b[39mraise\u001b[39;00m \u001b[39mself\u001b[39m\u001b[39m.\u001b[39m_exception\n\u001b[0;32m    404\u001b[0m     \u001b[39mfinally\u001b[39;00m:\n\u001b[0;32m    405\u001b[0m         \u001b[39m# Break a reference cycle with the exception in self._exception\u001b[39;00m\n\u001b[0;32m    406\u001b[0m         \u001b[39mself\u001b[39m \u001b[39m=\u001b[39m \u001b[39mNone\u001b[39;00m\n\nFile \u001b[1;32mc:\\Users\\Wilhelm\\AppData\\Local\\Programs\\Python\\Python310\\lib\\site-packages\\mars\\deploy\\oscar\\session.py:847\u001b[0m, in \u001b[0;36m_IsolatedSession.init\u001b[1;34m(cls, address, session_id, backend, new, timeout, **kwargs)\u001b[0m\n\u001b[0;32m    843\u001b[0m \u001b[39mif\u001b[39;00m init_local:\n\u001b[0;32m    844\u001b[0m     \u001b[39mfrom\u001b[39;00m \u001b[39m.\u001b[39;00m\u001b[39mlocal\u001b[39;00m \u001b[39mimport\u001b[39;00m new_cluster_in_isolation\n\u001b[0;32m    846\u001b[0m     \u001b[39mreturn\u001b[39;00m (\n\u001b[1;32m--> 847\u001b[0m         \u001b[39mawait\u001b[39;00m new_cluster_in_isolation(\n\u001b[0;32m    848\u001b[0m             address, timeout\u001b[39m=\u001b[39mtimeout, backend\u001b[39m=\u001b[39mbackend, \u001b[39m*\u001b[39m\u001b[39m*\u001b[39mkwargs\n\u001b[0;32m    849\u001b[0m         )\n\u001b[0;32m    850\u001b[0m     )\u001b[39m.\u001b[39msession\n\u001b[0;32m    852\u001b[0m \u001b[39mif\u001b[39;00m kwargs:  \u001b[39m# pragma: no cover\u001b[39;00m\n\u001b[0;32m    853\u001b[0m     unexpected_keys \u001b[39m=\u001b[39m \u001b[39m\"\u001b[39m\u001b[39m, \u001b[39m\u001b[39m\"\u001b[39m\u001b[39m.\u001b[39mjoin(\u001b[39mlist\u001b[39m(kwargs\u001b[39m.\u001b[39mkeys()))\n\nFile \u001b[1;32mc:\\Users\\Wilhelm\\AppData\\Local\\Programs\\Python\\Python310\\lib\\site-packages\\mars\\deploy\\oscar\\local.py:79\u001b[0m, in \u001b[0;36mnew_cluster_in_isolation\u001b[1;34m(address, n_worker, n_cpu, mem_bytes, cuda_devices, subprocess_start_method, backend, config, web, timeout, n_supervisor_process)\u001b[0m\n\u001b[0;32m     66\u001b[0m \u001b[39masync\u001b[39;00m \u001b[39mdef\u001b[39;00m \u001b[39mnew_cluster_in_isolation\u001b[39m(\n\u001b[0;32m     67\u001b[0m     address: \u001b[39mstr\u001b[39m \u001b[39m=\u001b[39m \u001b[39m\"\u001b[39m\u001b[39m0.0.0.0\u001b[39m\u001b[39m\"\u001b[39m,\n\u001b[0;32m     68\u001b[0m     n_worker: \u001b[39mint\u001b[39m \u001b[39m=\u001b[39m \u001b[39m1\u001b[39m,\n\u001b[1;32m   (...)\u001b[0m\n\u001b[0;32m     77\u001b[0m     n_supervisor_process: \u001b[39mint\u001b[39m \u001b[39m=\u001b[39m \u001b[39m0\u001b[39m,\n\u001b[0;32m     78\u001b[0m ) \u001b[39m-\u001b[39m\u001b[39m>\u001b[39m ClientType:\n\u001b[1;32m---> 79\u001b[0m     cluster \u001b[39m=\u001b[39m LocalCluster(\n\u001b[0;32m     80\u001b[0m         address,\n\u001b[0;32m     81\u001b[0m         n_worker,\n\u001b[0;32m     82\u001b[0m         n_cpu,\n\u001b[0;32m     83\u001b[0m         mem_bytes,\n\u001b[0;32m     84\u001b[0m         cuda_devices,\n\u001b[0;32m     85\u001b[0m         subprocess_start_method,\n\u001b[0;32m     86\u001b[0m         backend,\n\u001b[0;32m     87\u001b[0m         config,\n\u001b[0;32m     88\u001b[0m         web,\n\u001b[0;32m     89\u001b[0m         n_supervisor_process,\n\u001b[0;32m     90\u001b[0m     )\n\u001b[0;32m     91\u001b[0m     \u001b[39mawait\u001b[39;00m cluster\u001b[39m.\u001b[39mstart()\n\u001b[0;32m     92\u001b[0m     \u001b[39mreturn\u001b[39;00m \u001b[39mawait\u001b[39;00m LocalClient\u001b[39m.\u001b[39mcreate(cluster, timeout)\n\nFile \u001b[1;32mc:\\Users\\Wilhelm\\AppData\\Local\\Programs\\Python\\Python310\\lib\\site-packages\\mars\\deploy\\oscar\\local.py:160\u001b[0m, in \u001b[0;36mLocalCluster.__init__\u001b[1;34m(self, address, n_worker, n_cpu, mem_bytes, cuda_devices, subprocess_start_method, backend, config, web, n_supervisor_process)\u001b[0m\n\u001b[0;32m    158\u001b[0m \u001b[39mself\u001b[39m\u001b[39m.\u001b[39m_n_cpu \u001b[39m=\u001b[39m cpu_count() \u001b[39mif\u001b[39;00m n_cpu \u001b[39m==\u001b[39m \u001b[39m\"\u001b[39m\u001b[39mauto\u001b[39m\u001b[39m\"\u001b[39m \u001b[39melse\u001b[39;00m n_cpu\n\u001b[0;32m    159\u001b[0m \u001b[39mself\u001b[39m\u001b[39m.\u001b[39m_mem_bytes \u001b[39m=\u001b[39m mem_total() \u001b[39mif\u001b[39;00m mem_bytes \u001b[39m==\u001b[39m \u001b[39m\"\u001b[39m\u001b[39mauto\u001b[39m\u001b[39m\"\u001b[39m \u001b[39melse\u001b[39;00m mem_bytes\n\u001b[1;32m--> 160\u001b[0m \u001b[39mself\u001b[39m\u001b[39m.\u001b[39m_cuda_devices \u001b[39m=\u001b[39m \u001b[39mself\u001b[39;49m\u001b[39m.\u001b[39;49m_get_cuda_devices(cuda_devices, n_worker)\n\u001b[0;32m    161\u001b[0m \u001b[39mself\u001b[39m\u001b[39m.\u001b[39m_subprocess_start_method \u001b[39m=\u001b[39m subprocess_start_method\n\u001b[0;32m    162\u001b[0m \u001b[39mself\u001b[39m\u001b[39m.\u001b[39m_config \u001b[39m=\u001b[39m load_config(config, default_config_file\u001b[39m=\u001b[39mDEFAULT_CONFIG_FILE)\n\nFile \u001b[1;32mc:\\Users\\Wilhelm\\AppData\\Local\\Programs\\Python\\Python310\\lib\\site-packages\\mars\\deploy\\oscar\\local.py:195\u001b[0m, in \u001b[0;36mLocalCluster._get_cuda_devices\u001b[1;34m(cuda_devices, n_worker)\u001b[0m\n\u001b[0;32m    192\u001b[0m \u001b[39m@staticmethod\u001b[39m\n\u001b[0;32m    193\u001b[0m \u001b[39mdef\u001b[39;00m \u001b[39m_get_cuda_devices\u001b[39m(cuda_devices, n_worker):\n\u001b[0;32m    194\u001b[0m     \u001b[39mif\u001b[39;00m cuda_devices \u001b[39m==\u001b[39m \u001b[39m\"\u001b[39m\u001b[39mauto\u001b[39m\u001b[39m\"\u001b[39m:\n\u001b[1;32m--> 195\u001b[0m         total \u001b[39m=\u001b[39m cuda_count()\n\u001b[0;32m    196\u001b[0m         all_devices \u001b[39m=\u001b[39m np\u001b[39m.\u001b[39marange(total)\n\u001b[0;32m    197\u001b[0m         \u001b[39mreturn\u001b[39;00m [\u001b[39mlist\u001b[39m(arr) \u001b[39mfor\u001b[39;00m arr \u001b[39min\u001b[39;00m np\u001b[39m.\u001b[39marray_split(all_devices, n_worker)]\n\nFile \u001b[1;32mc:\\Users\\Wilhelm\\AppData\\Local\\Programs\\Python\\Python310\\lib\\site-packages\\mars\\resource.py:366\u001b[0m, in \u001b[0;36mcuda_count\u001b[1;34m()\u001b[0m\n\u001b[0;32m    365\u001b[0m \u001b[39mdef\u001b[39;00m \u001b[39mcuda_count\u001b[39m():\n\u001b[1;32m--> 366\u001b[0m     \u001b[39mreturn\u001b[39;00m nvutils\u001b[39m.\u001b[39;49mget_device_count() \u001b[39mor\u001b[39;00m \u001b[39m0\u001b[39m\n\nFile \u001b[1;32mc:\\Users\\Wilhelm\\AppData\\Local\\Programs\\Python\\Python310\\lib\\site-packages\\mars\\lib\\nvutils.py:256\u001b[0m, in \u001b[0;36mget_device_count\u001b[1;34m()\u001b[0m\n\u001b[0;32m    253\u001b[0m \u001b[39mif\u001b[39;00m _gpu_count \u001b[39mis\u001b[39;00m \u001b[39mnot\u001b[39;00m \u001b[39mNone\u001b[39;00m:\n\u001b[0;32m    254\u001b[0m     \u001b[39mreturn\u001b[39;00m _gpu_count\n\u001b[1;32m--> 256\u001b[0m _init_nvml()\n\u001b[0;32m    257\u001b[0m \u001b[39mif\u001b[39;00m _nvml_lib \u001b[39mis\u001b[39;00m \u001b[39mNone\u001b[39;00m:\n\u001b[0;32m    258\u001b[0m     \u001b[39mreturn\u001b[39;00m \u001b[39mNone\u001b[39;00m\n\nFile \u001b[1;32mc:\\Users\\Wilhelm\\AppData\\Local\\Programs\\Python\\Python310\\lib\\site-packages\\mars\\lib\\nvutils.py:225\u001b[0m, in \u001b[0;36m_init_nvml\u001b[1;34m()\u001b[0m\n\u001b[0;32m    223\u001b[0m     \u001b[39mreturn\u001b[39;00m\n\u001b[0;32m    224\u001b[0m \u001b[39mtry\u001b[39;00m:\n\u001b[1;32m--> 225\u001b[0m     _nvml_check_error(_nvml_lib\u001b[39m.\u001b[39;49mnvmlInit_v2())\n\u001b[0;32m    226\u001b[0m \u001b[39mexcept\u001b[39;00m NVMLAPIError \u001b[39mas\u001b[39;00m ex:\n\u001b[0;32m    227\u001b[0m     \u001b[39mif\u001b[39;00m ex\u001b[39m.\u001b[39merrno \u001b[39m==\u001b[39m NVML_DRIVER_NOT_LOADED:\n\nFile \u001b[1;32mc:\\Users\\Wilhelm\\AppData\\Local\\Programs\\Python\\Python310\\lib\\ctypes\\__init__.py:387\u001b[0m, in \u001b[0;36mCDLL.__getattr__\u001b[1;34m(self, name)\u001b[0m\n\u001b[0;32m    385\u001b[0m \u001b[39mif\u001b[39;00m name\u001b[39m.\u001b[39mstartswith(\u001b[39m'\u001b[39m\u001b[39m__\u001b[39m\u001b[39m'\u001b[39m) \u001b[39mand\u001b[39;00m name\u001b[39m.\u001b[39mendswith(\u001b[39m'\u001b[39m\u001b[39m__\u001b[39m\u001b[39m'\u001b[39m):\n\u001b[0;32m    386\u001b[0m     \u001b[39mraise\u001b[39;00m \u001b[39mAttributeError\u001b[39;00m(name)\n\u001b[1;32m--> 387\u001b[0m func \u001b[39m=\u001b[39m \u001b[39mself\u001b[39;49m\u001b[39m.\u001b[39;49m\u001b[39m__getitem__\u001b[39;49m(name)\n\u001b[0;32m    388\u001b[0m \u001b[39msetattr\u001b[39m(\u001b[39mself\u001b[39m, name, func)\n\u001b[0;32m    389\u001b[0m \u001b[39mreturn\u001b[39;00m func\n\nFile \u001b[1;32mc:\\Users\\Wilhelm\\AppData\\Local\\Programs\\Python\\Python310\\lib\\ctypes\\__init__.py:392\u001b[0m, in \u001b[0;36mCDLL.__getitem__\u001b[1;34m(self, name_or_ordinal)\u001b[0m\n\u001b[0;32m    391\u001b[0m \u001b[39mdef\u001b[39;00m \u001b[39m__getitem__\u001b[39m(\u001b[39mself\u001b[39m, name_or_ordinal):\n\u001b[1;32m--> 392\u001b[0m     func \u001b[39m=\u001b[39m \u001b[39mself\u001b[39;49m\u001b[39m.\u001b[39;49m_FuncPtr((name_or_ordinal, \u001b[39mself\u001b[39;49m))\n\u001b[0;32m    393\u001b[0m     \u001b[39mif\u001b[39;00m \u001b[39mnot\u001b[39;00m \u001b[39misinstance\u001b[39m(name_or_ordinal, \u001b[39mint\u001b[39m):\n\u001b[0;32m    394\u001b[0m         func\u001b[39m.\u001b[39m\u001b[39m__name__\u001b[39m \u001b[39m=\u001b[39m name_or_ordinal\n\n\u001b[1;31mAttributeError\u001b[0m: function 'nvmlInit_v2' not found"
}