Open SjeYinTeoIntel opened 10 months ago
Hi, @SjeYinTeoIntel , could you provide the bigdl-chronos
and psutil
version you used?
We have fixed similar errors in https://github.com/intel-analytics/BigDL/pull/9208. Please try to run pip install --pre --upgrade bigdl-chronos[pytorch, distributed]
.
Hi, below is my library installed using Dockerfile.
RUN apt-get update && \ apt-get install --no-install-recommends python-setuptools python-dev python3 libpq-dev gcc musl-dev postgresql-server-dev-all curl unzip -y && \ python3.9 -m pip install --upgrade pip && \ python3.9 -m pip install psycopg2 --no-cache-dir && \ python3.9 -m pip install python-dateutil && \ python3.9 -m pip install boto3 && \ python3.9 -m pip install sparknlp && \ python3.9 -m pip install pyarrow==9.0.0 && \ python3.9 -m pip install pystan==3.0.0 && \ python3.9 -m pip install prophet==1.1.3 && \ python3.9 -m pip install pyldavis && \ python3.9 -m pip install ydata_profiling==4.0.0 && \ python3.9 -m pip install pandas==1.4.4 && \ python3.9 -m pip install numpy==1.24.2 && \ python3.9 -m pip install unidecode==1.3.6 && \ python3.9 -m pip install matplotlib==3.7.0 && \ python3.9 -m pip install bigdl-spark3==2.4.0b20230912 && \ python3.9 -m pip install ray[default]==1.9.2 && \ python3.9 -m pip install protobuf && \ python3.9 -m pip install --ignore-installed PyYAML --pre --upgrade bigdl-chronos[pytorch]==2.4.0b20230912 && \ python3.9 -m pip install kmodes && \ python3.9 -m pip install tabulate && \ rm -r /root/.cache && \ rm -rf /var/lib/apt/lists/*
I updated to bigdl-chronos[pytorch, distributed] as ur mentioned. But still facing the same issue.
RUN apt-get update && \ apt-get install --no-install-recommends python-setuptools python-dev python3 libpq-dev gcc musl-dev postgresql-server-dev-all curl unzip -y && \ python3.9 -m pip install --upgrade pip && \ python3.9 -m pip install psycopg2 --no-cache-dir && \ python3.9 -m pip install python-dateutil && \ python3.9 -m pip install boto3 && \ python3.9 -m pip install sparknlp && \ python3.9 -m pip install pyarrow==9.0.0 && \ python3.9 -m pip install pystan==3.0.0 && \ python3.9 -m pip install prophet==1.1.3 && \ python3.9 -m pip install pyldavis && \ python3.9 -m pip install ydata_profiling==4.0.0 && \ python3.9 -m pip install pandas==1.4.4 && \ python3.9 -m pip install numpy==1.24.2 && \ python3.9 -m pip install unidecode==1.3.6 && \ python3.9 -m pip install matplotlib==3.7.0 && \ python3.9 -m pip install bigdl-spark3==2.4.0b20230912 && \ python3.9 -m pip install ray[default]==1.9.2 && \ python3.9 -m pip install protobuf && \ python3.9 -m pip install --ignore-installed PyYAML --pre --upgrade bigdl-chronos[pytorch, distributed]==2.4.0b20230912 && \ python3.9 -m pip install kmodes && \ python3.9 -m pip install tabulate && \ rm -r /root/.cache && \ rm -rf /var/lib/apt/lists/*
Regards, sjeyin
From: binbin Deng @.> Sent: Thursday, October 19, 2023 2:38 PM To: intel-analytics/BigDL @.> Cc: Teo, Sje Yin @.>; Mention @.> Subject: Re: [intel-analytics/BigDL] RuntimeError: List process with list2cmdline failed! (Issue #9218)
Hi, @SjeYinTeoIntelhttps://github.com/SjeYinTeoIntel , could you provide the bigdl-chronos and psutil version you used?
We have fixed similar errors in #9208https://github.com/intel-analytics/BigDL/pull/9208. Please try to run pip install --pre --upgrade bigdl-chronos[pytorch, distributed].
— Reply to this email directly, view it on GitHubhttps://github.com/intel-analytics/BigDL/issues/9218#issuecomment-1770155373, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ASXCPFY3AK62HHNQB7RGXS3YADDFFAVCNFSM6AAAAAA6GODRICVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTONZQGE2TKMZXGM. You are receiving this because you were mentioned.Message ID: @.**@.>>
Hi, @SjeYinTeoIntel , according to the command provided by you, I think you need to use python3.9 -m pip install --ignore-installed PyYAML --pre --upgrade bigdl-chronos[pytorch,distributed]
instead of python3.9 -m pip install --ignore-installed PyYAML --pre --upgrade bigdl-chronos[pytorch, distributed]==2.4.0b20230912
. Otherwise, you still use the version 2.4.0b20230912
because you specify==2.4.0b20230912
.
WARNING:root:Some values of column volume exceeds the mean plus/minus 10 times standard deviation, please call .repair_abnormal_data() to remove abnormal values. ds y volume high low open id 0 2013-02-08 00:00:00.000 14.75 8407500.0 15.12 14.63 15.07 0 1 2013-02-09 10:49:36.152 14.46 8882000.0 15.01 14.26 14.89 0 2 2013-02-10 21:39:12.304 14.27 8126000.0 14.51 14.10 14.45 0 3 2013-02-12 08:28:48.456 14.66 10259500.0 14.94 14.25 14.30 0 4 2013-02-13 19:18:24.608 13.99 31879900.0 14.96 13.16 14.94 0 Initializing orca context WARNING: sun.reflect.Reflection.getCallerClass is not supported. This will impact performance. 2023-10-18 09:44:28,023 Thread-4 WARN The bufferSize is set to 4000 but bufferedIo is false: false 2023-10-18 09:44:28,026 Thread-4 WARN The bufferSize is set to 4000 but bufferedIo is false: false 2023-10-18 09:44:28,026 Thread-4 WARN The bufferSize is set to 4000 but bufferedIo is false: false 2023-10-18 09:44:28,027 Thread-4 WARN The bufferSize is set to 4000 but bufferedIo is false: false 23-10-18 09:44:28 [Thread-4] INFO Engine$:122 - Auto detect executor number and executor cores number 23-10-18 09:44:28 [Thread-4] INFO Engine$:124 - Executor number is 1 and executor cores number is 6 23-10-18 09:44:28 [Thread-4] INFO ThreadPool$:95 - Set mkl threads to 1 on thread 19 23/10/18 09:44:28 WARN SparkContext: Using an existing SparkContext; some configuration may not take effect. 23-10-18 09:44:28 [Thread-4] INFO Engine$:461 - Find existing spark context. Checking the spark conf... 23-10-18 09:44:28 [Thread-4] WARN Engine$:467 - Engine.init: Can not find spark.shuffle.reduceLocality.enabled. For details please check https://bigdl-project.github.io/master/#APIGuide/Engine/ 23-10-18 09:44:28 [Thread-4] WARN Engine$:467 - Engine.init: Can not find spark.shuffle.blockTransferService. For details please check https://bigdl-project.github.io/master/#APIGuide/Engine/ 23-10-18 09:44:28 [Thread-4] WARN Engine$:467 - Engine.init: Can not find spark.scheduler.minRegisteredResourcesRatio. For details please check https://bigdl-project.github.io/master/#APIGuide/Engine/ 23-10-18 09:44:28 [Thread-4] WARN Engine$:467 - Engine.init: Can not find spark.scheduler.maxRegisteredResourcesWaitingTime. For details please check https://bigdl-project.github.io/master/#APIGuide/Engine/ 23-10-18 09:44:28 [Thread-4] WARN Engine$:467 - Engine.init: Can not find spark.speculation. For details please check https://bigdl-project.github.io/master/#APIGuide/Engine/ 23-10-18 09:44:28 [Thread-4] WARN Engine$:467 - Engine.init: Can not find spark.serializer. For details please check https://bigdl-project.github.io/master/#APIGuide/Engine/ 23-10-18 09:44:28 [Thread-4] WARN Engine$:470 - Engine.init: spark.driver.extraJavaOptions should be -Dlog4j2.info, but it is -Dcom.amazonaws.services.s3.enableV4=true. For details please check https://bigdl-project.github.io/master/#APIGuide/Engine/ cls.getname: com.intel.analytics.bigdl.dllib.utils.python.api.Sample BigDLBasePickler registering: bigdl.dllib.utils.common Sample cls.getname: com.intel.analytics.bigdl.dllib.utils.python.api.EvaluatedResult BigDLBasePickler registering: bigdl.dllib.utils.common EvaluatedResult cls.getname: com.intel.analytics.bigdl.dllib.utils.python.api.JTensor BigDLBasePickler registering: bigdl.dllib.utils.common JTensor cls.getname: com.intel.analytics.bigdl.dllib.utils.python.api.JActivity BigDLBasePickler registering: bigdl.dllib.utils.common JActivity Done init_orca_context Launching Ray on cluster with Spark barrier mode Start to launch ray driver Executing command: ray start --address 10.0.226.35:62449 --num-cpus 0 --node-ip-address 10.0.255.50 2023-10-18 09:44:43,021 INFO scripts.py:747 -- Local node IP: 10.0.255.50 2023-10-18 09:44:43,125 SUCC scripts.py:755 -- -------------------- 2023-10-18 09:44:43,125 SUCC scripts.py:756 -- Ray runtime started. 2023-10-18 09:44:43,125 SUCC scripts.py:757 -- -------------------- 2023-10-18 09:44:43,125 INFO scripts.py:759 -- To terminate the Ray runtime, run 2023-10-18 09:44:43,125 INFO scripts.py:760 -- ray stop
2023-10-18 09:44:43,036 WARNING services.py:1816 -- WARNING: The object store is using /tmp instead of /dev/shm because /dev/shm has only 67108864 bytes available. This will harm performance! You may be able to free up space by deleting files in /dev/shm. If you are inside a Docker container, you can increase /dev/shm size by passing '--shm-size=0.92gb' to 'docker run' (or add it to the run_options list in a Ray cluster config). Make sure to set this to more than 30% of available RAM. [2023-10-18 09:44:43,124 I 205 205] global_state_accessor.cc:360: This node has an IP address of 10.0.255.50, while we can not found the matched Raylet address. This maybe come from when you connect the Ray cluster with a different IP address or connect a container.
2023-10-18 09:44:43,021 INFO scripts.py:747 -- Local node IP: 10.0.255.50 2023-10-18 09:44:43,125 SUCC scripts.py:755 -- -------------------- 2023-10-18 09:44:43,125 SUCC scripts.py:756 -- Ray runtime started. 2023-10-18 09:44:43,125 SUCC scripts.py:757 -- -------------------- 2023-10-18 09:44:43,125 INFO scripts.py:759 -- To terminate the Ray runtime, run 2023-10-18 09:44:43,125 INFO scripts.py:760 -- ray stop
File "/usr/local/lib/python3.9/dist-packages/bigdl/orca/ray/ray_daemon.py", line 26 logging.info(f"Stopping pgid {pgid} by ray_daemon.") ^ SyntaxError: invalid syntax 2023-10-18 09:44:44,277 INFO worker.py:842 -- Connecting to existing Ray cluster at address: 10.0.226.35:62449 {'node_ip_address': '10.0.255.50', 'raylet_ip_address': '10.0.255.50', 'redis_address': '10.0.226.35:62449', 'object_store_address': '/tmp/ray/session_2023-10-18_09-44-32_155648_187/sockets/plasma_store', 'raylet_socket_name': '/tmp/ray/session_2023-10-18_09-44-32_155648_187/sockets/raylet', 'webui_url': '10.0.226.35:8265', 'session_dir': '/tmp/ray/session_2023-10-18_09-44-32_155648_187', 'metrics_export_port': 54656, 'node_id': '63d64a2f0bf4080aa38cb1a38eace36f89d1121623f6a8a3d7dc219e'} ERROR:bigdl.dllib.utils.log4Error:
****Usage Error**** List process with list2cmdline failed! ERROR:bigdl.dllib.utils.log4Error:
****Call Stack*** 2023-10-18 09:44:44,421 - DataTransformation - MainThread - ERROR - Exception in processing job: 284TC20-_AutoProphet_autoprophet1_5MXI4YR Exception: List process with list2cmdline failed! Traceback (most recent call last): File "/usr/local/lib/python3.9/dist-packages/bigdl/orca/ray/ray_on_spark_context.py", line 66, in kill_redundant_log_monitors cmdline = subprocess.list2cmdline(proc.cmdline()) File "/usr/local/lib/python3.9/dist-packages/psutil/init.py", line 702, in cmdline return self._proc.cmdline() File "/usr/local/lib/python3.9/dist-packages/psutil/_pslinux.py", line 1650, in wrapper return fun(self, *args, kwargs) File "/usr/local/lib/python3.9/dist-packages/psutil/_pslinux.py", line 1788, in cmdline self._raise_if_zombie() File "/usr/local/lib/python3.9/dist-packages/psutil/_pslinux.py", line 1693, in _raise_if_zombie raise ZombieProcess(self.pid, self._name, self._ppid) psutil.ZombieProcess: PID still exists but it's a zombie (pid=311, name='python')
During handling of the above exception, another exception occurred:
Traceback (most recent call last): File "/opt/easydata-app/python/operation/data_transformation.py", line 1349, in run File "/opt/easydata-app/python/transformation_analytics/ml_model_train_test.py", line 1086, in AutoProphet_Forecaster File "/usr/local/lib/python3.9/dist-packages/bigdl/chronos/autots/model/auto_prophet.py", line 112, in init self.auto_est = AutoEstimator(model_builder=model_builder, File "/usr/local/lib/python3.9/dist-packages/bigdl/orca/automl/auto_estimator.py", line 53, in init self.searcher = SearchEngineFactory.create_engine( File "/usr/local/lib/python3.9/dist-packages/bigdl/orca/automl/search/init.py", line 25, in create_engine return RayTuneSearchEngine(*args, *kwargs) File "/usr/local/lib/python3.9/dist-packages/bigdl/orca/automl/search/ray_tune/ray_tune_search_engine.py", line 53, in init self.remote_dir = remote_dir or RayTuneSearchEngine.get_default_remote_dir(name) File "/usr/local/lib/python3.9/dist-packages/bigdl/orca/automl/search/ray_tune/ray_tune_search_engine.py", line 60, in get_default_remote_dir ray_ctx = OrcaRayContext.get() File "/usr/local/lib/python3.9/dist-packages/bigdl/orca/ray/raycontext.py", line 103, in get ray_ctx.init() File "/usr/local/lib/python3.9/dist-packages/bigdl/orca/ray/raycontext.py", line 77, in init results = self._ray_on_spark_context.init(driver_cores=driver_cores) File "/usr/local/lib/python3.9/dist-packages/bigdl/orca/ray/ray_on_spark_context.py", line 605, in init kill_redundant_log_monitors(self._address_info["redis_address"]) # type: ignore File "/usr/local/lib/python3.9/dist-packages/bigdl/orca/ray/ray_on_spark_context.py", line 77, in kill_redundant_log_monitors invalidInputError(False, "List process with list2cmdline failed!") File "/usr/local/lib/python3.9/dist-packages/bigdl/dllib/utils/log4Error.py", line 33, in invalidInputError raise RuntimeError(errMsg) RuntimeError: List process with list2cmdline failed! ERROR:DataTransformation:Exception in processing job: 284TC20-_AutoProphet_autoprophet1_5MXI4YR Exception: List process with list2cmdline failed! Traceback (most recent call last): File "/usr/local/lib/python3.9/dist-packages/bigdl/orca/ray/ray_on_spark_context.py", line 66, in kill_redundant_log_monitors cmdline = subprocess.list2cmdline(proc.cmdline()) File "/usr/local/lib/python3.9/dist-packages/psutil/init.py", line 702, in cmdline return self._proc.cmdline() File "/usr/local/lib/python3.9/dist-packages/psutil/_pslinux.py", line 1650, in wrapper return fun(self, args, **kwargs) File "/usr/local/lib/python3.9/dist-packages/psutil/_pslinux.py", line 1788, in cmdline self._raise_if_zombie() File "/usr/local/lib/python3.9/dist-packages/psutil/_pslinux.py", line 1693, in _raise_if_zombie raise ZombieProcess(self.pid, self._name, self._ppid) psutil.ZombieProcess: PID still exists but it's a zombie (pid=311, name='python')
During handling of the above exception, another exception occurred:
Traceback (most recent call last): File "/opt/easydata-app/python/operation/data_transformation.py", line 1349, in run File "/opt/easydata-app/python/transformation_analytics/ml_model_train_test.py", line 1086, in AutoProphet_Forecaster File "/usr/local/lib/python3.9/dist-packages/bigdl/chronos/autots/model/auto_prophet.py", line 112, in init self.auto_est = AutoEstimator(model_builder=model_builder, File "/usr/local/lib/python3.9/dist-packages/bigdl/orca/automl/auto_estimator.py", line 53, in init self.searcher = SearchEngineFactory.create_engine( File "/usr/local/lib/python3.9/dist-packages/bigdl/orca/automl/search/init.py", line 25, in create_engine return RayTuneSearchEngine(*args, **kwargs) File "/usr/local/lib/python3.9/dist-packages/bigdl/orca/automl/search/ray_tune/ray_tune_search_engine.py", line 53, in init self.remote_dir = remote_dir or RayTuneSearchEngine.get_default_remote_dir(name) File "/usr/local/lib/python3.9/dist-packages/bigdl/orca/automl/search/ray_tune/ray_tune_search_engine.py", line 60, in get_default_remote_dir ray_ctx = OrcaRayContext.get() File "/usr/local/lib/python3.9/dist-packages/bigdl/orca/ray/raycontext.py", line 103, in get ray_ctx.init() File "/usr/local/lib/python3.9/dist-packages/bigdl/orca/ray/raycontext.py", line 77, in init results = self._ray_on_spark_context.init(driver_cores=driver_cores) File "/usr/local/lib/python3.9/dist-packages/bigdl/orca/ray/ray_on_spark_context.py", line 605, in init kill_redundant_log_monitors(self._address_info["redis_address"]) # type: ignore File "/usr/local/lib/python3.9/dist-packages/bigdl/orca/ray/ray_on_spark_context.py", line 77, in kill_redundant_log_monitors invalidInputError(False, "List process with list2cmdline failed!") File "/usr/local/lib/python3.9/dist-packages/bigdl/dllib/utils/log4Error.py", line 33, in invalidInputError raise RuntimeError(errMsg) RuntimeError: List process with list2cmdline failed! 2023-10-18 09:44:44,422 - DataTransformation - MainThread - INFO - Spark Session is stopped INFO:DataTransformation:Spark Session is stopped 23/10/18 09:44:44 WARN ExecutorPodsWatchSnapshotSource: Kubernetes client has been closed. (raylet, ip=10.0.226.35) /usr/local/lib/python3.9/dist-packages/ray/dashboard/agent.py:152: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead. (raylet, ip=10.0.226.35) if LooseVersion(aiohttp.version) < LooseVersion("4.0.0"): 2023-10-18 09:44:44,902 ERROR import_thread.py:89 -- ImportThread: Connection closed by server. 2023-10-18 09:44:44,911 ERROR worker.py:478 -- print_logs: Connection closed by server. 2023-10-18 09:44:44,911 ERROR worker.py:1247 -- listen_error_messages_raylet: Connection closed by server. Stopping ray_orca context