ERROR services.py:1129 -- --include-dashboard is not supported when minimal ray is used.Download ray[default] to use the dashboard.

SjeYinTeoIntel commented 9 months ago

I'm using this spark version with python3.9 -m pip install --pre --upgrade bigdl-spark3. saw the bigdl spark version is updated ... and causing below issue. Please help to check. Thanks =)

Initializing orca context WARNING: sun.reflect.Reflection.getCallerClass is not supported. This will impact performance. 2023-09-21 16:35:56,120 Thread-4 WARN The bufferSize is set to 4000 but bufferedIo is false: false 2023-09-21 16:35:56,123 Thread-4 WARN The bufferSize is set to 4000 but bufferedIo is false: false 2023-09-21 16:35:56,124 Thread-4 WARN The bufferSize is set to 4000 but bufferedIo is false: false 2023-09-21 16:35:56,124 Thread-4 WARN The bufferSize is set to 4000 but bufferedIo is false: false 23-09-21 16:35:56 [Thread-4] INFO Engine$:122 - Auto detect executor number and executor cores number 23-09-21 16:35:56 [Thread-4] INFO Engine$:124 - Executor number is 1 and executor cores number is 6 23-09-21 16:35:56 [Thread-4] INFO ThreadPool$:95 - Set mkl threads to 1 on thread 19 23/09/21 16:35:56 WARN SparkContext: Using an existing SparkContext; some configuration may not take effect. 23-09-21 16:35:56 [Thread-4] INFO Engine$:461 - Find existing spark context. Checking the spark conf... 23-09-21 16:35:56 [Thread-4] WARN Engine$:467 - Engine.init: Can not find spark.shuffle.reduceLocality.enabled. For details please check https://bigdl-project.github.io/master/#APIGuide/Engine/ 23-09-21 16:35:56 [Thread-4] WARN Engine$:467 - Engine.init: Can not find spark.shuffle.blockTransferService. For details please check https://bigdl-project.github.io/master/#APIGuide/Engine/ 23-09-21 16:35:56 [Thread-4] WARN Engine$:467 - Engine.init: Can not find spark.scheduler.minRegisteredResourcesRatio. For details please check https://bigdl-project.github.io/master/#APIGuide/Engine/ 23-09-21 16:35:56 [Thread-4] WARN Engine$:467 - Engine.init: Can not find spark.scheduler.maxRegisteredResourcesWaitingTime. For details please check https://bigdl-project.github.io/master/#APIGuide/Engine/ 23-09-21 16:35:56 [Thread-4] WARN Engine$:467 - Engine.init: Can not find spark.speculation. For details please check https://bigdl-project.github.io/master/#APIGuide/Engine/ 23-09-21 16:35:56 [Thread-4] WARN Engine$:467 - Engine.init: Can not find spark.serializer. For details please check https://bigdl-project.github.io/master/#APIGuide/Engine/ 23-09-21 16:35:56 [Thread-4] WARN Engine$:470 - Engine.init: spark.driver.extraJavaOptions should be -Dlog4j2.info, but it is -Dcom.amazonaws.services.s3.enableV4=true. For details please check https://bigdl-project.github.io/master/#APIGuide/Engine/ cls.getname: com.intel.analytics.bigdl.dllib.utils.python.api.Sample BigDLBasePickler registering: bigdl.dllib.utils.common Sample cls.getname: com.intel.analytics.bigdl.dllib.utils.python.api.EvaluatedResult BigDLBasePickler registering: bigdl.dllib.utils.common EvaluatedResult cls.getname: com.intel.analytics.bigdl.dllib.utils.python.api.JTensor BigDLBasePickler registering: bigdl.dllib.utils.common JTensor cls.getname: com.intel.analytics.bigdl.dllib.utils.python.api.JActivity BigDLBasePickler registering: bigdl.dllib.utils.common JActivity Launching Ray on cluster with Spark barrier mode ERROR:bigdl.dllib.utils.log4Error:

****Usage Error**** node_ip: 172.31.213.61 tag: ray-master, pgid: 156, pids: [157, 189, 190], returncode: 1, master_addr: 172.31.213.61:63679,
2023-09-21 16:35:59,841 INFO usage_lib.py:416 -- Usage stats collection is enabled by default without user confirmation because this terminal is detected to be non-interactive. To disable this, add --disable-usage-stats to the command that starts the cluster, or run the following command: ray disable-usage-stats before starting the cluster. See https://docs.ray.io/en/master/cluster/usage-stats.html for more details. 2023-09-21 16:35:59,842 INFO scripts.py:736 -- Local node IP: 172.31.213.61 2023-09-21 16:36:00,864 ERROR services.py:1129 -- --include-dashboard is not supported when minimal ray is used.Download ray[default] to use the dashboard. Traceback (most recent call last): File "/usr/local/bin/ray", line 8, in sys.exit(main()) File "/usr/local/lib/python3.9/dist-packages/ray/scripts/scripts.py", line 2490, in main return cli() File "/usr/local/lib/python3.9/dist-packages/click/core.py", line 1157, in call return self.main(args, kwargs) File "/usr/local/lib/python3.9/dist-packages/click/core.py", line 1078, in main rv = self.invoke(ctx) File "/usr/local/lib/python3.9/dist-packages/click/core.py", line 1688, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/usr/local/lib/python3.9/dist-packages/click/core.py", line 1434, in invoke return ctx.invoke(self.callback, ctx.params) File "/usr/local/lib/python3.9/dist-packages/click/core.py", line 783, in invoke return __callback(args, *kwargs) File "/usr/local/lib/python3.9/dist-packages/ray/autoscaler/_private/cli_logger.py", line 856, in wrapper return f(args, **kwargs) File "/usr/local/lib/python3.9/dist-packages/ray/scripts/scripts.py", line 763, in start node = ray._private.node.Node( File "/usr/local/lib/python3.9/dist-packages/ray/_private/node.py", line 307, in init self.start_head_processes() File "/usr/local/lib/python3.9/dist-packages/ray/_private/node.py", line 1419, in start_head_processes self.start_api_server( File "/usr/local/lib/python3.9/dist-packages/ray/_private/node.py", line 1168, in start_api_server self._webui_url, process_info = ray._private.services.start_api_server( File "/usr/local/lib/python3.9/dist-packages/ray/_private/services.py", line 1289, in start_api_server raise e from e File "/usr/local/lib/python3.9/dist-packages/ray/_private/services.py", line 1133, in start_api_server raise Exception("Cannot include dashboard with missing packages.") Exception: Cannot include dashboard with missing packages. ERROR:bigdl.dllib.utils.log4Error:

****Call Stack***** 2023-09-21 16:36:04,318 - DataTransformation - MainThread - ERROR - Exception in processing job: 168_usecase-autoprophet1_autoprophet1_VLOCHYV Exception: node_ip: 172.31.213.61 tag: ray-master, pgid: 156, pids: [157, 189, 190], returncode: 1, master_addr: 172.31.213.61:63679,
2023-09-21 16:35:59,841 INFO usage_lib.py:416 -- Usage stats collection is enabled by default without user confirmation because this terminal is detected to be non-interactive. To disable this, add --disable-usage-stats to the command that starts the cluster, or run the following command: ray disable-usage-stats before starting the cluster. See https://docs.ray.io/en/master/cluster/usage-stats.html for more details. 2023-09-21 16:35:59,842 INFO scripts.py:736 -- Local node IP: 172.31.213.61 2023-09-21 16:36:00,864 ERROR services.py:1129 -- --include-dashboard is not supported when minimal ray is used.Download ray[default] to use the dashboard.

lalalapotter commented 9 months ago

Please install ray[default] package, which includes dashboard related dependencies.

pip install ray[default]==1.9.2

Besides, you may add spark-bigdl.conf in spark-submit command follow the guidance of this known issue.

SjeYinTeoIntel commented 9 months ago

Hi, after pip install ray[default]==1.9.2. it will complaint no tabulate module. after pip install tabulate. it will show below issue.

Initializing orca context WARNING: sun.reflect.Reflection.getCallerClass is not supported. This will impact performance. 2023-09-22 16:22:38,539 Thread-4 WARN The bufferSize is set to 4000 but bufferedIo is false: false 2023-09-22 16:22:38,542 Thread-4 WARN The bufferSize is set to 4000 but bufferedIo is false: false 2023-09-22 16:22:38,543 Thread-4 WARN The bufferSize is set to 4000 but bufferedIo is false: false 2023-09-22 16:22:38,543 Thread-4 WARN The bufferSize is set to 4000 but bufferedIo is false: false 23-09-22 16:22:38 [Thread-4] INFO Engine$:122 - Auto detect executor number and executor cores number 23-09-22 16:22:38 [Thread-4] INFO Engine$:124 - Executor number is 1 and executor cores number is 6 23-09-22 16:22:38 [Thread-4] INFO ThreadPool$:95 - Set mkl threads to 1 on thread 19 23/09/22 16:22:38 WARN SparkContext: Using an existing SparkContext; some configuration may not take effect. 23-09-22 16:22:38 [Thread-4] INFO Engine$:461 - Find existing spark context. Checking the spark conf... 23-09-22 16:22:38 [Thread-4] WARN Engine$:467 - Engine.init: Can not find spark.shuffle.reduceLocality.enabled. For details please check https://bigdl-project.github.io/master/#APIGuide/Engine/ 23-09-22 16:22:38 [Thread-4] WARN Engine$:467 - Engine.init: Can not find spark.shuffle.blockTransferService. For details please check https://bigdl-project.github.io/master/#APIGuide/Engine/ 23-09-22 16:22:38 [Thread-4] WARN Engine$:467 - Engine.init: Can not find spark.scheduler.minRegisteredResourcesRatio. For details please check https://bigdl-project.github.io/master/#APIGuide/Engine/ 23-09-22 16:22:38 [Thread-4] WARN Engine$:467 - Engine.init: Can not find spark.scheduler.maxRegisteredResourcesWaitingTime. For details please check https://bigdl-project.github.io/master/#APIGuide/Engine/ 23-09-22 16:22:38 [Thread-4] WARN Engine$:467 - Engine.init: Can not find spark.speculation. For details please check https://bigdl-project.github.io/master/#APIGuide/Engine/ 23-09-22 16:22:38 [Thread-4] WARN Engine$:467 - Engine.init: Can not find spark.serializer. For details please check https://bigdl-project.github.io/master/#APIGuide/Engine/ 23-09-22 16:22:38 [Thread-4] WARN Engine$:470 - Engine.init: spark.driver.extraJavaOptions should be -Dlog4j2.info, but it is -Dcom.amazonaws.services.s3.enableV4=true. For details please check https://bigdl-project.github.io/master/#APIGuide/Engine/ cls.getname: com.intel.analytics.bigdl.dllib.utils.python.api.Sample BigDLBasePickler registering: bigdl.dllib.utils.common Sample cls.getname: com.intel.analytics.bigdl.dllib.utils.python.api.EvaluatedResult BigDLBasePickler registering: bigdl.dllib.utils.common EvaluatedResult cls.getname: com.intel.analytics.bigdl.dllib.utils.python.api.JTensor BigDLBasePickler registering: bigdl.dllib.utils.common JTensor cls.getname: com.intel.analytics.bigdl.dllib.utils.python.api.JActivity BigDLBasePickler registering: bigdl.dllib.utils.common JActivity Launching Ray on cluster with Spark barrier mode Start to launch ray driver Executing command: ray start --address 172.31.162.63:48385 --num-cpus 0 --node-ip-address 172.31.188.167 2023-09-22 16:22:50,039 INFO scripts.py:747 -- Local node IP: 172.31.188.167 2023-09-22 16:22:50,152 SUCC scripts.py:755 -- -------------------- 2023-09-22 16:22:50,152 SUCC scripts.py:756 -- Ray runtime started. 2023-09-22 16:22:50,152 SUCC scripts.py:757 -- -------------------- 2023-09-22 16:22:50,152 INFO scripts.py:759 -- To terminate the Ray runtime, run 2023-09-22 16:22:50,152 INFO scripts.py:760 -- ray stop

2023-09-22 16:22:50,057 WARNING services.py:1816 -- WARNING: The object store is using /tmp instead of /dev/shm because /dev/shm has only 67108864 bytes available. This will harm performance! You may be able to free up space by deleting files in /dev/shm. If you are inside a Docker container, you can increase /dev/shm size by passing '--shm-size=0.90gb' to 'docker run' (or add it to the run_options list in a Ray cluster config). Make sure to set this to more than 30% of available RAM. [2023-09-22 16:22:50,151 I 218 218] global_state_accessor.cc:360: This node has an IP address of 172.31.188.167, while we can not found the matched Raylet address. This maybe come from when you connect the Ray cluster with a different IP address or connect a container.

2023-09-22 16:22:50,039 INFO scripts.py:747 -- Local node IP: 172.31.188.167 2023-09-22 16:22:50,152 SUCC scripts.py:755 -- -------------------- 2023-09-22 16:22:50,152 SUCC scripts.py:756 -- Ray runtime started. 2023-09-22 16:22:50,152 SUCC scripts.py:757 -- -------------------- 2023-09-22 16:22:50,152 INFO scripts.py:759 -- To terminate the Ray runtime, run 2023-09-22 16:22:50,152 INFO scripts.py:760 -- ray stop

File "/usr/local/lib/python3.9/dist-packages/bigdl/orca/ray/ray_daemon.py", line 26 logging.info(f"Stopping pgid {pgid} by ray_daemon.") ^ SyntaxError: invalid syntax 2023-09-22 16:22:51,307 INFO worker.py:842 -- Connecting to existing Ray cluster at address: 172.31.162.63:48385 {'node_ip_address': '172.31.188.167', 'raylet_ip_address': '172.31.188.167', 'redis_address': '172.31.162.63:48385', 'object_store_address': '/tmp/ray/session_2023-09-22_16-22-42_606186_153/sockets/plasma_store', 'raylet_socket_name': '/tmp/ray/session_2023-09-22_16-22-42_606186_153/sockets/raylet', 'webui_url': '172.31.162.63:8265', 'session_dir': '/tmp/ray/session_2023-09-22_16-22-42_606186_153', 'metrics_export_port': 58447, 'node_id': '952484ca668860d4331ca98bcfd18193dc87149ef738aa87f857e163'}

/bin/sh: line 1: hadoop: command not found /bin/sh: line 1: hadoop: command not found

ERROR:bigdl.dllib.utils.log4Error:

****Usage Error**** /bin/sh: line 1: hadoop: command not found /bin/sh: line 1: hadoop: command not found

ERROR:bigdl.dllib.utils.log4Error:

****Call Stack***** 2023-09-22 16:22:51,477 WARNING function_runner.py:561 -- Function checkpointing is disabled. This may result in unexpected behavior when using checkpointing features or certain schedulers. To enable, set the train function arguments to be func(config, checkpoint_dir=None). 2023-09-22 16:22:51,488 INFO logger.py:605 -- pip install "ray[tune]" to see TensorBoard files. 2023-09-22 16:22:51,488 WARNING callback.py:114 -- The TensorboardX logger cannot be instantiated because either TensorboardX or one of it's dependencies is not installed. Please make sure you have the latest version of TensorboardX installed: pip install -U tensorboardx (raylet, ip=172.31.130.119) /usr/local/lib/python3.9/dist-packages/ray/dashboard/agent.py:152: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead. (raylet, ip=172.31.130.119) if LooseVersion(aiohttp.version) < LooseVersion("4.0.0"): == Status == Current time: 2023-09-22 16:22:51 (running for 00:00:00.17) Memory usage on this node: 2.4/20.0 GiB Using FIFO scheduling algorithm. Resources requested: 0/18 CPUs, 0/0 GPUs, 0.0/27.01 GiB heap, 0.0/12.12 GiB objects (0.0/3.0 _mxnet_server, 0.0/6.0 _reserved, 0.0/3.0 _mxnet_worker) Result logdir: /tmp/auto_prophet_logs/auto_prophet Number of trials: 16/16 (16 PENDING) +------------------------+----------+-------+---------------------------+---------------------+------------------------+--------------------+---------------------------+ | Trial name | status | loc | changepoint_prior_scale | changepoint_range | holidays_prior_scale | seasonality_mode | seasonality_prior_scale | |------------------------+----------+-------+---------------------------+---------------------+------------------------+--------------------+---------------------------| | train_func_46d2c_00000 | PENDING | | 0.005 | 0.924879 | 4.06874 | multiplicative | 0.01 | | train_func_46d2c_00001 | PENDING | | 0.05 | 0.915135 | 0.299019 | additive | 0.01 | | train_func_46d2c_00002 | PENDING | | 0.1 | 0.913239 | 0.0180456 | additive | 0.01 | | train_func_46d2c_00003 | PENDING | | 0.5 | 0.927584 | 0.126795 | multiplicative | 0.01 | | train_func_46d2c_00004 | PENDING | | 0.005 | 0.921992 | 3.07079 | multiplicative | 0.1 | | train_func_46d2c_00005 | PENDING | | 0.05 | 0.93944 | 2.34032 | additive | 0.1 | | train_func_46d2c_00006 | PENDING | | 0.1 | 0.87637 | 0.473946 | multiplicative | 0.1 | | train_func_46d2c_00007 | PENDING | | 0.5 | 0.833584 | 2.49943 | additive | 0.1 | | train_func_46d2c_00008 | PENDING | | 0.005 | 0.821207 | 3.96971 | multiplicative | 1 | | train_func_46d2c_00009 | PENDING | | 0.05 | 0.86216 | 1.61319 | multiplicative | 1 | | train_func_46d2c_00010 | PENDING | | 0.1 | 0.806524 | 0.568673 | additive | 1 | | train_func_46d2c_00011 | PENDING | | 0.5 | 0.866704 | 0.0895926 | additive | 1 | | train_func_46d2c_00012 | PENDING | | 0.005 | 0.931741 | 7.89272 | additive | 10 | | train_func_46d2c_00013 | PENDING | | 0.05 | 0.929883 | 0.357612 | additive | 10 | | train_func_46d2c_00014 | PENDING | | 0.1 | 0.916191 | 4.9683 | additive | 10 | | train_func_46d2c_00015 | PENDING | | 0.5 | 0.904779 | 9.52464 | additive | 10 | +------------------------+----------+-------+---------------------------+---------------------+------------------------+--------------------+---------------------------+

(raylet, ip=172.31.207.254) /usr/local/lib/python3.9/dist-packages/ray/dashboard/agent.py:152: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead. (raylet, ip=172.31.207.254) if LooseVersion(aiohttp.version) < LooseVersion("4.0.0"): 2023-09-22 16:22:52,527 ERROR syncer.py:111 -- Log sync requires rsync to be installed. (bundle_reservation_check_func pid=329, ip=172.31.207.254) Importing plotly failed. Interactive plots will not work. (bundle_reservation_check_func pid=325, ip=172.31.207.254) Importing plotly failed. Interactive plots will not work. (bundle_reservation_check_func pid=497, ip=172.31.162.63) Importing plotly failed. Interactive plots will not work. (bundle_reservation_check_func pid=331, ip=172.31.130.119) Importing plotly failed. Interactive plots will not work. (bundle_reservation_check_func pid=329, ip=172.31.130.119) Importing plotly failed. Interactive plots will not work. (bundle_reservation_check_func pid=326, ip=172.31.207.254) Importing plotly failed. Interactive plots will not work. (bundle_reservation_check_func pid=495, ip=172.31.162.63) Importing plotly failed. Interactive plots will not work. (bundle_reservation_check_func pid=333, ip=172.31.130.119) Importing plotly failed. Interactive plots will not work. (pid=324, ip=172.31.207.254) Importing plotly failed. Interactive plots will not work. (pid=328, ip=172.31.130.119) Importing plotly failed. Interactive plots will not work. (raylet) /usr/local/lib/python3.9/dist-packages/ray/dashboard/agent.py:152: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead. (raylet) if LooseVersion(aiohttp.version) < LooseVersion("4.0.0"): (bundle_reservation_check_func pid=327, ip=172.31.207.254) Importing plotly failed. Interactive plots will not work. (bundle_reservation_check_func pid=496, ip=172.31.162.63) Importing plotly failed. Interactive plots will not work. (bundle_reservation_check_func pid=500, ip=172.31.162.63) Importing plotly failed. Interactive plots will not work. (bundle_reservation_check_func pid=499, ip=172.31.162.63) Importing plotly failed. Interactive plots will not work. (bundle_reservation_check_func pid=498, ip=172.31.162.63) Importing plotly failed. Interactive plots will not work. (bundle_reservation_check_func pid=330, ip=172.31.130.119) Importing plotly failed. Interactive plots will not work. (raylet, ip=172.31.162.63) /usr/local/lib/python3.9/dist-packages/ray/dashboard/agent.py:152: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead. (raylet, ip=172.31.162.63) if LooseVersion(aiohttp.version) < LooseVersion("4.0.0"): (ImplicitFunc pid=329, ip=172.31.207.254) 16:22:54 - cmdstanpy - INFO - Chain [1] start processing (ImplicitFunc pid=326, ip=172.31.207.254) 16:22:54 - cmdstanpy - INFO - Chain [1] start processing (ImplicitFunc pid=325, ip=172.31.207.254) 16:22:54 - cmdstanpy - INFO - Chain [1] start processing (ImplicitFunc pid=497, ip=172.31.162.63) 16:22:55 - cmdstanpy - INFO - Chain [1] start processing (ImplicitFunc pid=327, ip=172.31.207.254) 16:22:55 - cmdstanpy - INFO - Chain [1] start processing (ImplicitFunc pid=326, ip=172.31.207.254) 16:22:55 - cmdstanpy - INFO - Chain [1] done processing (ImplicitFunc pid=331, ip=172.31.130.119) 16:22:55 - cmdstanpy - INFO - Chain [1] start processing (ImplicitFunc pid=333, ip=172.31.130.119) 16:22:55 - cmdstanpy - INFO - Chain [1] start processing (ImplicitFunc pid=330, ip=172.31.130.119) 16:22:55 - cmdstanpy - INFO - Chain [1] start processing (ImplicitFunc pid=329, ip=172.31.130.119) 16:22:55 - cmdstanpy - INFO - Chain [1] start processing (ImplicitFunc pid=329, ip=172.31.207.254) 16:22:55 - cmdstanpy - INFO - Chain [1] done processing (ImplicitFunc pid=495, ip=172.31.162.63) 16:22:55 - cmdstanpy - INFO - Chain [1] start processing (ImplicitFunc pid=333, ip=172.31.130.119) 16:22:55 - cmdstanpy - INFO - Chain [1] done processing (ImplicitFunc pid=497, ip=172.31.162.63) 16:22:55 - cmdstanpy - INFO - Chain [1] done processing (ImplicitFunc pid=328, ip=172.31.130.119) 16:22:55 - cmdstanpy - INFO - Chain [1] start processing (ImplicitFunc pid=327, ip=172.31.207.254) 16:22:55 - cmdstanpy - INFO - Chain [1] done processing (ImplicitFunc pid=324, ip=172.31.207.254) 16:22:55 - cmdstanpy - INFO - Chain [1] start processing (ImplicitFunc pid=331, ip=172.31.130.119) 16:22:55 - cmdstanpy - INFO - Chain [1] done processing (ImplicitFunc pid=329, ip=172.31.130.119) 16:22:55 - cmdstanpy - INFO - Chain [1] done processing (ImplicitFunc pid=327, ip=172.31.207.254) 2023-09-22 16:22:55,375 ERROR function_runner.py:268 -- Runner Thread raised error. (ImplicitFunc pid=327, ip=172.31.207.254) Traceback (most recent call last): (ImplicitFunc pid=327, ip=172.31.207.254) File "/usr/local/lib/python3.9/dist-packages/ray/tune/function_runner.py", line 262, in run (ImplicitFunc pid=327, ip=172.31.207.254) self._entrypoint() (ImplicitFunc pid=327, ip=172.31.207.254) File "/usr/local/lib/python3.9/dist-packages/ray/tune/function_runner.py", line 330, in entrypoint (ImplicitFunc pid=327, ip=172.31.207.254) return self._trainable_func(self.config, self._status_reporter, (ImplicitFunc pid=327, ip=172.31.207.254) File "/usr/local/lib/python3.9/dist-packages/ray/util/tracing/tracing_helper.py", line 451, in _resume_span (ImplicitFunc pid=327, ip=172.31.207.254) return method(self, _args, _kwargs) (ImplicitFunc pid=327, ip=172.31.207.254) File "/usr/local/lib/python3.9/dist-packages/ray/tune/function_runner.py", line 597, in _trainable_func (ImplicitFunc pid=327, ip=172.31.207.254) output = fn() (ImplicitFunc pid=327, ip=172.31.207.254) File "/usr/local/lib/python3.9/dist-packages/bigdl/orca/automl/search/ray_tune/ray_tune_search_engine.py", line 349, in train_func (ImplicitFunc pid=327, ip=172.31.207.254) result = trial_model.fit_eval(data=train_data, (ImplicitFunc pid=327, ip=172.31.207.254) File "/usr/local/lib/python3.9/dist-packages/bigdl/chronos/model/prophet.py", line 87, in fit_eval (ImplicitFunc pid=327, ip=172.31.207.254) self._fit(data, config) (ImplicitFunc pid=327, ip=172.31.207.254) File "/usr/local/lib/python3.9/dist-packages/bigdl/chronos/model/prophet.py", line 67, in _fit (ImplicitFunc pid=327, ip=172.31.207.254) self.model.fit(data) (ImplicitFunc pid=327, ip=172.31.207.254) File "/usr/local/lib/python3.9/dist-packages/prophet/forecaster.py", line 1114, in fit (ImplicitFunc pid=327, ip=172.31.207.254) raise Exception('Prophet object can only be fit once. ' (ImplicitFunc pid=327, ip=172.31.207.254) Exception: Prophet object can only be fit once. Instantiate a new object. (ImplicitFunc pid=327, ip=172.31.207.254) Exception in thread Thread-2: (ImplicitFunc pid=327, ip=172.31.207.254) Traceback (most recent call last): (ImplicitFunc pid=327, ip=172.31.207.254) File "/usr/lib/python3.9/threading.py", line 954, in _bootstrap_inner (ImplicitFunc pid=327, ip=172.31.207.254) self.run() (ImplicitFunc pid=327, ip=172.31.207.254) File "/usr/local/lib/python3.9/dist-packages/ray/tune/function_runner.py", line 281, in run (ImplicitFunc pid=327, ip=172.31.207.254) raise e (ImplicitFunc pid=327, ip=172.31.207.254) File "/usr/local/lib/python3.9/dist-packages/ray/tune/function_runner.py", line 262, in run (ImplicitFunc pid=327, ip=172.31.207.254) self._entrypoint() (ImplicitFunc pid=327, ip=172.31.207.254) File "/usr/local/lib/python3.9/dist-packages/ray/tune/function_runner.py", line 330, in entrypoint (ImplicitFunc pid=327, ip=172.31.207.254) return self._trainable_func(self.config, self._status_reporter, (ImplicitFunc pid=327, ip=172.31.207.254) File "/usr/local/lib/python3.9/dist-packages/ray/util/tracing/tracing_helper.py", line 451, in _resume_span (ImplicitFunc pid=327, ip=172.31.207.254) return method(self, _args, _kwargs) (ImplicitFunc pid=327, ip=172.31.207.254) File "/usr/local/lib/python3.9/dist-packages/ray/tune/function_runner.py", line 597, in _trainable_func (ImplicitFunc pid=327, ip=172.31.207.254) output = fn() (ImplicitFunc pid=327, ip=172.31.207.254) File "/usr/local/lib/python3.9/dist-packages/bigdl/orca/automl/search/ray_tune/ray_tune_search_engine.py", line 349, in train_func (ImplicitFunc pid=327, ip=172.31.207.254) result = trial_model.fit_eval(data=train_data, (ImplicitFunc pid=327, ip=172.31.207.254) File "/usr/local/lib/python3.9/dist-packages/bigdl/chronos/model/prophet.py", line 87, in fit_eval (ImplicitFunc pid=327, ip=172.31.207.254) self._fit(data, config) (ImplicitFunc pid=327, ip=172.31.207.254) File "/usr/local/lib/python3.9/dist-packages/bigdl/chronos/model/prophet.py", line 67, in _fit (ImplicitFunc pid=327, ip=172.31.207.254) self.model.fit(data) (ImplicitFunc pid=327, ip=172.31.207.254) File "/usr/local/lib/python3.9/dist-packages/prophet/forecaster.py", line 1114, in fit (ImplicitFunc pid=327, ip=172.31.207.254) raise Exception('Prophet object can only be fit once. ' (ImplicitFunc pid=327, ip=172.31.207.254) Exception: Prophet object can only be fit once. Instantiate a new object. (ImplicitFunc pid=325, ip=172.31.207.254) 16:22:55 - cmdstanpy - INFO - Chain [1] done processing (ImplicitFunc pid=496, ip=172.31.162.63) 16:22:55 - cmdstanpy - INFO - Chain [1] start processing (ImplicitFunc pid=500, ip=172.31.162.63) 16:22:55 - cmdstanpy - INFO - Chain [1] start processing (ImplicitFunc pid=499, ip=172.31.162.63) 16:22:55 - cmdstanpy - INFO - Chain [1] start processing (ImplicitFunc pid=330, ip=172.31.130.119) 16:22:55 - cmdstanpy - INFO - Chain [1] done processing (ImplicitFunc pid=330, ip=172.31.130.119) 2023-09-22 16:22:55,450 ERROR function_runner.py:268 -- Runner Thread raised error. (ImplicitFunc pid=330, ip=172.31.130.119) Traceback (most recent call last): (ImplicitFunc pid=330, ip=172.31.130.119) File "/usr/local/lib/python3.9/dist-packages/ray/tune/function_runner.py", line 262, in run (ImplicitFunc pid=330, ip=172.31.130.119) self._entrypoint() (ImplicitFunc pid=330, ip=172.31.130.119) File "/usr/local/lib/python3.9/dist-packages/ray/tune/function_runner.py", line 330, in entrypoint (ImplicitFunc pid=330, ip=172.31.130.119) return self._trainable_func(self.config, self._status_reporter, (ImplicitFunc pid=330, ip=172.31.130.119) File "/usr/local/lib/python3.9/dist-packages/ray/util/tracing/tracing_helper.py", line 451, in _resume_span (ImplicitFunc pid=330, ip=172.31.130.119) return method(self, *_args, _kwargs) (ImplicitFunc pid=330, ip=172.31.130.119) File "/usr/local/lib/python3.9/dist-packages/ray/tune/function_runner.py", line 597, in _trainable_func (ImplicitFunc pid=330, ip=172.31.130.119) output = fn() (ImplicitFunc pid=330, ip=172.31.130.119) File "/usr/local/lib/python3.9/dist-packages/bigdl/orca/automl/search/ray_tune/ray_tune_search_engine.py", line 349, in train_func (ImplicitFunc pid=330, ip=172.31.130.119) result = trial_model.fit_eval(data=train_data, (ImplicitFunc pid=330, ip=172.31.130.119) File "/usr/local/lib/python3.9/dist-packages/bigdl/chronos/model/prophet.py", line 87, in fit_eval (ImplicitFunc pid=330, ip=172.31.130.119) self._fit(data, config) (ImplicitFunc pid=330, ip=172.31.130.119) File "/usr/local/lib/python3.9/dist-packages/bigdl/chronos/model/prophet.py", line 67, in _fit (ImplicitFunc pid=330, ip=172.31.130.119) self.model.fit(data) (ImplicitFunc pid=330, ip=172.31.130.119) File "/usr/local/lib/python3.9/dist-packages/prophet/forecaster.py", line 1114, in fit (ImplicitFunc pid=330, ip=172.31.130.119) raise Exception('Prophet object can only be fit once. ' (ImplicitFunc pid=330, ip=172.31.130.119) Exception: Prophet object can only be fit once. Instantiate a new object. (ImplicitFunc pid=330, ip=172.31.130.119) Exception in thread Thread-2:

sgwhat commented 9 months ago

Hi, could you please provide me with your prophet version?

sgwhat commented 9 months ago

Seems 1.1.0 version is not suitable on the cluster.

WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.apache.spark.unsafe.Platform (file:/opt/spark/jars/spark-unsafe_2.12-3.2.1.jar) to constructor java.nio.DirectByteBuffer(long,int)
WARNING: Please consider reporting this to the maintainers of org.apache.spark.unsafe.Platform
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
23/09/25 10:41:59 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Traceback (most recent call last):
  File "/opt/easydata-app/main.py", line 6, in <module>
    from python.operation.data_transformation import DataTransformation
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "/opt/easydata-app/python/operation/data_transformation.py", line 15, in <module python.operation.data_transformation>
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "/opt/easydata-app/python/transformation_analytics/ml_model_train_test.py", line 12, in <module python.transformation_analytics.ml_model_train_test>
  File "/usr/local/lib/python3.9/dist-packages/prophet/__init__.py", line 8, in <module>
    from prophet.forecaster import Prophet
  File "/usr/local/lib/python3.9/dist-packages/prophet/forecaster.py", line 17, in <module>
    from prophet.make_holidays import get_holiday_names, make_holidays_df
  File "/usr/local/lib/python3.9/dist-packages/prophet/make_holidays.py", line 14, in <module>
    import prophet.hdays as hdays_part2
  File "/usr/local/lib/python3.9/dist-packages/prophet/hdays.py", line 683, in <module>
    class TU(Turkey):
  File "/usr/local/lib/python3.9/dist-packages/holidays/registry.py", line 175, in __init__
    raise TypeError(
TypeError: This is a python-holidays entity loader class. For entity inheritance purposes please import a class you want to derive from directly: e.g., `from holidays.countries import Entity` or `from holidays.financial import Entity`.
log4j:WARN No appenders could be found for logger (org.apache.spark.util.ShutdownHookManager).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.

sgwhat commented 9 months ago

Hi, could you please provide me with your prophet version?

Using 1.1.0 version.

intel-analytics / ipex-llm

ERROR services.py:1129 -- --include-dashboard is not supported when minimal ray is used.Download ray[default] to use the dashboard. #9031