statmike / vertex-ai-mlops

Google Cloud Platform Vertex AI end-to-end workflows for machine learning operations
Apache License 2.0
494 stars 221 forks source link

Error on Docker Run Command #50

Closed varshasahasrabuddhe closed 9 months ago

varshasahasrabuddhe commented 1 year ago

Hi Steve, i am getting below errors on Docker Run command !docker run {IMAGE_URI} --PROJECT_ID {PROJECT_ID} --DATANAME {DATANAME} --NOTEBOOK {NOTEBOOK} --horizon {14} --no-yearly

Errors - model-stock-management applied_forecasting 04f 0%| | 0/12 [00:00<?, ?it/s]23:01:57 - cmdstanpy - INFO - Chain [1] start processing 23:01:57 - cmdstanpy - INFO - Chain [1] start processing 23:01:57 - cmdstanpy - INFO - Chain [1] start processing 23:01:57 - cmdstanpy - INFO - Chain [1] done processing 23:01:57 - cmdstanpy - ERROR - Chain [1] error: terminated by signal 11 Unknown error -11 0%| | 0/12 [00:00<?, ?it/s]23:01:57 - cmdstanpy - INFO - Chain [1] done processing 23:01:57 - cmdstanpy - ERROR - Chain [1] error: terminated by signal 11 Unknown error -11 23:01:57 - cmdstanpy - INFO - Chain [1] done processing 23:01:57 - cmdstanpy - ERROR - Chain [1] error: terminated by signal 11 Unknown error -11

multiprocessing.pool.RemoteTraceback: """ Traceback (most recent call last): File "/opt/conda/lib/python3.7/multiprocessing/pool.py", line 121, in worker result = (True, func(*args, kwds)) File "/fit/prophet.py", line 48, in run_prophet p.fit(series) File "/opt/conda/lib/python3.7/site-packages/prophet/forecaster.py", line 1181, in fit self.params = self.stan_backend.fit(stan_init, dat, kwargs) File "/opt/conda/lib/python3.7/site-packages/prophet/models.py", line 94, in fit raise e File "/opt/conda/lib/python3.7/site-packages/prophet/models.py", line 90, in fit self.stan_fit = self.model.optimize(**args) File "/opt/conda/lib/python3.7/site-packages/cmdstanpy/model.py", line 738, in optimize raise RuntimeError(msg) RuntimeError: Error during optimization! Command '/opt/conda/lib/python3.7/site-packages/prophet/stan_model/prophet_model.bin random seed=52644 data file=/tmp/tmptt1na6uf/ouw39fzc.json init=/tmp/tmptt1na6uf/q6k325gk.json output file=/tmp/tmptt1na6uf/prophet_modeli11498bg/prophet_model-20230627230157.csv method=optimize algorithm=newton iter=10000' failed: """

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "/opt/conda/lib/python3.7/runpy.py", line 193, in _run_module_as_main "main", mod_spec) File "/opt/conda/lib/python3.7/runpy.py", line 85, in _run_code exec(code, run_globals) File "/fit/prophet.py", line 55, in predictions = list(tqdm(pool.imap(run_prophet, seriesFrames), total = len(seriesFrames))) File "/opt/conda/lib/python3.7/site-packages/tqdm/std.py", line 1178, in iter for obj in iterable: File "/opt/conda/lib/python3.7/multiprocessing/pool.py", line 748, in next raise value RuntimeError: Error during optimization! Command '/opt/conda/lib/python3.7/site-packages/prophet/stan_model/prophet_model.bin random seed=52644 data file=/tmp/tmptt1na6uf/ouw39fzc.json init=/tmp/tmptt1na6uf/q6k325gk.json output file=/tmp/tmptt1na6uf/prophet_modeli11498bg/prophet_model-20230627230157.csv method=optimize algorithm=newton iter=10000' failed:

Appreciate your help in this Thanks, Varsha

statmike commented 1 year ago

Hello @varshasahasrabuddhe Can you point to which notebook you are using that experiences this error? Thank You, @statmike