search

Winson-030 / dify-kubernetes

Deploy Dify on Kubernetes
GNU General Public License v3.0
50 stars 14 forks source link

api-worker does not work! #2

Closed GeekLockie closed 2 months ago

GeekLockie commented 3 months ago

stdout : None of PyTorch, TensorFlow >= 2.0, or Flax have been found. Models won't be available and only tokenizers, configuration and file/data utilities can be used. 2024-06-20 06:07:09.996 WARNING [MainThread] [ext_mail.py:50] - MAIL_TYPE is not set /usr/local/lib/python3.10/site-packages/celery/platforms.py:829: SecurityWarning: You're running the worker with superuser privileges: this is absolutely not recommended! Please specify a different user using the --uid option. User information: uid=0 euid=0 gid=0 egid=0 warnings.warn(SecurityWarning(ROOT_DISCOURAGED.format( Traceback (most recent call last): File "/usr/local/bin/celery", line 8, in sys.exit(main()) File "/usr/local/lib/python3.10/site-packages/celery/main.py", line 15, in main sys.exit(_main()) File "/usr/local/lib/python3.10/site-packages/celery/bin/celery.py", line 236, in main return celery(auto_envvar_prefix="CELERY") File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1157, in call return self.main(args, kwargs) File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1078, in main rv = self.invoke(ctx) File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1688, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1434, in invoke return ctx.invoke(self.callback, ctx.params) File "/usr/local/lib/python3.10/site-packages/click/core.py", line 783, in invoke return __callback(args, kwargs) File "/usr/local/lib/python3.10/site-packages/click/decorators.py", line 33, in new_func return f(get_current_context(), *args, *kwargs) File "/usr/local/lib/python3.10/site-packages/celery/bin/base.py", line 134, in caller return f(ctx, args, kwargs) File "/usr/local/lib/python3.10/site-packages/celery/bin/worker.py", line 348, in worker worker = app.Worker( File "/usr/local/lib/python3.10/site-packages/celery/worker/worker.py", line 98, in init self.setup_instance(self.prepare_args(kwargs)) File "/usr/local/lib/python3.10/site-packages/celery/worker/worker.py", line 119, in setup_instance self._conninfo = self.app.connection_for_read() File "/usr/local/lib/python3.10/site-packages/celery/app/base.py", line 819, in connection_for_read return self._connection(url or self.conf.broker_read_url, **kwargs) File "/usr/local/lib/python3.10/site-packages/celery/app/base.py", line 878, in _connection return self.amqp.Connection( File "/usr/local/lib/python3.10/site-packages/kombu/connection.py", line 203, in init url_params = parse_url(hostname) File "/usr/local/lib/python3.10/site-packages/kombu/utils/url.py", line 38, in parse_url scheme, host, port, user, password, path, query = _parse_url(url) File "/usr/local/lib/python3.10/site-packages/kombu/utils/url.py", line 70, in url_to_parts parts.port, File "/usr/local/lib/python3.10/urllib/parse.py", line 185, in port raise ValueError(f"Port could not be cast to integer value as {port!r}") ValueError: Port could not be cast to integer value as '$(REDIS_PORT)'

I didn't change the Yaml file ,and in the pod ,when I typed "env", it shows "REDIS_PORT" param exists. So please help me ...

Winson-030 commented 3 months ago

This was pre configured in secret, you can check it out in the yaml file

Winson-030 commented 3 months ago

Can you show the execute log like below? I will debug with you.

❯ kubectl apply -f https://raw.githubusercontent.com/Winson-030/dify-kubernetes/main/dify-deployment.yaml

namespace/dify created
secret/dify-credentials created
serviceaccount/dify-pg created
role.rbac.authorization.k8s.io/dify-pg created
rolebinding.rbac.authorization.k8s.io/dify-pg created
statefulset.apps/dify-pg created
service/dify-pg created
serviceaccount/dify-redis created
role.rbac.authorization.k8s.io/dify-redis created
rolebinding.rbac.authorization.k8s.io/dify-redis created
statefulset.apps/dify-redis created
service/dify-redis created
serviceaccount/dify-weaviate created
role.rbac.authorization.k8s.io/dify-weaviate created
rolebinding.rbac.authorization.k8s.io/dify-weaviate created
statefulset.apps/dify-weaviate created
service/dify-weaviate created
deployment.apps/dify-sandbox created
service/dify-sandbox created
statefulset.apps/dify-api created
service/dify-api created
statefulset.apps/dify-worker created
service/dify-worker created
deployment.apps/dify-web created
service/dify-web created
configmap/dify-nginx created
deployment.apps/dify-nginx created
service/dify-nginx created
service/dify-nginx-nodeport created
Winson-030 commented 3 months ago 
None of PyTorch, TensorFlow >= 2.0, or Flax have been found. Models won't be available and only tokenizers, configuration and file/data utilities can be used.
2024-06-20 12:09:10.198 WARNING [MainThread] [ext_mail.py:50] - MAIL_TYPE is not set
/usr/local/lib/python3.10/site-packages/celery/platforms.py:829: SecurityWarning: You're running the worker with superuser privileges: this is
absolutely not recommended!

Please specify a different user using the --uid option.

User information: uid=0 euid=0 gid=0 egid=0

  warnings.warn(SecurityWarning(ROOT_DISCOURAGED.format(

 -------------- celery@dify-worker-0 v5.3.6 (emerald-rush)
--- ***** ----- 
-- ******* ---- Linux-6.9.3-orbstack-00146-g1a8d02c90788-aarch64-with-glibc2.36 2024-06-20 12:09:11
- *** --- * --- 
- ** ---------- [config]
- ** ---------- .> app:         app:0xffff652590f0
- ** ---------- .> transport:   redis://:**@dify-redis:6379/1
- ** ---------- .> results:     postgresql://postgres:**@dify-pg:5432/dify
- *** --- * --- .> concurrency: 1 (gevent)
-- ******* ---- .> task events: OFF (enable -E to monitor tasks in this worker)
--- ***** ----- 
 -------------- [queues]
                .> dataset          exchange=dataset(direct) key=dataset
                .> generation       exchange=generation(direct) key=generation

[tasks]
  . schedule.clean_embedding_cache_task.clean_embedding_cache_task
  . schedule.clean_unused_datasets_task.clean_unused_datasets_task
  . tasks.add_document_to_index_task.add_document_to_index_task
  . tasks.annotation.add_annotation_to_index_task.add_annotation_to_index_task
  . tasks.annotation.batch_import_annotations_task.batch_import_annotations_task
  . tasks.annotation.delete_annotation_index_task.delete_annotation_index_task
  . tasks.annotation.disable_annotation_reply_task.disable_annotation_reply_task
  . tasks.annotation.enable_annotation_reply_task.enable_annotation_reply_task
  . tasks.annotation.update_annotation_to_index_task.update_annotation_to_index_task
  . tasks.batch_create_segment_to_index_task.batch_create_segment_to_index_task
  . tasks.clean_dataset_task.clean_dataset_task
  . tasks.clean_document_task.clean_document_task
  . tasks.clean_notion_document_task.clean_notion_document_task
  . tasks.deal_dataset_vector_index_task.deal_dataset_vector_index_task
  . tasks.delete_segment_from_index_task.delete_segment_from_index_task
  . tasks.disable_segment_from_index_task.disable_segment_from_index_task
  . tasks.document_indexing_sync_task.document_indexing_sync_task
  . tasks.document_indexing_task.document_indexing_task
  . tasks.document_indexing_update_task.document_indexing_update_task
  . tasks.duplicate_document_indexing_task.duplicate_document_indexing_task
  . tasks.enable_segment_to_index_task.enable_segment_to_index_task
  . tasks.mail_invite_member_task.send_invite_member_mail_task
  . tasks.recover_document_indexing_task.recover_document_indexing_task
  . tasks.remove_document_from_index_task.remove_document_from_index_task
  . tasks.retry_document_indexing_task.retry_document_indexing_task
  . tasks.sync_website_document_indexing_task.sync_website_document_indexing_task

[2024-06-20 12:09:11,190: INFO/MainProcess] Connected to redis://:**@dify-redis:6379/1
[2024-06-20 12:09:11,194: INFO/MainProcess] mingle: searching for neighbors
[2024-06-20 12:09:12,203: INFO/MainProcess] mingle: all alone
[2024-06-20 12:09:12,215: INFO/MainProcess] celery@dify-worker-0 ready.
[2024-06-20 12:09:12,221: INFO/MainProcess] pidbox: Connected to redis://:**@dify-redis:6379/1.

I applied with the link and did not found any error, here is the log from worker.

Winson-030 commented 3 months ago 
None of PyTorch, TensorFlow >= 2.0, or Flax have been found. Models won't be available and only tokenizers, configuration and file/data utilities can be used.
2024-06-20 12:09:10.198 WARNING [MainThread] [ext_mail.py:50] - MAIL_TYPE is not set
/usr/local/lib/python3.10/site-packages/celery/platforms.py:829: SecurityWarning: You're running the worker with superuser privileges: this is
absolutely not recommended!

Please specify a different user using the --uid option.

User information: uid=0 euid=0 gid=0 egid=0

  warnings.warn(SecurityWarning(ROOT_DISCOURAGED.format(

 -------------- celery@dify-worker-0 v5.3.6 (emerald-rush)
--- ***** ----- 
-- ******* ---- Linux-6.9.3-orbstack-00146-g1a8d02c90788-aarch64-with-glibc2.36 2024-06-20 12:09:11
- *** --- * --- 
- ** ---------- [config]
- ** ---------- .> app:         app:0xffff652590f0
- ** ---------- .> transport:   redis://:**@dify-redis:6379/1
- ** ---------- .> results:     postgresql://postgres:**@dify-pg:5432/dify
- *** --- * --- .> concurrency: 1 (gevent)
-- ******* ---- .> task events: OFF (enable -E to monitor tasks in this worker)
--- ***** ----- 
 -------------- [queues]
                .> dataset          exchange=dataset(direct) key=dataset
                .> generation       exchange=generation(direct) key=generation

[tasks]
  . schedule.clean_embedding_cache_task.clean_embedding_cache_task
  . schedule.clean_unused_datasets_task.clean_unused_datasets_task
  . tasks.add_document_to_index_task.add_document_to_index_task
  . tasks.annotation.add_annotation_to_index_task.add_annotation_to_index_task
  . tasks.annotation.batch_import_annotations_task.batch_import_annotations_task
  . tasks.annotation.delete_annotation_index_task.delete_annotation_index_task
  . tasks.annotation.disable_annotation_reply_task.disable_annotation_reply_task
  . tasks.annotation.enable_annotation_reply_task.enable_annotation_reply_task
  . tasks.annotation.update_annotation_to_index_task.update_annotation_to_index_task
  . tasks.batch_create_segment_to_index_task.batch_create_segment_to_index_task
  . tasks.clean_dataset_task.clean_dataset_task
  . tasks.clean_document_task.clean_document_task
  . tasks.clean_notion_document_task.clean_notion_document_task
  . tasks.deal_dataset_vector_index_task.deal_dataset_vector_index_task
  . tasks.delete_segment_from_index_task.delete_segment_from_index_task
  . tasks.disable_segment_from_index_task.disable_segment_from_index_task
  . tasks.document_indexing_sync_task.document_indexing_sync_task
  . tasks.document_indexing_task.document_indexing_task
  . tasks.document_indexing_update_task.document_indexing_update_task
  . tasks.duplicate_document_indexing_task.duplicate_document_indexing_task
  . tasks.enable_segment_to_index_task.enable_segment_to_index_task
  . tasks.mail_invite_member_task.send_invite_member_mail_task
  . tasks.recover_document_indexing_task.recover_document_indexing_task
  . tasks.remove_document_from_index_task.remove_document_from_index_task
  . tasks.retry_document_indexing_task.retry_document_indexing_task
  . tasks.sync_website_document_indexing_task.sync_website_document_indexing_task

[2024-06-20 12:09:11,190: INFO/MainProcess] Connected to redis://:**@dify-redis:6379/1
[2024-06-20 12:09:11,194: INFO/MainProcess] mingle: searching for neighbors
[2024-06-20 12:09:12,203: INFO/MainProcess] mingle: all alone
[2024-06-20 12:09:12,215: INFO/MainProcess] celery@dify-worker-0 ready.
[2024-06-20 12:09:12,221: INFO/MainProcess] pidbox: Connected to redis://:**@dify-redis:6379/1.

I applied with the link and did not found any error, here is the log from worker.

my cluster environement is a 3 node k3s. what is yours?

Winson-030 commented 3 months ago

I updated the yaml file just now. remove the secret and set default value. try deploy with new verison and see what would happen.