search

finaldie / auto-news

A personal news aggregator to pull information from multi-sources + LLM (ChatGPT/Gemini/Ollama via LangChain) to help us reading efficiently with less noises, the sources including: Tweets, RSS, YouTube, Web Articles, Reddit, and personal Journal notes.
MIT License
504 stars 54 forks source link

When I use the Ollama model, an error occurs when executing get_num_tokens_from_messages. #86

Closed WiiCoder closed 2 months ago

WiiCoder commented 2 months ago

Description: When I am executing the news_pull task, there are no problems with other steps in the save operation. An error occurs when executing the summarize method. How can I solve this problem?

Env:

    OLLAMA_MODEL: llama3.1
    EMBEDDING_PROVIDER: ollama    
    EMBEDDING_MODEL: nomic-embed-text

Executing：

ops_rss.summarize()

Stack Error Log:


[2024-08-21T09:28:52.956+0000] {subprocess.py:93} INFO - OSError: Can't load tokenizer for 'gpt2'. If you were trying to load it from 'https://huggingface.co/models', make sure you don't have a local directory with the same name. Otherwise, make sure 'gpt2' is the correct path to a directory containing all relevant files for a GPT2TokenizerFast tokenizer.
finaldie commented 2 months ago

Thanks for the reporting.

For the error message, do you mind sharing the entire stack? so that we could locate the source easily.

Also, I assume the ollama service has been started already (with llama3.1 model pulled), if not yet, please set up it first and use env var OLLAMA_URL point it to.

WiiCoder commented 2 months ago

Yes, I specified the OLLAMA_URL parameter, and ollama serve also received the request. The request was successful.

ollama log:

[GIN] 2024/08/22 - 17:57:15 | 200 |  3.143224337s | 183.250.214.101 | POST     "/api/embeddings"
[GIN] 2024/08/22 - 17:57:32 | 200 |   31.812767ms | 183.250.214.101 | POST     "/api/embeddings"
[GIN] 2024/08/22 - 17:57:45 | 200 |   29.386679ms | 183.250.214.101 | POST     "/api/embeddings"
[GIN] 2024/08/22 - 18:00:40 | 200 |  539.889733ms | 183.250.214.101 | POST     "/api/embeddings"
[GIN] 2024/08/22 - 18:00:41 | 200 |  337.827406ms | 183.250.214.101 | POST     "/api/embeddings"
[GIN] 2024/08/22 - 18:00:41 | 200 |  208.223058ms | 183.250.214.101 | POST     "/api/embeddings"

news pull stack error log:

3d5f028cbec1
*** Found local files:
***   * /opt/airflow/logs/dag_id=news_pulling/run_id=manual__2024-08-22T10:00:18.605448+00:00/task_id=save/attempt=1.log
[2024-08-22, 18:00:35 CST] {taskinstance.py:1979} INFO - Dependencies all met for dep_context=non-requeueable deps ti=<TaskInstance: news_pulling.save manual__2024-08-22T10:00:18.605448+00:00 [queued]>
[2024-08-22, 18:00:35 CST] {taskinstance.py:1979} INFO - Dependencies all met for dep_context=requeueable deps ti=<TaskInstance: news_pulling.save manual__2024-08-22T10:00:18.605448+00:00 [queued]>
[2024-08-22, 18:00:35 CST] {taskinstance.py:2193} INFO - Starting attempt 1 of 2
[2024-08-22, 18:00:35 CST] {taskinstance.py:2217} INFO - Executing <Task(BashOperator): save> on 2024-08-22 10:00:18.605448+00:00
[2024-08-22, 18:00:35 CST] {standard_task_runner.py:60} INFO - Started process 1449 to run task
[2024-08-22, 18:00:35 CST] {standard_task_runner.py:87} INFO - Running: ['***', 'tasks', 'run', 'news_pulling', 'save', 'manual__2024-08-22T10:00:18.605448+00:00', '--job-id', '267', '--raw', '--subdir', 'DAGS_FOLDER/news_pulling.py', '--cfg-path', '/tmp/tmploac2aw_']
[2024-08-22, 18:00:35 CST] {standard_task_runner.py:88} INFO - Job 267: Subtask save
[2024-08-22, 18:00:35 CST] {task_command.py:423} INFO - Running <TaskInstance: news_pulling.save manual__2024-08-22T10:00:18.605448+00:00 [running]> on host 3d5f028cbec1
[2024-08-22, 18:00:35 CST] {taskinstance.py:2513} INFO - Exporting env vars: AIRFLOW_CTX_DAG_EMAIL='' AIRFLOW_CTX_DAG_OWNER='***' AIRFLOW_CTX_DAG_ID='news_pulling' AIRFLOW_CTX_TASK_ID='save' AIRFLOW_CTX_EXECUTION_DATE='2024-08-22T10:00:18.605448+00:00' AIRFLOW_CTX_TRY_NUMBER='1' AIRFLOW_CTX_DAG_RUN_ID='manual__2024-08-22T10:00:18.605448+00:00'
[2024-08-22, 18:00:35 CST] {subprocess.py:63} INFO - Tmp dir root location: /tmp
[2024-08-22, 18:00:35 CST] {subprocess.py:75} INFO - Running command: ['/usr/bin/bash', '-c', 'cd ~/***/run/auto-news/src && python3 af_save.py --start 2024-08-22 --prefix=./run --run-id=manual__2024-08-22T10:00:18.605448+00:00 --job-id=267 --data-folder="data/news" --targets=notion --dedup=True ']
[2024-08-22, 18:00:35 CST] {subprocess.py:86} INFO - Output:
[2024-08-22, 18:00:36 CST] {subprocess.py:93} INFO - /home/***/.local/lib/python3.11/site-packages/langchain/__init__.py:29: UserWarning: Importing LLMChain from langchain root module is no longer supported. Please use langchain.chains.LLMChain instead.
[2024-08-22, 18:00:36 CST] {subprocess.py:93} INFO -   warnings.warn(
[2024-08-22, 18:00:36 CST] {subprocess.py:93} INFO - /home/***/.local/lib/python3.11/site-packages/langchain/chat_models/__init__.py:31: LangChainDeprecationWarning: Importing chat models from langchain is deprecated. Importing from langchain will no longer be supported as of langchain==0.2.0. Please import from langchain-community instead:
[2024-08-22, 18:00:36 CST] {subprocess.py:93} INFO - 
[2024-08-22, 18:00:36 CST] {subprocess.py:93} INFO - `from langchain_community.chat_models import ChatOpenAI`.
[2024-08-22, 18:00:36 CST] {subprocess.py:93} INFO - 
[2024-08-22, 18:00:36 CST] {subprocess.py:93} INFO - To install langchain-community run `pip install -U langchain-community`.
[2024-08-22, 18:00:36 CST] {subprocess.py:93} INFO -   warnings.warn(
[2024-08-22, 18:00:40 CST] {subprocess.py:93} INFO - /home/***/.local/lib/python3.11/site-packages/langchain/embeddings/__init__.py:29: LangChainDeprecationWarning: Importing embeddings from langchain is deprecated. Importing from langchain will no longer be supported as of langchain==0.2.0. Please import from langchain-community instead:
[2024-08-22, 18:00:40 CST] {subprocess.py:93} INFO - 
[2024-08-22, 18:00:40 CST] {subprocess.py:93} INFO - `from langchain_community.embeddings import HuggingFaceEmbeddings`.
[2024-08-22, 18:00:40 CST] {subprocess.py:93} INFO - 
[2024-08-22, 18:00:40 CST] {subprocess.py:93} INFO - To install langchain-community run `pip install -U langchain-community`.
[2024-08-22, 18:00:40 CST] {subprocess.py:93} INFO -   warnings.warn(
[2024-08-22, 18:00:40 CST] {subprocess.py:93} INFO - /home/***/.local/lib/python3.11/site-packages/langchain/embeddings/__init__.py:29: LangChainDeprecationWarning: Importing embeddings from langchain is deprecated. Importing from langchain will no longer be supported as of langchain==0.2.0. Please import from langchain-community instead:
[2024-08-22, 18:00:40 CST] {subprocess.py:93} INFO - 
[2024-08-22, 18:00:40 CST] {subprocess.py:93} INFO - `from langchain_community.embeddings import HuggingFaceInstructEmbeddings`.
[2024-08-22, 18:00:40 CST] {subprocess.py:93} INFO - 
[2024-08-22, 18:00:40 CST] {subprocess.py:93} INFO - To install langchain-community run `pip install -U langchain-community`.
[2024-08-22, 18:00:40 CST] {subprocess.py:93} INFO -   warnings.warn(
[2024-08-22, 18:00:40 CST] {subprocess.py:93} INFO - environment: environ({'CELERY_LOG_FILE': '', 'AIRFLOW__CELERY__RESULT_BACKEND': 'db+postgresql://***:***@postgres/***', 'MYSQL_PORT': '3306', 'AIRFLOW_CTX_DAG_OWNER': '***', 'TEXT_CHUNK_SIZE': '10240', 'REDDIT_PULLING_COUNT': '25', 'NOTION_ENTRY_PAGE_ID': '869e4e8986254061bc2249e2b41673ac', 'DUMB_INIT_SETSID': '0', 'PYTHONUNBUFFERED': '1', 'AUTOGEN_GPT4_API_KEY': '', 'OLLAMA_MODEL': 'llama3.1', 'AIRFLOW__WEBSERVER__SHOW_TRIGGER_FORM_IF_NO_PARAMS': 'true', 'HOSTNAME': 'appledeMBP', 'PYTHON_VERSION': '3.11.8', 'LANGUAGE': 'C.UTF-8', 'TWITTER_API_KEY': '', 'BOT_NETWORK_NAME': 'bot_network', 'CELERY_LOG_REDIRECT': '1', 'AIRFLOW_CTX_DAG_RUN_ID': 'manual__2024-08-22T10:00:18.605448+00:00', 'AIRFLOW_USER_HOME_DIR': '/home/***', 'CONTENT_SOURCES': 'RSS', 'REDDIT_FILTER_MIN_SCORES': '', 'ADDITIONAL_RUNTIME_APT_DEPS': '', 'OPENAI_API_KEY': 'sk-hL9sMycBk6ZW16b725B3989a1f7e4aA9Bf2f919e63743f9d', 'PIP_USER': 'true', 'AUTOGEN_GPT3_MODEL': 'gpt-3.5-turbo-0125', 'MILVUS_SIMILARITY_METRICS': 'L2', 'OPENAI_MODEL': 'gpt-4o-mini', 'REDDIT_APP_VERSION': '', 'PWD': '/home/***/***/run/auto-news/src', 'AIRFLOW__CELERY__BROKER_URL': 'redis://:@redis:6379/0', 'AIRFLOW_VERSION': '2.8.4', 'AIRFLOW__CORE__LOAD_EXAMPLES': 'false', '_MP_FORK_LOGFORMAT_': '[%(asctime)s: %(levelname)s/%(processName)s] %(message)s', 'AIRFLOW__API__AUTH_BACKENDS': '***.api.auth.backend.basic_auth,***.api.auth.backend.session', 'INSTALL_MSSQL_CLIENT': 'true', 'EMBEDDING_MAX_LENGTH': '5000', 'PYTHON_SETUPTOOLS_VERSION': '65.5.1', 'RSS_ENABLE_CLASSIFICATION': 'false', 'ACTION_DEEPDIVE_ITERATIONS': '1', 'AUTOGEN_GPT4_API_VERSION': '2023-08-01-preview', 'OLLAMA_URL': 'http://wzstest.nat300.top', 'INSTALL_MYSQL_CLIENT_TYPE': 'mariadb', 'MILVUS_HOST': 'milvus-standalone', '_AIRFLOW_PARSING_CONTEXT_DAG_ID': 'news_pulling', 'MYSQL_PASSWORD': 'bot', 'GUNICORN_CMD_ARGS': '--worker-tmp-dir /dev/shm', 'EMBEDDING_MODEL': 'nomic-embed-text', 'AIRFLOW_CTX_TRY_NUMBER': '1', 'MYSQL_USER': 'bot', 'HOME': '/home/***', 'LANG': 'C.UTF-8', 'TWITTER_ACCESS_TOKEN_SECRET': '', 'WORKDIR': '/opt/***', 'AUTOGEN_GPT4_MODEL': 'gpt-4-0125-preview', 'BOT_REDIS_URL': 'redis://:@redis:6379/1', 'LLM_PROVIDER': 'ollama', 'GOOGLE_MODEL': 'gemini-1.5-flash-latest', 'TEXT_CHUNK_OVERLAP': '256', 'AIRFLOW_HOME': '/opt/***', 'SUMMARY_MAX_LENGTH': '20000', 'EMBEDDING_PROVIDER': 'ollama', 'CELERY_LOG_REDIRECT_LEVEL': 'WARNING', 'GPG_KEY': 'A035C8C19219BA821ECEA86B64E628F8D684696D', 'AIRFLOW_CTX_TASK_ID': 'save', 'NOTION_TOKEN': 'secret_Hd05AwlTOW7fAEhIEACIMeEL4flJfTJfRW0GD7mxEBL', 'AIRFLOW__DATABASE__SQL_ALCHEMY_CONN': 'postgresql+psycopg2://***:***@postgres/***', 'AIRFLOW_CTX_DAG_ID': 'news_pulling', 'AIRFLOW__CORE__EXECUTOR': 'CeleryExecutor', 'COMMIT_SHA': 'effef9add822bad13a1dd8a61ba3a0305b583217', 'AIRFLOW_PIP_VERSION': '24.0', 'AIRFLOW__CORE__DAGS_ARE_PAUSED_AT_CREATION': 'true', 'MYSQL_HOST': 'mysql-db', 'TWITTER_ACCESS_TOKEN': '', 'AIRFLOW_CTX_EXECUTION_DATE': '2024-08-22T10:00:18.605448+00:00', 'AUTOGEN_GPT3_API_KEY': '', 'ADDITIONAL_RUNTIME_APT_COMMAND': '', '_PIP_ADDITIONAL_REQUIREMENTS': '', 'REDDIT_ENABLE_CLASSIFICATION': 'false', 'AUTOGEN_GPT3_API_VERSION': '2023-08-01-preview', 'INSTALL_POSTGRES_CLIENT': 'true', 'BOT_REDIS_KEY_EXPIRE_TIME': '604800', 'SHLVL': '0', 'REDDIT_CLIENT_ID': '', 'MYSQL_DATABASE': 'bot', 'LC_MESSAGES': 'C.UTF-8', 'RUNTIME_APT_DEPS': '', 'PYTHON_PIP_VERSION': '24.0', 'CELERY_LOG_LEVEL': '20', 'AIRFLOW_CTX_DAG_EMAIL': '', 'RUNTIME_APT_COMMAND': 'echo', '_AIRFLOW_PARSING_CONTEXT_TASK_ID': 'save', 'LD_LIBRARY_PATH': '/usr/local/lib', 'TWITTER_API_KEY_SECRET': '', 'LC_CTYPE': 'C.UTF-8', 'AIRFLOW__SCHEDULER__ENABLE_HEALTH_CHECK': 'true', 'MILVUS_PORT': '19530', 'PYTHON_GET_PIP_SHA256': 'dfe9fd5c28dc98b5ac17979a953ea550cec37ae1b47a5116007395bfacff2ab9', 'GOOGLE_API_KEY': 'AIzaSyBE4Pub-w-QG7veOZjXZ24orc_ZejJYpZY', '_MP_FORK_LOGFILE_': '', 'AIRFLOW__CORE__SQL_ALCHEMY_CONN': 'postgresql+psycopg2://***:***@postgres/***', 'AIRFLOW_INSTALLATION_METHOD': '', 'TRANSLATION_LANG': 'Chinese', 'LC_ALL': 'C.UTF-8', 'PYTHON_GET_PIP_URL': 'https://github.com/pypa/get-pip/raw/dbf0c85f76fb6e1ab42aa672ffca6f0a675d9ee4/public/get-pip.py', 'INSTALL_MYSQL_CLIENT': 'true', 'REDDIT_APP_ID': '', '_MP_FORK_LOGLEVEL_': '20', 'PATH': '/root/bin:/home/***/.local/bin:/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin', 'REDDIT_CLIENT_SECRET': '', 'PYTHON_BASE_IMAGE': 'python:3.11-slim-bookworm', 'AIRFLOW_UID': '50000', 'ACTION_DEEPDIVE_ENABLED': 'false', 'BUILD_ID': '', 'AIRFLOW__CORE__FERNET_KEY': '', 'DEBIAN_FRONTEND': 'noninteractive', 'OLDPWD': '/tmp/***tmpcup_zf4o', '_': '/usr/local/bin/python3'})
[2024-08-22, 18:00:40 CST] {subprocess.py:93} INFO - Pushing data for source: RSS ...
[2024-08-22, 18:00:40 CST] {subprocess.py:93} INFO - #####################################################
[2024-08-22, 18:00:40 CST] {subprocess.py:93} INFO - # Process RSS, dedup: True
[2024-08-22, 18:00:40 CST] {subprocess.py:93} INFO - #####################################################
[2024-08-22, 18:00:40 CST] {subprocess.py:93} INFO - Retrieve data from /opt/***/data/news/manual__2024-08-22T10:00:18.605448+00:00/rss.json, data: {'618d9bf1878e326d398b040b539da73e': {'id': '618d9bf1878e326d398b040b539da73e', 'source': 'RSS', 'list_name': '少数派', 'title': '把女朋友拍好看，技巧可能没有你以为的那么重要', 'url': 'https://sspai.com/post/90666', 'created_time': '2024-08-22T09:00:00', 'summary': '写在前面本文会出现三位女性模特。但其中只有一位是女朋友。谢谢她们给予的信任、鼓励和支持，以及愿意在镜头前展现美好的自己。我很喜欢拍照，也自认为拍摄不同题材照片都积累了一些经验。但是对于我来说，拍人比拍 ...<a href="https://sspai.com/post/90666" target="_blank">查看全文</a>', 'content': '', 'tags': [], 'published': 'Thu, 22 Aug 2024 17:00:00 +0800', 'published_key': '2024-08-22'}, '7a055a88d019d31349b41989d5742a2c': {'id': '7a055a88d019d31349b41989d5742a2c', 'source': 'RSS', 'list_name': '少数派', 'title': '键盘鼠标都最低，派商店 Keychron 系列清仓促销专场', 'url': 'https://sspai.com/post/91677', 'created_time': '2024-08-22T08:09:02', 'summary': '活动时间：即日起至8.29省流版：少数派定制KeychronK3史低¥338起售少数派定制KeychronQ1系列史低仅需¥769KeychronM系列鼠标¥179起购买K3系列赠送捣蛋獭贴纸我们曾与 ...<a href="https://sspai.com/post/91677" target="_blank">查看全文</a>', 'content': '', 'tags': [], 'published': 'Thu, 22 Aug 2024 16:09:02 +0800', 'published_key': '2024-08-22'}, '0ea87107cd73d55bd838483defe8a14a': {'id': '0ea87107cd73d55bd838483defe8a14a', 'source': 'RSS', 'list_name': '少数派', 'title': '一派 | 天命人，来聊聊你的《黑神话：悟空》游玩体验', 'url': 'https://sspai.com/post/91709', 'created_time': '2024-08-22T07:20:54', 'summary': '「上香」之余想来点新鲜空气？本文所述的游戏体验应该能为你提供一些参考。<a href="https://sspai.com/post/91709" target="_blank">查看全文</a>', 'content': '', 'tags': [], 'published': 'Thu, 22 Aug 2024 15:20:54 +0800', 'published_key': '2024-08-22'}}
[2024-08-22, 18:00:40 CST] {subprocess.py:93} INFO - #####################################################
[2024-08-22, 18:00:40 CST] {subprocess.py:93} INFO - # Dedup RSS
[2024-08-22, 18:00:40 CST] {subprocess.py:93} INFO - #####################################################
[2024-08-22, 18:00:40 CST] {subprocess.py:93} INFO - Number of pages: 3
[2024-08-22, 18:00:40 CST] {subprocess.py:93} INFO - [INFO] Initialized default DB driver (Redis)
[2024-08-22, 18:00:40 CST] {subprocess.py:93} INFO - Dedupping page, title: 把女朋友拍好看，技巧可能没有你以为的那么重要, list_name: 少数派, created_time: 2024-08-22T09:00:00, page_id: 618d9bf1878e326d398b040b539da73e
[2024-08-22, 18:00:40 CST] {subprocess.py:93} INFO -  - No duplicate RSS article found, move to next. title: 把女朋友拍好看，技巧可能没有你以为的那么重要, page_id: 618d9bf1878e326d398b040b539da73e
[2024-08-22, 18:00:40 CST] {subprocess.py:93} INFO - Dedupping page, title: 键盘鼠标都最低，派商店 Keychron 系列清仓促销专场, list_name: 少数派, created_time: 2024-08-22T08:09:02, page_id: 7a055a88d019d31349b41989d5742a2c
[2024-08-22, 18:00:40 CST] {subprocess.py:93} INFO -  - No duplicate RSS article found, move to next. title: 键盘鼠标都最低，派商店 Keychron 系列清仓促销专场, page_id: 7a055a88d019d31349b41989d5742a2c
[2024-08-22, 18:00:40 CST] {subprocess.py:93} INFO - Dedupping page, title: 一派 | 天命人，来聊聊你的《黑神话：悟空》游玩体验, list_name: 少数派, created_time: 2024-08-22T07:20:54, page_id: 0ea87107cd73d55bd838483defe8a14a
[2024-08-22, 18:00:40 CST] {subprocess.py:93} INFO -  - No duplicate RSS article found, move to next. title: 一派 | 天命人，来聊聊你的《黑神话：悟空》游玩体验, page_id: 0ea87107cd73d55bd838483defe8a14a
[2024-08-22, 18:00:40 CST] {subprocess.py:93} INFO - #####################################################
[2024-08-22, 18:00:40 CST] {subprocess.py:93} INFO - # Scoring RSS
[2024-08-22, 18:00:40 CST] {subprocess.py:93} INFO - #####################################################
[2024-08-22, 18:00:40 CST] {subprocess.py:93} INFO - start_date: 2024-08-22, max_distance: 0.5
[2024-08-22, 18:00:40 CST] {subprocess.py:93} INFO - [INFO] Initialized default DB driver (Redis)
[2024-08-22, 18:00:40 CST] {subprocess.py:93} INFO - Scoring page: 把女朋友拍好看，技巧可能没有你以为的那么重要, score_text: 把女朋友拍好看，技巧可能没有你以为的那么重要 - 少数派 - 写在前面本文会出现三位女性模特。但其中只有一位是女朋友。谢谢她们给予的信任、鼓励和支持，以及愿意在镜头前展现美好的自己。我很喜欢拍照，也自认为拍摄不同题材照片都积累了一些经验。但是对于我来说，拍人比拍 ...<a href="https://sspai.com/post/90666" target="_blank">查看全文</a>
[2024-08-22, 18:00:40 CST] {subprocess.py:93} INFO - [EmbeddingAgent] provider: ollama, model_name: nomic-embed-text
[2024-08-22, 18:00:40 CST] {subprocess.py:93} INFO - Initialized EmbeddingOllama: model_name: nomic-embed-text, base_url: http://wzstest.nat300.top
[2024-08-22, 18:00:40 CST] {subprocess.py:93} INFO - [get_relevant] collection_name: embedding__news__ollama_nomic_embed_text__2024_08_22
[2024-08-22, 18:00:40 CST] {subprocess.py:93} INFO - [MilvusClient] server.alias: default, host: milvus-standalone, port: 19530
[2024-08-22, 18:00:40 CST] {subprocess.py:93} INFO - [get_relevant] Fallback collection name: embedding__news__ollama_nomic_embed_text__2024_08_21
[2024-08-22, 18:00:40 CST] {subprocess.py:93} INFO - [EmbeddingOllama] Embedding not found, create a new one and cache it
[2024-08-22, 18:00:40 CST] {subprocess.py:93} INFO - [Redis Client] Set key: milvus_embedding_item_id_ollama-norm_nomic-embed-text_default_a6072bc0ed812e3b308391987c409a2b, val: [0.05974891904322014, 0.04252539413432246, -0.12833233642723224, 0.010076228414382889, 0.014879058901815217, 0.03167659085312175, 0.0036107572177843856, -0.026074086744390032, -0.026247950276199288, -0.029200013568753645, -0.08672753899578509, 0.007976299496765855, -0.015507847884781977, 0.009468254384306514, -0.015627294575125676, -0.07311486269321446, 0.013129103340894115, -0.015716459951616083, -0.08705313585752766, 0.059503565052087565, -0.05813762787761591, 0.06897084512262487, -0.057624843407557046, -0.05225789345406786, 0.07609325483840519, -0.009392853987414705, -0.00707553671875144, 0.018257838800012888, 0.004178058497184048, -0.03815236359005884, 0.01367432514838439, -0.04589788650078434, -0.010808823591390458, -0.005234298178251174, -0.01659840340410049, -0.016331718954093973, -0.00048753008557417575, 0.05277834710142191, 0.04997317988728511, 0.04634976516983352, 0.049235115572266744, -0.03917028424169112, -0.0009704082145655369, -0.04653202600316367, 0.0328886401661398, 0.003912549415740622, 0.04068167915082303, -0.016286018714526958, 0.11846163375804238, 0.0007047636822463654, -0.036498796457629316, -0.0018304414504517247, 0.06668081087758636, 0.0035044495887755996, 0.0472293899644353, 0.009110594691650237, -0.005438920482422375, -0.006330125510931213, -0.026875772405686033, -0.044360577531095444, 0.01635614990915846, 0.049651298249562964, -0.033491076423544756, 0.07086013053665927, 0.05538223272889021, -0.00710710120234673, -0.05962235672903971, 0.04792468891765701, 0.03624653722281539, 0.0033045724015485687, 0.03789368253830357, 0.010204385738393775, 0.042487292945332183, 0.04029909761607306, 0.0008892452105366121, -0.013849298771226258, -0.03724204119746655, -0.012040127910827664, -0.020465681546902963, -0.01828233391356448, -0.0005442241803173173, -0.06708012136483316, 0.08127098201668498, 0.012007346654064237, 0.045771995612631705, 0.041911842046013, 0.025029027052714516, -0.02621183054001991, -0.04856961301410047, 0.028444677192184853, 0.05588853569969414, 0.039708929058224735, 0.0338887128218956, 0.016099809896153415, 7.805694176323464e-05, 0.0037958477388071988, -0.004165543489040517, 0.023070417647339796, -0.05088891248160355, 0.023751558463722202, -0.032618130079832726, -0.0368071331942842, 0.02832039474238323, 0.042274916416568935, 0.011339778341819132, 0.0062832808630012785, -0.01348077988161387, 0.052534294184725404, -0.01686622181806505, 0.03647812548832045, 0.0037921862288689883, 0.04421001695862399, -0.05733314013169726, -0.007662047496806152, 0.031516669109765076, -0.01746245857514675, 0.0603496871799605, -0.00996734549354307, 0.004294137985998386, 0.0334812497306136, -0.0003484958697337376, -0.032243283372922885, -0.014006322938258561, 0.0766463964889077, -0.004825852566881841, 0.024881981768512997, -0.08918376461122197, 0.03427816379093878, -0.005494388478635882, -0.09390054140878824, -0.0012801807771301225, 0.03560803792742896, -0.0025710601684913762, -0.013420111317800677, -0.020956798353016143, 0.0006583086945465868, -0.06479408583480459, -0.019022778060836906, -0.010949173266037077, 0.004092289417350874, 0.008536951420403645, 0.007629862317149616, 0.003934663577994779, -0.06985515199472708, -0.027868886103678256, 0.009829776268680418, 0.01625986592471565, -0.010439120007853483, -0.07640423550155172, 0.02087721198785553, -0.019343600337492987, 0.03526247136356209, -0.0058142211479298945, -0.009694656256761358, 0.005779304383383636, -0.0484060416972683, 0.01849539529687607, -0.011540977119277528, 0.00579365797979993, 0.02624281461311562, 0.033152549388564795, 0.04889023237089037, -0.020380861042841272, 0.019305117181695787, -0.019411474048613278, -0.0053709121130820665, -0.05889309257115889, 0.03641635727787952, 0.008426131808488117, 0.019998036302663454, -0.06866866163250021, -0.03515665760569819, -0.04176038623883782, 0.00710254594976264, 0.005839684234004881, -0.023465620499354366, 0.014889594322220149, -0.06344782591912067, -0.008261827145227817, -0.026143550989167263, -0.022715964386980675, -0.022069076742420396, 0.05207450164089535, -0.02768428273954005, -0.04493420513573472, -0.04822164125299627, 0.0034387968057494507, 0.003645849299126789, -0.029463916822947276, -0.04271093749489957, -0.05889411313872114, -0.007458430093589875, -0.0556708563973692, -0.001643741372034517, -0.03621251830407407, -0.05403412862971663, 0.02672425094796592, 0.04398248410632678, 0.015589767828403617, -0.023063676530020793, -0.03949591989693753, 0.013800636796847775, -0.0017400137930290735, 0.01539023194940738, -0.03301928176645351, 0.0376511216794466, 0.014969047894233046, 0.05457560835614536, -0.010117700162032669, -0.014573706280839399, -0.000659614058648947, -0.03499960956574045, -0.026266305571741207, -0.01120714932046317, 0.024966402401073697, -0.010157796232354402, 0.01342009788928012, -0.026578691753372578, -0.049227780615927606, 0.0050921848912382295, 0.003920779606783478, -0.052336709917383276, -0.02717418397359235, -0.02533522268636967, 0.01942752262273528, -0.005377920308754346, 0.00595298998729553, -0.022076175953620887, 0.036652654477926136, -0.01543077862119665, -0.06241786436072338, -0.0760853588683184, 0.04991611464315439, 0.04865215962201471, 0.0006155987253972251, 0.04441550913251564, 0.019602773768335302, 0.07766621205399618, -0.0076059334395481275, 0.011503937529440076, -0.019052974327393345, 0.009451338178550825, 0.012186378674229632, -0.03427939623071423, 0.00866231038993649, 0.00839805724817949, 0.02355935455694942, -0.03903614824191723, -0.01372034916450258, -0.016001454935429333, 0.008748822887588211, 0.009666671219923104, 0.08444966800625443, 0.059148688047302014, -0.014051001118205496, -0.01766740167175345, -0.04206468845109461, 0.0005353219570981745, 0.03244680901458203, -0.009637941646222923, 0.03517572312077172, -0.1082134165825256, 0.02069083605900396, -0.04690949873187108, 0.030514164399730842, -0.037862313514285255, -0.03078023710602315, -0.013302829603382135, -0.011137882027320755, 0.03722309206290449, 0.014826323609534812, 0.04802538193301564, -0.009943674741947822, 0.020435797120434984, -0.025371136518567195, -0.03489513865993213, -0.00249921814303962, 0.0011993476111373492, -0.031214193176303875, 0.04758401928761948, 0.03847118650920995, 0.020215209797380795, 0.0002212474187677595, -0.013278282267806162, -0.02014533374463985, 0.04528124883686273, -0.02936824309016083, 0.017171590851677104, 0.03730669803188481, 0.030439489888977964, 0.01617914261148131, -0.016657597814768225, 0.023794716236730653, 0.0654162679856579, 0.0427105346392829, 0.029439727581252788, 0.01631044370136009, -0.00010590459854158439, -0.053154357613446584, 0.03555937893716616, -0.00887464737916699, 0.024675219853399908, 0.039511461171394084, -0.0014559864087247605, 0.013764130617687956, -0.0426616607926912, 0.08038804186956378, 0.010134177702783536, 0.03412517713242025, -0.02666733789373483, -0.09527384338075583, -0.01896908934359704, 0.005712253542161846, 0.030773994336022552, -0.06438715988434482, 0.04420756998376716, 0.039403051232889404, 0.005983326880303346, 0.009967064986669239, 0.03076258606178149, 0.01739750631327626, -0.05087559735744357, 0.012634663721972297, -0.05919275746765042, 0.01735688951476853, 0.05299230819561074, 0.007681345772902653, 0.044648795359842085, -0.01894083424428983, -0.0077651084079561165, -0.0034483284442428383, -0.004143881420311738, 0.02109613268230296, -0.01986474779174242, 0.024501624892001767, 0.07420463189805154, -0.0021976061110675076, 0.01816737533320209, -0.019602978180259318, 0.06525748915860631, 0.026760925729661607, 0.010782755102847083, 0.03617420524287058, -0.0369237942126415, 0.003657341874635781, 0.02784545781148193, -0.05782257091245177, 0.012908016178463683, 0.03207622959309325, 0.03695569440925068, 0.0064489731400523415, 0.05119442027659468, -0.025226389003438557, -0.021206653882650583, 0.0015265295045735893, 0.0009812358917997877, -0.01168386030813567, 0.0724426727318114, -0.014613017528737537, 0.0006894540963961991, 0.06894003114412287, 0.01863715123603645, 0.016517216060878057, 0.05908212436296483, -0.02908767354989963, 0.059442671187540604, 0.0504227339302203, 0.03172409797473244, 0.07150406092258159, 0.021000218728204352, -0.036612741930718666, -0.035437206257149634, -0.004298359017626421, 0.007202234810195264, -0.009096957282996919, 0.022348046796677616, -0.03613592500693956, 0.018421105737045793, -0.03264885453486436, 0.06449734537167855, 0.03925277116729178, -0.011041167584191236, -0.006582753284031503, 0.051046903502115874, 0.025013015780038502, -0.021745285270663647, 0.0248326095746029, 0.0056748592155730455, -0.06003548964011193, -0.0008163699350309804, -0.03429654594352181, -0.052637073096943696, -0.0613178178614876, -0.027428791707445697, 0.027065275687769258, -0.07874954153719783, 1.3600142177436022e-05, 0.022370072554504784, 0.04650558972236284, 0.03341277024401046, -0.05455876600725273, -0.04850000254765496, -0.005863186010049802, 0.018616548901388197, -0.005330454218734248, 0.03893265015782028, -0.03524131696751321, -0.044960593670684675, 0.09224521663222375, 0.010285417161571976, -0.020312036890877197, 0.006734470201328593, -0.02477211558155693, 0.0019886773549569583, -0.002833653955423588, 0.04132434128758176, 0.0321584330277044, -0.03156495210145232, -0.00450343416135192, -0.0013522354208240117, 0.013182774153439904, 0.041655360287513196, -0.004990894306714868, -0.007702216677962021, 0.040258298786508905, 0.0598898289855855, -0.006574117253256294, 0.022192281926461877, -0.009310054475696659, -0.06249172719201155, 0.006081318524943502, 0.04799551988741526, -0.022576448026637434, -0.018240939752922354, -0.012013877391227867, 0.0025078120231808625, 0.08688358437287465, 0.015918080235414148, -0.06763988775214318, 0.027491458136706025, 0.01616873700010842, -0.02232981384987854, -0.04988911436449023, -0.021563343737711165, 0.02338416309366298, 0.039377408726859214, 0.03590575718049774, -0.08147100729065258, -0.00910912352262046, 0.020415860243583165, 0.03956604363127983, 0.0011621714047539652, -0.006442828099840231, 0.028239686349727285, 0.06560790192633598, -0.030237575669785514, 0.007246086390099098, -0.05329895592279127, -0.035536392294090335, 0.05842525485145809, 0.037243210970812735, 0.03573032997207265, -0.006108742175019633, -0.004561926185791698, -0.020622416254714396, -0.03432416095001586, 0.009223269677489224, -0.015086050593835594, 0.03248315554355302, 0.05778643327157832, -0.030554127676904855, 0.030549666423964655, 0.060258874571615245, 0.01680372249928275, 0.02484219455416405, 0.008555882650271622, -0.05566056119827643, -0.028690884640401638, -0.04000538901447694, 0.0656863901369267, 0.007380024692209249, 0.0046483726598840185, -0.0051275030193433215, -0.06548407903034893, 0.005995849348736075, -0.005689263168941423, -0.03490242288630472, 0.013230421528487506, 0.03211254031267609, -0.006983675939773918, -0.006139679248293401, -0.030895549304093512, -0.02591956625041246, -0.001838735053952761, 0.005670244280675374, -0.020787248360420972, -0.03792554394140893, 0.0389857107187097, 0.024686641556161522, 0.0192014892885668, -0.041809149673149394, -0.026310712197161346, -0.004193387899427396, 0.0018681783908219276, 0.047494886736401544, -0.020622450572044707, -0.04035844272458259, -0.07842836116666027, -0.046063134890628146, -0.007755622650241295, 0.006667158996013805, -0.04142478363722334, -0.009070841794631598, -0.012489808032650294, 0.03250599894907625, -0.008907416699467702, 0.056392385727409886, -0.04642218070501735, -0.044566678464585625, 0.022514714133526813, 0.001798841716989975, -0.03151481597392838, 0.008480767982456084, 0.0051694798285717835, -0.07110794940734275, 0.002357636774646468, 0.013406270989281181, -0.06386540516255469, 0.06284097615462639, -0.025781327091579858, 0.012688146535230307, 0.018131012391594866, -0.010917131323933137, -0.004921608735863919, -0.04040013977296614, 0.00279905462618269, -0.0010569828191248278, 0.06898940632214864, -0.006512393804550551, 0.04676168354582437, -0.05824687635262641, 0.03247843467254874, 0.01897288364668297, 0.029905046707320467, 0.018617803722031243, 0.04083474936458058, 0.005302363245789386, 0.002002582401485243, -0.008592684256883669, -0.018577470414513023, 0.09547993834601465, -0.047946962357085544, -0.06399736275788287, -0.00866193066121633, 0.03810330174417936, -0.08351145010029426, -0.015926710297958, 0.003380775898572506, -0.027074046003750026, -0.07541732876417195, -0.020462236385351485, 0.0065957893933754905, -0.019374400887474438, -0.032973338321458656, 0.0536021540122468, -0.0544080442925346, -0.003821005326039541, 0.007372527847594523, 0.012221260002402201, -0.012905018634264064, 0.014401321378349118, -0.04875929534312444, -0.04544218517955029, -0.04890332368637442, -0.008540587565358585, 0.00671196101676142, -0.004035037665990208, 0.013102417886434093, -0.09339717480781251, -0.010995468090653144, 0.011700647450888241, 0.004423436361242055, -0.020042951719806874, 0.03390035982539098, 0.008792736387892386, 0.05116499092776776, -0.04178666436150765, 0.004079266736527443, 0.04688115858526702, -0.01783498214800044, -0.046077031917345544, 0.034261393060822445, 0.00788059965296722, -0.012897091330962632, 0.030401895999653126, -0.012201062015428463, 0.04030390502643203, -0.04007879527606622, -0.01752414770652218, -0.022035054839563396, 0.0038591255387672736, 0.0366635196431136, 0.038986024050856, -0.03286367803848444, 0.0577701161270452, 0.06507847802725067, 0.0027398221680400457, 0.03967205434077855, 0.0039662019505778785, -0.030919965338579606, 0.023089317543993144, 0.06103722146418529, -0.04623265355000869, 0.01565560040439943, 0.038241346935492825, -0.03441174176108068, 0.07473279650032029, -0.0010425187171747936, -0.053809418750851816, 0.03358267087019792, 0.021411935676386856, -0.014583692623959383, 0.02852209708936247, -0.01896909083565488, 0.023508484336968432, 0.004390045971865637, -0.05232990613363501, -0.02523061152712443, -0.0683213045992259, 0.029148495795670297, -0.053093768128698884, -0.0025038241255902543, -0.05166438865489034, -0.06390096985321778, -0.071832104759181, 0.01591231044774868, 0.025903153614177613, -0.026639534887952995, 0.04808099689692635, 0.01591059308917538, 0.04755241750257821, 0.0021330266771496715, 0.03554755587084571, 0.05199371566123175, 0.06530025153628728, -0.015610517876777984, -0.007704829271239041, 0.037380739910113724, 0.025984276798912947, -0.029092066168179213, 0.05376989115456694, 0.07481503872843527, 0.01324735787702403, 0.03566290089418854, -0.015895824700679694, -0.01973740214719624, 0.05683581338085596, -0.05111628419165409, -0.07801362876958406, -0.03137670513206998, 0.010165675789804967, -0.002036873620782278, 0.014258421030825818, -0.0493480195889841, 0.008349632510997492, -0.024066134986262002, 0.022503529667961684, -0.013760804820763639, -0.08050360473335118, -0.02057192203330897, 0.032967188059044104, -0.005201777658595204, -0.012315214138528586, -0.010193292288356857, 0.03807187602196315, 0.018081225405605375, 0.024958175194146517, 0.02832741039834471, -0.00755381511318437, 0.015637298822939738, -0.023484036969267708, 0.00660713201707161, 0.03489979089627579, -0.026940074130338484, -0.03239956747926678, -0.04597248640864524, 0.08086765490973416, -0.006997760219750177, -0.0010905530947245238, -0.017899701648847962, -0.054919354609448, -0.05688427541948394, -0.03241939394383848, -0.04376484956573703, -0.028330749623789578, -0.002445059614087533, -0.0035564138601674265, 0.03941092034593509, -0.03266479269670624, 0.04422699359272218, 0.012272747188299838, 0.008503409960170976, 0.011035707398527472, -0.0035296642472203038, -0.0038537600987763178, -0.0054054372124454474, -0.04007144838326437, 0.014494561564799285, -0.0011838114656296116, -0.01748545118645393, 0.009391541722544838, -0.022862855989224717, -0.046096789747256624, 0.03051878679491771, -0.00458622248263617, 0.0071258951628934706, -0.005422416084630397, 0.0010400191472791185, -0.002062961319570338, 0.024800354268227905, 0.00705842952959229, 0.019348161558308437, -0.06728404389387761, -0.03881352724402776, -0.060109131646839856, 0.0018927951070874395, -0.026235339403339566, 0.003910507534587223, 0.06684279464487726, -0.002446275454719521, 0.0049824369498659435, 0.006966526972995262, -0.0036780687961136343, 0.012138212063053874, 0.02716778006134508, 0.015387198595819612, -0.0031671833426700404, -0.013013955269186105, -0.03649660014848953, -0.03711221127661754, -0.024537632723843184, 0.0205760938270283, 0.01805247643515328, 0.003916671225522327, 0.024560334383871656, -0.02199783694881412, -0.016650938760630396, 0.06420839345047485, -0.00048581002264604074, -0.02051076258246657, 0.020875496121340067, 0.0668728924356163, -0.018483054486485306, 0.026780844701819354, 0.038183968359215795, 0.019848136711814915, -0.04221686343014844, 0.08603255635882534, 0.07358767791786794, 0.010360692225605265, 0.023934451929576304, -0.0520123663842259, -0.020609462208551584, 0.00014803717627715298, -0.01560495399309437, 1.509337588676888e-05, -0.05399539480820239, -0.029411548576895395], expired_time: 2592000, overwrite: False
[2024-08-22, 18:00:41 CST] {subprocess.py:93} INFO - [Milvus Client] get relevant results: data: ['[]'], cost: 0
[2024-08-22, 18:00:41 CST] {subprocess.py:93} INFO - [get_relevant] metric_type: L2, max_distance: 0.5, raw emb response_arr size: 0, post emb_utils.topk: 0
[2024-08-22, 18:00:41 CST] {subprocess.py:93} INFO - relevant_page_metas(0): []
[2024-08-22, 18:00:41 CST] {subprocess.py:93} INFO - RSS article scored -1
[2024-08-22, 18:00:41 CST] {subprocess.py:93} INFO - Scoring page: 键盘鼠标都最低，派商店 Keychron 系列清仓促销专场, score_text: 键盘鼠标都最低，派商店 Keychron 系列清仓促销专场 - 少数派 - 活动时间：即日起至8.29省流版：少数派定制KeychronK3史低¥338起售少数派定制KeychronQ1系列史低仅需¥769KeychronM系列鼠标¥179起购买K3系列赠送捣蛋獭贴纸我们曾与 ...<a href="https://sspai.com/post/91677" target="_blank">查看全文</a>
[2024-08-22, 18:00:41 CST] {subprocess.py:93} INFO - [EmbeddingAgent] provider: ollama, model_name: nomic-embed-text
[2024-08-22, 18:00:41 CST] {subprocess.py:93} INFO - Initialized EmbeddingOllama: model_name: nomic-embed-text, base_url: http://wzstest.nat300.top
[2024-08-22, 18:00:41 CST] {subprocess.py:93} INFO - [get_relevant] collection_name: embedding__news__ollama_nomic_embed_text__2024_08_22
[2024-08-22, 18:00:41 CST] {subprocess.py:93} INFO - [MilvusClient] server.alias: default, host: milvus-standalone, port: 19530
[2024-08-22, 18:00:41 CST] {subprocess.py:93} INFO - [get_relevant] Fallback collection name: embedding__news__ollama_nomic_embed_text__2024_08_21
[2024-08-22, 18:00:41 CST] {subprocess.py:93} INFO - [EmbeddingOllama] Embedding not found, create a new one and cache it
[2024-08-22, 18:00:41 CST] {subprocess.py:93} INFO - [Redis Client] Set key: milvus_embedding_item_id_ollama-norm_nomic-embed-text_default_f2cc6b382351274ca0b8c587fb3b36da, val: [0.02395702812536385, 0.060698208871675645, -0.15517868433338142, -0.05737696603735675, -0.02337336503968691, -0.02287401161467209, -0.03423255738281064, -0.02159392573578011, 0.001565746113412487, -0.0329905444513069, -0.043726650784983265, 0.03347677489971143, -0.010631406497944477, -0.008471398563581948, 0.0193570769706876, -0.05881695952661658, 0.01696817960514657, 0.049350014032191115, -0.036772987726331925, 0.05644927040582258, -0.017529404648010376, -0.005868141931059612, -0.1088785818185061, -0.01833151619467629, 0.10464505969486845, -0.0038688712205287965, 0.027333048535871297, -0.01606502760310192, -0.02281833054478541, -0.024796741632924606, 0.07961178895279147, -0.044312484102541565, -0.026394778795808774, -0.025154897616897723, -0.04906842259765528, -0.08065155474301067, 0.0031932033860401763, 0.06781838566497483, 0.0009402500766542668, 0.007357140718413604, 0.11152612926667393, 0.03163605501982972, -0.002495505773991278, 0.013413998548894619, 0.04558800172028185, 0.05322530321327829, 0.007481765129822266, 0.014021274065275108, 0.09973743584379131, -0.07787091205282488, -0.00787747920830113, -0.010774252391713087, 0.06214294916598662, -0.03666334330200509, 0.03707692485338807, 0.03662858573012923, -0.0199402121205239, 0.0006331460560409174, 0.055176689527394095, -0.034477519612869385, 0.011493614228079835, 0.0517698687550583, 0.014290630625183312, -0.0006429527432557197, 0.06991856061420247, -0.002469734462796525, -0.0009017384653751132, 0.06440042744732447, 0.0280151139368953, -0.010750515288114226, 0.008869126648030148, 0.010618413581329913, 0.04131768108329694, 0.04372868249142539, -0.024476844836096922, 0.010827202971275273, -0.047544892111849965, -0.025496810723534644, 0.028339531280556718, 0.049986353724894864, 0.009339676016713164, -0.010393650669069098, 0.05495167496058533, -0.01875861321141407, 0.029800974664496462, -0.006720538596961081, -0.018379804623622414, -0.01721532745880215, 0.03756968754583836, 0.012512223335798819, 0.024540615791634985, 0.028288273789696304, -0.023427519251398903, 0.03784247184078188, -0.009597203273696387, 0.007128964693247167, -0.010992246797095678, 0.05281866640391567, 0.00663364774561453, 0.025249194961728758, -0.04741889552900709, -0.04543550676008468, -0.02689279621992074, 0.038304528060868176, -0.019679214801287814, 0.06707882604998171, 0.019976250283126915, -0.006013860688520822, -0.054441371869184685, -0.011423331037726131, -0.04898866580642938, 0.031125003891057195, -0.007953174125814318, 0.0005396376809250858, 0.0511967274460771, -0.024794154285553986, 0.03330786929247943, -0.012724723668344156, 0.021673696379549932, -0.015775571001960576, -0.03721783600852203, -0.0702049327155635, -0.021461256844280702, 0.020760983043854903, -0.013817663217999855, 0.015104329055255545, -0.0442550114369741, -0.012353211523271368, 0.00581270481986314, -0.023337371513058758, 0.006953987823847767, 0.007563017226206139, -0.020542997873501594, 0.048865627511299976, 0.02549779425415322, 0.07315088858659043, -0.027472873035893016, -0.026383001055130622, 0.00499360543683225, 0.021799143478150502, 0.015239237441355934, -0.002466003703362879, -0.01866573036523479, -0.055129553937936726, -0.003113280171890752, -0.008045449768818281, 0.010159701211429614, -0.061045015004660697, -0.046638960365991314, -0.011768740372532217, 0.027152131233885993, -0.02800914810797887, 0.009506194368875658, -0.004866626092956575, 0.022788147390747433, -0.07639166353242867, 0.022296756100145587, -0.002413030998209704, 0.008311260077482044, 0.02486597972579785, -0.004146001751128347, 0.023099261674727554, -0.020390907715421302, 0.03462713324393045, 0.028223530077740495, -0.06456819714595473, 0.02565011682713712, 0.03788395867066151, 0.04490813733457016, 0.025000872720182764, -0.07280809045631394, -0.0019609322560507065, -0.051786534904569935, -0.026327052169394752, 0.030751308431144892, -0.07516663689811835, 0.05140630412226879, -0.024693023019053975, 0.022277553395924386, -0.02558093414443957, -0.009630117687644633, 0.03178223629834082, 0.029116098736227516, -0.003360839176745842, -0.05669647982634004, -0.006232667006894491, -0.012014406773573745, 0.013245862527436156, -0.027188215571635425, -0.03214121727826317, -0.03221323511495042, -0.0008176183152097252, 0.011653274801414374, 0.009388429275466499, 0.007651456484129039, -0.03967889743204727, 0.012839043326245208, -0.00990241329562196, -0.011633113962905122, -0.02745795538525885, -0.0432599555018541, 0.011265744498051265, -0.00683989134582104, -0.002640396457160167, -0.06546607594627912, 0.0318236461696431, -0.017930263407385308, 0.05072200076581535, -0.01938303048131424, -0.02181661307520971, -0.006068150732121843, 0.014737041166493681, -0.004080685852149915, 0.03951379973522272, 0.03294554215361377, -0.004247986488251207, -0.008526790269217781, -0.046188103158081466, -0.006074970801246895, 0.002651835002908636, 0.028548553854991457, -0.019130646366055538, 0.03776250572222558, -0.024356127611660488, 0.00413012211827726, 0.011570293445797372, 0.0008550888693337317, -0.029044367185448023, 0.015547973705295835, -0.015327267279647314, -0.031205850415738416, -0.1229522246564986, -0.02859893401807004, -0.029681334860142974, -0.02760222809105159, 0.05152112324133686, -0.01584851234157606, 0.046287185787252334, -0.011599665456430045, 0.09612676719176202, -9.357236497042312e-05, 0.030496772554063394, 0.023926883450614552, -0.02777539104678277, -0.04974966300273006, -0.0008262035255045977, -0.02782266208333628, 0.0026017874932806615, -0.017979840122916323, 0.017503808225182656, 0.034026739363536895, -0.030657049565602532, 0.030259303011095768, 0.03148136212266334, 0.05001502849081672, -0.029868430396718205, -0.020140930864462215, 0.019619755065253083, -0.01326217312832042, 0.03410858634972371, 0.013877441562544827, -0.0536331036363305, 0.04631066738837438, -0.0825130903788949, 0.010551949075584483, -0.03782276428829324, -0.04227881059917649, 0.027546168384964327, -0.004186598555060889, 0.0545937375389719, 0.025007297222220223, 0.05626814069816378, -0.0024668617915003686, 0.0881777218936227, -0.040573106847403465, -0.05248020258071573, 0.013239519601490694, -0.033493810450394366, -0.0261337229880539, -0.016645324520605426, -0.014433302592567103, -0.026540806157165173, 0.011436366281558238, 0.005488683428082206, -0.034177064091852706, 0.041788335115645156, -0.03713073736901662, 0.018837963661351542, 0.007000082163753545, 0.016766642021945218, 5.4469020392477525e-05, 0.0034077311923058863, -0.008138369555114693, 0.03146106968498681, -0.0006875412923973574, 0.031073823358774136, 0.0329438983184015, -0.020181461868811118, 0.03629839033648013, 0.006615672530702112, 0.08973336257624293, -0.005116123491978752, 0.038878149591372056, 0.015827956705564986, -0.005576927596011515, -0.06704034060461886, 0.036640326530690256, 0.013324392598939042, 0.022701311949576575, 0.02565520224992863, 0.014930356495290606, -0.00900295992280491, 0.003595905303342722, 0.053663843970468514, -0.07946229229876799, 0.04636584976667981, 0.022358121330555588, 0.019843473649619216, 0.015104469119866328, 0.0473298513766662, 0.0011126116050005352, -0.019793439800138722, -0.03558885380168408, -0.022232412572792015, 0.022899794277257003, 0.045651858869426555, 0.004953454992648357, 0.06348554387974807, -0.020291663473237995, -0.030281910362779083, -0.03765204861532189, -0.0021000739410948583, 0.018493364700364986, -0.0042539696328475415, -0.050070854242828824, 0.029992435291579172, -0.01814635693590727, 0.005423693448678865, -0.03531423789259955, 0.04033085663761065, 0.07231794435549367, 0.03500897707801292, 0.03281538365090763, -0.07364874901520459, 0.04157398700765756, 0.024702907578729234, -0.049901514589220555, -0.013898940710714253, 0.022379712829017837, 0.024845223997485656, -0.028774528864845588, 0.05287816461923908, -0.05054786197532333, -0.018547608184026707, 0.020450749165999704, 0.03571699598966356, -0.028839069406157186, 0.05874480316449009, 0.051148308187549486, -0.032918012531322934, 0.07449689410450017, 0.02397662177915801, 0.02024709676109268, -0.03340536965329992, -0.017416498719175305, 0.016386152658824163, 0.028342089383667942, 0.038303158198191294, 0.012944414240771614, 0.018357044893955936, -0.04284210428859955, -0.012433314628095356, 0.015852795855991548, 0.024180808276591866, -0.021468063060861832, 0.005155024128866427, -0.05418832591016074, -0.003815007527446314, -0.061804725453548016, 0.01341730930689236, 0.026422961026835563, -0.03839482817719148, 0.07399554283481725, -0.009526669198380513, 0.04428346148385008, -0.02337717910678054, 0.05290939133158597, -0.014557506040557644, -0.0737075022714992, 0.03138628133951485, -0.0031511641859715894, -0.017505641378495213, -0.05787334270459955, 0.025501360514627775, 0.02440726812548332, -0.09316828862102816, 0.025280489397623715, 0.008727036487491147, 0.005698125811556408, 0.021260221031003692, -0.05599929668238075, -0.04604084445948742, 0.032968244933933, 0.01622443960023161, -0.009986296605765542, 0.040211536980938954, -0.027634556850226175, -0.048710051129665574, 0.09268907063819207, 0.004020824007548514, 0.01062855826099663, 0.006623703927834706, -0.027040569001811513, -0.03628795783173411, 0.030779487583828587, 0.025198169885771952, 0.008984392896432646, 0.03043852106769311, -0.012449125767812597, -0.020223602846598257, -0.018104602290178448, -0.026352445421578254, -0.010207854962865366, -0.0029931940077886155, 0.035878886052984026, 0.027031401696077182, 0.017661935014070547, 0.0031813286775545574, -0.015111600101643887, -0.05442165816000985, 0.07587081403758753, 0.0703260839863762, 0.036936980297985686, -0.020954279902593263, 0.06231559496007799, -0.006267097889609484, 0.02150482001657733, 0.02423647087641998, -0.03079034797826469, 0.010186001805240114, 0.003600838732943957, -0.020637716951336527, -0.06685640344805821, -0.046538046122679436, 0.02040782167155202, 0.045561598771563475, 0.012212561257714647, -0.056386324446233746, -0.016696163356805037, 0.048968191746510306, 0.033048537356857276, -0.004982411811756208, 0.04451003677060757, 0.05240531880660784, 0.06190730815881691, -0.04576780774041018, -0.027007609182302626, 0.004562393829756276, 0.010771000122234135, 0.04270298780748948, 0.00887339630990171, -0.0072992193843401665, -0.031162584303550372, 0.009491255939425604, 0.04370987689346333, -0.023027139174375107, -0.019928769919243007, -0.017980612787032953, -0.00621992266648478, 0.0723879766608852, -0.017350040370116063, -0.023711671867389057, -0.012864720555579147, -0.02201278972391263, -0.008172973979421619, 0.009444585949361223, -0.03848377382255337, -0.034479483595763444, -0.056425671827662956, 0.05477183815702607, 0.07424919830577384, -0.02700884667622647, -0.054059940533076827, 0.00026357572017208646, 0.004257644020123276, 0.010529264765323816, -0.021600191703148218, -0.0006881246865128065, -0.0347452184850214, 0.02395496717466233, 0.03460466441768637, -0.016020988826797252, 0.05768356285284623, -0.01070979034815039, 0.006549249582588452, 0.02109692416488921, -0.025104560550622455, -0.020812120479334645, 0.04365696325401828, 0.01237715795420063, 0.018403958842710528, -0.009712668075228456, -0.0009821914431313932, -0.016104430394706826, 0.034363377729282024, -0.014352046648254364, -0.0073710117323957675, -0.04098228328981875, -0.03554679132164584, 0.029297345420923892, 0.030120115728094535, -0.02112210962891386, -0.027122132780433666, 0.01656979583361932, 0.05096287495624509, -0.01572391024815483, 0.02341720988037665, -0.04311913669701298, -0.04763209849336367, -0.044990347565242075, -0.019893955398019907, -0.04534846045241188, -0.01184620687649608, -0.04014615605196266, -0.056781417468993366, 0.024632870655823064, 0.04700964520635755, -0.09453005525557988, 0.03493000526427603, -0.03094156850442089, -0.028505555558652605, 0.012183815689901631, -0.06840498856830668, 0.018584460568378123, -0.0306679284300972, 0.0062550150081721245, -0.012049768470281826, 0.045483895235181364, 0.05306404728863522, 0.03759066953236797, -0.011039761792339398, 0.005754678822124503, 0.016739998962465495, -0.0009593292623588177, -0.002715317556178068, 0.013420729346069941, 0.009040281754478182, 0.03665308626281544, 0.0016719339432374992, -0.06709930934493007, 0.06422944395850393, -0.0537743010773722, 0.008031573367735703, -0.016569594202146657, 0.024237612941707905, -0.008456033783613356, 0.015582923673615077, -0.007996754228997963, -0.04635632229480347, -0.05532584140699104, -0.012907197842349317, 0.014156202460605607, 0.06379243621617459, 0.013182876549387096, 0.06806806391665379, -0.043426019798410216, -0.014774161366694287, -0.02934839974114009, 0.011952952271214018, 0.029570074305691717, 0.0015929983962301834, -0.04039893419513741, -0.07037105857898149, 0.02035932699361859, -0.029429643372080455, 0.014225066534792915, -0.03664874579905271, -0.011888520241496487, -0.0666851429083592, -0.034898418385787125, -0.03452646218972301, -0.02767808462157722, 0.013504026231854092, 0.014483736626649835, -0.037682659659049954, 0.015464561381645654, -0.024908450086296056, -0.007231265730122949, 0.030218487260010658, 0.0024934631009518775, -0.04920480551009776, 0.027780079363315134, -0.00539175486949146, -0.01660234469432514, 0.02921836745050081, 0.012214753037997671, 0.005792908765010543, 0.02646657191345015, -0.01294388630493097, -0.06867731111178607, -0.000959484526288628, 0.007180052105651628, 0.06386748006412338, -0.0543821753314845, 0.02587534226044783, 0.024958560894353905, 0.013889608713624388, 0.03509531844511717, 0.026500738443451934, -0.03471127667406549, 0.04986354938384061, 0.05076183760379653, -0.003378454995102019, -0.02132760442216272, 0.08033294007608169, -0.06103282476600793, 0.03687813469139824, -0.018835890397277642, -0.035580637703965835, -0.06024968196948806, 0.04238744840531155, -0.02968192590201705, 0.00965801209359212, -0.03918098159325741, -0.005496604389656297, 0.020552488405260808, 0.0015494548483707823, -0.05316137834058553, 0.014240502886238446, 0.048195893952711066, -0.03293794172451435, 0.005778514047909489, -0.07067499570603764, -0.021579811532693508, -0.018136338468307416, 0.03073959841401481, -0.03157704933940138, 0.022739882049374518, -0.001487268373742202, 0.009286108998946378, 0.04651127992947595, -0.014057349167995255, -0.028842578717284496, 0.016392988119664685, 0.04786297114708056, -0.01955011370943451, 0.038180415423998915, 0.04276561977608272, 0.059011843271219964, -0.05565726198119161, 0.05992816750336441, 0.06156848724781805, 0.035339440287511216, -0.00953320759911245, 0.0288846596673813, -0.03067046190646367, 0.0718038672154596, -0.0757822993602588, -0.04718691159343321, -0.07309086089625483, -0.010852188342998805, 0.005518051975578897, -0.01040165590028539, -0.0500583284647788, -0.036371245482488976, -0.025819187125724653, -0.01971865453300985, -0.02197435968872547, -0.07008027829030966, -0.009155284034960357, 0.01419596387918098, 0.04995562570412923, -0.007001879916120519, -0.018600888146299965, 0.01618944653510905, 0.04966424206021095, 0.04953456685737059, 0.01714929238191835, -0.008023033274406803, 0.0356680749612116, -0.007998197202323337, 0.039743656535718555, 0.012070992106744328, -0.0187080406518921, -0.0015367157988632639, -0.002169988307466595, -0.00526953887676107, -0.0695285837976653, 0.00017479017270065163, -0.010755951642018466, -0.01674976192758854, -0.03360652090444296, -0.042592924728501846, -0.01227568730037449, -0.006194382808211416, 0.047931091801410634, -0.05045191925610848, -0.04449630736040773, -0.03244507436839887, 0.044662679491273255, -0.010017460981664774, 0.0006154110961871749, -0.020740091868446567, 0.055715374942122656, -0.0024887276472908837, 0.015426152894860155, 0.006857827311859008, 0.020873442612941325, -0.020758107871404984, 0.013841458810117504, -0.01665936330428622, -0.04797033144083155, 0.0028899548458548962, 0.06514557733337654, 0.008413200948215812, 0.027350087164897324, 0.01633651745477436, -0.003644415757784507, 0.016623682229482125, 0.007924817198816936, 0.03608543671791418, 0.016330370003615376, -0.03435206789675418, -0.062121345354152, -0.02909671440976376, 0.01885136522801185, 0.010028379095033891, -0.02674140330655127, 0.027974098093509068, 0.03129546713989482, 0.020872205119017485, -0.011224459299644296, -0.051034788129267136, 0.1069666475494847, 0.04399908414881405, 0.048071153333850605, -0.06179329248729641, -0.010690456044761914, -0.03384815236727385, -0.029391308765529214, -0.06501487088559965, 0.029941975091580142, 0.0012758394969900671, 0.03671135621924428, -0.003350034962071208, -0.003513480743032871, -0.03771560716518646, 0.05133634569711152, -0.0076662248351231435, 0.01672738852998038, -0.003506040772567238, -0.03900014740653565, -0.022965963262065396, -0.005422743010248552, -0.0006282898255125897, 0.02125473850195304, 0.014523310266296465, 0.03266361517968067, 0.040383037631399304, 0.02545015535559996, 0.02967263854090205, -0.022387113165816135, 0.00040439173526478144, -0.005307108899846305, 0.020259353184121972, -0.032873604353846966, -0.01633448728750378, -0.06549880489005595], expired_time: 2592000, overwrite: False
[2024-08-22, 18:00:41 CST] {subprocess.py:93} INFO - [Milvus Client] get relevant results: data: ['[]'], cost: 0
[2024-08-22, 18:00:41 CST] {subprocess.py:93} INFO - [get_relevant] metric_type: L2, max_distance: 0.5, raw emb response_arr size: 0, post emb_utils.topk: 0
[2024-08-22, 18:00:41 CST] {subprocess.py:93} INFO - relevant_page_metas(0): []
[2024-08-22, 18:00:41 CST] {subprocess.py:93} INFO - RSS article scored -1
[2024-08-22, 18:00:41 CST] {subprocess.py:93} INFO - Scoring page: 一派 | 天命人，来聊聊你的《黑神话：悟空》游玩体验, score_text: 一派 | 天命人，来聊聊你的《黑神话：悟空》游玩体验 - 少数派 - 「上香」之余想来点新鲜空气？本文所述的游戏体验应该能为你提供一些参考。<a href="https://sspai.com/post/91709" target="_blank">查看全文</a>
[2024-08-22, 18:00:41 CST] {subprocess.py:93} INFO - [EmbeddingAgent] provider: ollama, model_name: nomic-embed-text
[2024-08-22, 18:00:41 CST] {subprocess.py:93} INFO - Initialized EmbeddingOllama: model_name: nomic-embed-text, base_url: http://wzstest.nat300.top
[2024-08-22, 18:00:41 CST] {subprocess.py:93} INFO - [get_relevant] collection_name: embedding__news__ollama_nomic_embed_text__2024_08_22
[2024-08-22, 18:00:41 CST] {subprocess.py:93} INFO - [MilvusClient] server.alias: default, host: milvus-standalone, port: 19530
[2024-08-22, 18:00:41 CST] {subprocess.py:93} INFO - [get_relevant] Fallback collection name: embedding__news__ollama_nomic_embed_text__2024_08_21
[2024-08-22, 18:00:41 CST] {subprocess.py:93} INFO - [EmbeddingOllama] Embedding not found, create a new one and cache it
[2024-08-22, 18:00:41 CST] {subprocess.py:93} INFO - [Redis Client] Set key: milvus_embedding_item_id_ollama-norm_nomic-embed-text_default_4424a1aba3c1e2ece1d7ed02f778944d, val: [0.006565957473590565, 0.03645917459180454, -0.14501376889468642, -0.046410348714992125, -0.05662200699228635, 0.047273375000166505, -0.003679611532337908, -0.00866328244959772, -0.07514364824034209, -0.0455441384976571, -0.03525649530912435, 0.0009166179721451297, 0.0197297924513562, 0.05917421739428787, 0.01287831549103489, -0.07723326736007113, 0.028517529625968253, 0.0047031842646909116, -0.07911551327593051, 0.07637470746531375, -0.013213050424026507, -0.036265381228824695, -0.029792793676519127, -0.07588997970958203, 0.03716224012522798, 0.028402688895180394, -0.02425414228085226, -0.02135372521057289, -0.029248521946687273, -0.06061557607891963, 0.05072197633458863, -0.05978929385177781, -0.024215553023065214, -0.018418542562896607, -0.05711388452684206, -0.01585931714385091, 0.03254092993559878, 0.00861374564832342, 0.02657207599267597, 0.020550527232045097, 0.04541849313010455, -0.05771928183056363, -0.019641917404621, -0.0613620497144328, 0.048763502640107315, 0.018823580198894, 0.007664603624044941, 0.033305799294507096, 0.0783725167905165, -0.037880798820635005, -0.009343581387173854, -0.0016869868334116207, -0.05029956479528485, -0.027575204178310917, 0.0017429581394477862, 0.09512079964080966, 0.02614962298820917, -0.03469418030978304, -0.021020985048102386, -0.045262702589034726, 0.0336363210707559, 0.024311812918123003, 0.02095190260560841, 0.027502228453188875, 0.06406027080262558, -0.027044793660117975, -0.032566111136742806, 0.042569889742819014, -0.002483865261937531, -0.03727716082454453, 0.04638669876308257, 0.02226232911440227, -0.01807923757658045, 0.06666465030330397, -0.04072839815861593, 0.03888327244905367, -0.045796747232586844, -0.015007239142613121, 0.014112622326810402, 0.015505078775251977, 0.006205254612031508, -0.06782571633309384, 0.05588618990366994, 0.00508429650162439, 0.0415620996881414, -0.00258343015262301, -0.027818524716723643, 0.017333698388133967, -0.053288409287506826, 0.008204255690446497, 0.01771042420317915, 0.018456831198251745, 0.02528888321059574, 0.00707168659394377, 0.0063203037793084895, -0.011781477517868027, 0.004199000091884961, 0.0656121993062539, -0.025369488525713902, -0.043579767864634536, -0.023981927928306328, -0.06244813488228875, 0.02502556757821415, 0.049170219615207116, -0.010299484454373002, 0.033319260663502576, -0.03319608543494722, 0.03856048392570854, -0.04729899750911698, -0.017695696665812805, -0.0013199312867427323, 0.03628892159423197, -0.0381295986841614, -0.018451828722512827, 0.03841844797157271, -0.06056283239155516, 0.05508946641416836, 0.01240718238517981, -0.014768919599040631, 0.06362012994433237, -0.030578671064092997, 0.015808615615765288, 0.013210757992870841, 0.06902179080766403, -0.00027996290035628153, 0.06856367776299797, -0.08432678980504223, 0.005109849038112266, -0.02583472173123037, -0.09298395762399282, -0.02941528594697124, -0.014696087521090489, -0.0027538380892199016, 0.025197002332932848, 0.005787505682219732, -0.05082019545859923, -0.05915626297869918, -0.008802278118744799, 0.013180969715935293, -0.01685171473765514, 0.04720603853721551, -0.03143759229812816, 0.054045977815925836, -0.05297866451972959, -0.009219430246388789, -0.010414063066094612, 0.006327421718810886, -0.02102889008525287, -0.04495594331299511, 0.010393889523834882, -0.028002162076560363, -0.0022231885910025772, 0.049898413037422124, -0.0279364812583329, 0.007285352136302093, -0.019736653454937723, 0.027449745403991966, 0.02053581006096951, 0.05048300667649589, 0.04613885556010305, 0.04652000630040219, 0.08107833785073182, -0.02463687017824493, 0.009596872075945445, -0.01133279631689089, -0.006523562306197533, -0.029108037974367427, -0.0006904358741487635, 0.0018160686263496515, 0.026162696361750656, -0.057906313410173466, -0.04133076554363951, -0.04574518234057114, 0.0037883335600072767, 0.03207218107661749, -0.021699374366831472, 0.04287528955387097, -0.03234216371485361, -0.027471929266208943, -0.031501774446892096, -0.00234355455476317, -0.0688114558064356, 0.049042281816498896, -0.0318955839091025, -0.009089241481687927, -0.023760191488145434, -0.02321467284162555, -0.01587803866495552, -0.019653994133351272, -0.03261966339478622, -0.019445775336335677, -0.027149428037207612, -0.04556902944265997, 0.023829330204789227, -0.04362130707259089, -0.03870952453059779, 0.057882097754968394, 0.00024053244584126636, -0.018044981428329242, 0.022133516104574546, 0.03449423233189173, 0.013301357893176076, 0.0031873989444559914, 0.027828745879408662, -0.028944961411795096, 0.018149048621224677, 0.06133262129587634, 0.015479140834882795, -0.019344332295900084, -0.031532198029361094, 0.033501017282016494, 0.0030363709735390224, 0.015740889676461613, -0.016074590942175032, 0.036210679793406086, 0.05228921733056164, 0.03548960357024413, -0.03842436860449284, -0.07369667991151287, 0.011911510047758212, 0.03891519173918883, -0.039590472651589916, -0.008268902841395757, -0.0162584400705229, 0.02101648163521842, -0.024276942196919855, -0.019541736089860238, -0.00908047826375275, 0.04405815737401853, -0.03252271784363988, -0.016150476633202036, -0.06507646347640994, 0.012962907385397918, 0.0036393070236554774, -0.020239279352108307, 0.009807812764733725, 0.012841695828491404, 0.07810371590942652, -0.004212181200805372, 0.04607147763198903, -0.03968189741245956, 0.0064549389422941614, 0.022618426010843812, -0.05112210331056523, -0.01698837058688881, 0.0019454084658825628, -0.004630291840119584, -0.04772226500785744, -0.029165372447637953, 0.019453553016199734, 0.03152864091073324, -0.01284992222065531, 0.03116555417252507, 0.030921004488208465, 0.03173098794448513, -0.05762551724978058, -0.06028931040547534, -0.00035204083065395553, 0.008627536517275063, -0.004463237953919132, 0.028211025064501485, -0.0843561886056251, 0.020714748529296584, -0.02965334633327484, 0.007009894615648337, -0.05461486208180884, -0.053818985666351855, 0.04760998326998753, 0.022123816218224665, 0.0065219792255092855, 0.026447331011520477, 0.06593676490801427, -0.025823703845055848, 0.0030253575300605404, -0.027521702270822603, 0.008522280903189431, -0.003628085513912544, -0.007400825951275384, -0.014309125734373495, 0.012220777225711348, -0.030323221965496222, 0.0019856698433909563, -0.010002339172559858, -0.02278812070961425, -0.055305713162924826, 0.04370303490890801, -0.019237529883143853, 0.027361848143776582, 0.018929122366874093, -0.005022576717139576, 0.05712476617033808, -0.01955537664759592, 0.035932629219164756, 0.0518849438382828, 0.0019453444170146798, 0.055408677086304134, 0.05116321898149928, -0.025676099711885586, -0.012943814158724788, 0.007211605603417438, -0.035487841300815684, 0.008085764544413987, 0.08870505490341307, 0.02428249260517014, -0.030620195463062556, -0.010459620692617442, 0.0344658286952214, -0.03099268590788474, 0.029838337234504502, -0.030726005673703614, -0.03618446492498386, 0.016421860201588843, 0.04929265735622015, 0.03745562688619288, -0.030818007985058214, 0.015153486772593082, 0.001633122661083865, 0.03447222913931364, 0.009761960439373702, 0.02042026590330895, -0.037532568457978276, -0.07812226860808168, -0.017536661475736477, -0.047869617388252204, 0.018479737739024255, 0.05295085916612572, 0.01661449254000406, 0.022280958819788758, 0.013887992210630017, -0.025317597835988744, -0.013069920085766623, -0.010221130845696823, 0.0424504582261251, -0.01804792841670119, -0.002815419779902864, 0.04032830376241028, -0.00682175828562369, 0.0060364980625744495, -0.04194291574533901, 0.020709346210914237, 0.0063877957366208216, 0.030809836386145446, 0.06317122216585729, -0.02871590488946207, 0.057317188221140826, 0.024452079198336984, -0.0545784971338382, 0.03139487429482302, 0.010958569518367147, 0.017150457909143324, 0.0003764047608393016, 0.05983974510798662, -0.0022908267870594967, 0.0077886970097815105, 0.01180199536907071, 0.018261041583852108, -0.0025583316966926958, 0.026670712730114152, 0.01813882301584362, 0.02445103516476803, 0.05832689455870954, 0.018077761120597117, 0.03222461886307722, 0.04556095558306004, -0.07534583829883397, 0.02707565262679871, 0.008622012765201009, 0.009341123095366098, 0.0340637558183801, 0.060056833007192965, 0.007363011943946986, 0.003302472916785852, -0.015865057107132152, 0.01545895322408561, 0.003480031557768422, 0.0028894325043251363, -0.020305484408468265, -0.003062694502901846, 0.014427476195031463, 0.037477200618153954, 0.027949818231839316, -0.05483626828156423, -0.057576363260814896, 0.03585282655113081, 0.03549206482384925, 0.035548596650035555, -0.013743352836615853, 0.009813825213371971, 0.00965456270534836, 0.0035631492174961364, -0.0010225225900769954, -0.026139869789506835, -0.03852716962901741, -0.052986519206324974, 0.039925351231753434, -0.05979256367606186, -0.0014600925157066443, 0.041477552220738755, 0.05249174503395242, 0.02939433123065814, -0.03859894582620916, -0.07331438295563589, -0.0026077081906849525, 0.0041903153615797896, 0.023435582940323354, 0.0445488176098598, -0.003227115536362711, -0.028259077264849712, 0.08251953067471149, 0.001588505220110102, 0.03796356032423033, 0.02166731439132177, 0.006414286792646749, -0.06172837073535672, 0.010344983585397994, 0.020317983193322155, 0.03960885652779563, -0.011252116956838759, -0.02900510663075907, 0.039636244267971917, 0.02126431447190719, 0.015333601074471575, 0.0037614126731396616, 0.024432540221161363, 0.003226263279172733, 0.04080978242643944, 0.03205539953278286, -0.0006111914586388598, 0.0053630630903772, -0.03545945839672686, -0.008761174295000179, 0.0685168280523774, 0.022620727327391555, -0.04036717142914904, -0.0146425693237167, 0.020745345376911067, 0.08291266780892145, -0.03577410790093017, 0.005376567035209714, 0.034588070957608755, 0.04664669125882766, -0.032851252253860964, -0.05552194806449186, -0.02821628521661061, 0.047634868291395675, 0.03093424076060468, 0.01547178669204104, -0.09250191325666814, -0.05469901266836541, 0.04698913130764804, -0.0013880401309111895, -0.005948436792830325, 0.0015645733421144623, 0.027470211423740874, 0.033768837808180734, -0.06990293736907918, -0.017728978382732987, -0.02076779728178901, -0.03620258223942729, 0.0037461990307818137, -0.02577057216223738, 0.04425928710905598, -0.035416041409245064, -0.033616796902567074, 0.03307103242686641, -0.04334558447107353, -0.018562645851588837, -0.012511182937634674, 0.04310196775292492, 0.0875312472213859, -0.0376217333674632, 0.011073391737921514, 0.01444935647301917, -0.002792089146771049, 0.021098847738866354, 0.01585944302023866, -0.05539649817556498, 0.010410079448647105, -0.05062030375485774, 0.060057691928427005, 0.0605626428365242, -0.017582875880823606, 0.0056623627095441584, -0.05497008228623219, -0.003958343690196422, -0.009934029018727173, -0.027819608734556942, 0.0006291644317391293, 0.03631639226473431, -0.022799486606978423, -0.013062644430554873, -0.007026813142610811, -0.0026609233591664734, 0.028663833530098028, -0.03043382436426372, -0.020423774152280378, -0.0819659648247679, 0.0022997223603141432, 0.028966709889800338, 0.03733489810225584, 0.014573732710403499, 0.003002673124089936, 0.010780334476912774, 0.002592839412607063, 0.02127921823621638, -0.023332211769807223, 0.04285016462687676, -0.05241248141303776, -0.033425675080804816, 0.005264150905784654, -0.0259772167639568, -0.008140898402246969, -0.04712235887643897, 0.004355625489585282, 0.057992246998740005, -0.04873213128248352, 0.04796605351025284, 0.011381074354287722, -0.027624325587505653, -0.05946764858221315, 0.0027011795535294893, -0.05715673581102834, 0.007378513250873302, 0.026748581344472836, -0.02401450325655645, 0.02050440168087873, 0.018453833859324698, -0.05841206992715224, 0.0513220542502539, 0.02401592047659261, 0.004289524466357549, -0.00529144757069216, -0.03730336680757468, -0.022201714450556936, -0.00657053641230718, -0.04151005794675095, -0.03881406116837459, 0.09769410396360013, 0.003266147212630222, 0.07144265734399477, -0.03449875795825589, 0.004419887236653236, -0.03656080274457505, 0.0037552425087920625, 0.0065359589090425475, 0.06753772892871392, -0.014789281955881983, 0.0703391508840747, 0.019075803900167106, -0.0016637789297578585, 0.033981305303507445, -0.061865774438423476, -0.03369970057243562, -0.003003160709980119, 0.024005685985819477, -0.024012092353506435, 0.04198505619815919, 0.014638856710727517, -0.012647638865546632, -0.06903297455449063, -0.017470487518248784, 0.012157629484674951, 0.06737360981347132, -0.023881527440541013, 0.08574347079408096, 0.02645534859697059, -0.005451391662332951, -0.014048197310663047, 0.02986468834560523, -0.019359204961337008, -0.0329147028386773, 0.001937539710749725, -0.048305087492110675, -0.023934376271711726, -0.021260677384750654, 0.03724372213236433, 0.014741675506036243, -0.03327651304222385, -0.09262560976155865, -0.031720393595332914, 0.0025662726456361527, -0.025845464170250528, 0.025572750754849646, -0.0532425606643935, -0.030128919173370673, 0.009668081829392338, -0.0623758847975196, -0.022734817242548975, 0.02148942588015837, -0.0004021447004272439, -0.03890037979059776, -0.001203990896129272, 0.05582017144054358, -0.022546759400154333, -0.012693308299919754, -0.01412455985106481, 0.0043830095275148705, -0.024743363083177178, -0.011408007458569442, 0.003247965849318915, 0.025598553933439006, 0.031619014233540535, 0.0519757762396807, -0.02365757853775314, 0.04915729433153357, 0.07185486252960033, -0.019431915605595443, 0.055178557278719315, 0.026044713163965172, -0.025357170410498897, 0.055978603904857406, 0.025752262330965635, -0.018716980669256842, 0.0208895849082818, 0.03318021908649643, -0.05204918734901448, 0.06200961708894814, 0.012578731169096823, -0.05290232123101066, -0.015283924328272077, 0.004423145583972522, 0.032293353294381544, 0.02055474335058527, -0.035783532340125684, 0.009071714305368232, -0.039953295789833164, -0.05302389116539758, -0.010437798169678231, -0.026432501292145257, 0.04378917878508702, -0.0442018489728779, 0.0015058384014740884, -0.08295037148929824, -0.04737049529735701, -0.0415607431849511, 0.06062695530437192, 0.029867682722734917, 0.02198242593591304, -0.04181331045471782, 0.03553115166359257, 0.004290982781332059, -0.01653793155917926, 0.024000910087578504, 0.012079828991655517, 0.02034998393288468, -0.06400405588875666, 0.014067749615926783, 0.025255231269005717, 0.017727769969410622, -0.0658295419200331, 0.010835186223547335, 0.04619398645613867, 0.04119760313439047, 0.012290248404108658, -0.01721756187100166, -0.030302853685060152, 0.0553899644505916, -0.05197146978632109, -0.08693158395735084, -0.044211430387333425, 0.03925300546053486, -0.0006377420282804646, 0.020247261395990115, -0.009835122757729345, 0.012590893789950496, -0.038645608943596144, -0.011243416677097454, -0.013615780819881441, -0.07904903277241619, -0.0021183485541088442, 0.033091696887038356, 0.04099033655522571, -0.03170028891486178, -0.01371977100604555, 0.024970016106953586, 0.021380876006960477, -0.005761005370577059, 0.016185652419533136, -0.03823255372214863, 0.006761442022862251, -0.005837072471692288, 0.03516580803579703, 0.05591228333840046, 0.0007249328072696217, -0.03388447230066073, -0.01655617919270647, 0.015230844476907382, -0.026266051242381315, -0.029247885160255144, -0.03689328226887798, -0.0499332674687398, -0.004967536526148065, -0.03359449160665839, 0.003378671817883546, 0.027880390739953045, 0.021870853548510552, -0.012453435293632603, 0.04287557684821477, -0.053286300487787396, 0.01172423411986628, -0.011656751788395365, 0.013945560665891884, -0.026066840752032333, 0.0286195931629505, -0.0013561542535524811, 0.014947393065182382, 0.03660169924250456, 0.008372052617559323, -0.0009119048269856967, 0.01576935847267384, -0.012642902951570004, 0.0006612127451717314, 0.03954867872906071, 0.06510033556312139, 0.003534450141539267, 0.054601658389183565, 0.022613099218294117, 0.02369892078618514, -0.008875761051664526, 0.041638748041988284, 0.02303016619426404, -0.008867314005597393, -0.06762057632443265, -0.04891360949204571, -0.01941791963217671, 0.06841860892836679, -0.0072965988213225755, -0.008575787253373783, 0.02380161074330876, -0.025864837286774058, -0.032726859726591166, 0.013225090130289793, -0.016181113465080863, 0.048176030134000854, 0.006425620110239975, -0.005106267854880877, -0.025126594486120837, -0.05484755272950104, -0.004867719142237754, -0.030790045656194334, -0.022007549382008572, 0.04321202814277578, 0.03973446931557723, 0.01776028754261225, 0.010876190827080438, -0.05541580909434398, 0.01983805354931161, 0.056373559582645644, -0.011729845244962418, -0.016282910440300825, 0.03848838785440204, 0.04332764782626898, -0.01478528501034636, 0.04816978666516862, 0.030292176405581888, 0.039304105350365236, -0.04081615621435545, 0.07385235790428306, 0.026431162559739103, -0.04185930420590172, 0.04644108624798115, -0.005530625295049346, 0.03626363673018039, -0.03628393688927721, -0.010529528736125168, -0.007276358638622273, -0.014402115805147235, -0.0196446022739267], expired_time: 2592000, overwrite: False
[2024-08-22, 18:00:41 CST] {subprocess.py:93} INFO - [Milvus Client] get relevant results: data: ['[]'], cost: 0
[2024-08-22, 18:00:41 CST] {subprocess.py:93} INFO - [get_relevant] metric_type: L2, max_distance: 0.5, raw emb response_arr size: 0, post emb_utils.topk: 0
[2024-08-22, 18:00:41 CST] {subprocess.py:93} INFO - relevant_page_metas(0): []
[2024-08-22, 18:00:41 CST] {subprocess.py:93} INFO - RSS article scored -1
[2024-08-22, 18:00:41 CST] {subprocess.py:93} INFO - Scored_pages (3): [{'id': '618d9bf1878e326d398b040b539da73e', 'source': 'RSS', 'list_name': '少数派', 'title': '把女朋友拍好看，技巧可能没有你以为的那么重要', 'url': 'https://sspai.com/post/90666', 'created_time': '2024-08-22T09:00:00', 'summary': '写在前面本文会出现三位女性模特。但其中只有一位是女朋友。谢谢她们给予的信任、鼓励和支持，以及愿意在镜头前展现美好的自己。我很喜欢拍照，也自认为拍摄不同题材照片都积累了一些经验。但是对于我来说，拍人比拍 ...<a href="https://sspai.com/post/90666" target="_blank">查看全文</a>', 'content': '', 'tags': [], 'published': 'Thu, 22 Aug 2024 17:00:00 +0800', 'published_key': '2024-08-22', '__relevant_score': -1}, {'id': '7a055a88d019d31349b41989d5742a2c', 'source': 'RSS', 'list_name': '少数派', 'title': '键盘鼠标都最低，派商店 Keychron 系列清仓促销专场', 'url': 'https://sspai.com/post/91677', 'created_time': '2024-08-22T08:09:02', 'summary': '活动时间：即日起至8.29省流版：少数派定制KeychronK3史低¥338起售少数派定制KeychronQ1系列史低仅需¥769KeychronM系列鼠标¥179起购买K3系列赠送捣蛋獭贴纸我们曾与 ...<a href="https://sspai.com/post/91677" target="_blank">查看全文</a>', 'content': '', 'tags': [], 'published': 'Thu, 22 Aug 2024 16:09:02 +0800', 'published_key': '2024-08-22', '__relevant_score': -1}, {'id': '0ea87107cd73d55bd838483defe8a14a', 'source': 'RSS', 'list_name': '少数派', 'title': '一派 | 天命人，来聊聊你的《黑神话：悟空》游玩体验', 'url': 'https://sspai.com/post/91709', 'created_time': '2024-08-22T07:20:54', 'summary': '「上香」之余想来点新鲜空气？本文所述的游戏体验应该能为你提供一些参考。<a href="https://sspai.com/post/91709" target="_blank">查看全文</a>', 'content': '', 'tags': [], 'published': 'Thu, 22 Aug 2024 15:20:54 +0800', 'published_key': '2024-08-22', '__relevant_score': -1}]
[2024-08-22, 18:00:41 CST] {subprocess.py:93} INFO - #####################################################
[2024-08-22, 18:00:41 CST] {subprocess.py:93} INFO - # Filter RSS (After Scoring)
[2024-08-22, 18:00:41 CST] {subprocess.py:93} INFO - #####################################################
[2024-08-22, 18:00:41 CST] {subprocess.py:93} INFO - k: 1, input size: 3, min_score: 4
[2024-08-22, 18:00:41 CST] {subprocess.py:93} INFO - After sorting: [{'id': '618d9bf1878e326d398b040b539da73e', 'source': 'RSS', 'list_name': '少数派', 'title': '把女朋友拍好看，技巧可能没有你以为的那么重要', 'url': 'https://sspai.com/post/90666', 'created_time': '2024-08-22T09:00:00', 'summary': '写在前面本文会出现三位女性模特。但其中只有一位是女朋友。谢谢她们给予的信任、鼓励和支持，以及愿意在镜头前展现美好的自己。我很喜欢拍照，也自认为拍摄不同题材照片都积累了一些经验。但是对于我来说，拍人比拍 ...<a href="https://sspai.com/post/90666" target="_blank">查看全文</a>', 'content': '', 'tags': [], 'published': 'Thu, 22 Aug 2024 17:00:00 +0800', 'published_key': '2024-08-22', '__relevant_score': -1}, {'id': '7a055a88d019d31349b41989d5742a2c', 'source': 'RSS', 'list_name': '少数派', 'title': '键盘鼠标都最低，派商店 Keychron 系列清仓促销专场', 'url': 'https://sspai.com/post/91677', 'created_time': '2024-08-22T08:09:02', 'summary': '活动时间：即日起至8.29省流版：少数派定制KeychronK3史低¥338起售少数派定制KeychronQ1系列史低仅需¥769KeychronM系列鼠标¥179起购买K3系列赠送捣蛋獭贴纸我们曾与 ...<a href="https://sspai.com/post/91677" target="_blank">查看全文</a>', 'content': '', 'tags': [], 'published': 'Thu, 22 Aug 2024 16:09:02 +0800', 'published_key': '2024-08-22', '__relevant_score': -1}, {'id': '0ea87107cd73d55bd838483defe8a14a', 'source': 'RSS', 'list_name': '少数派', 'title': '一派 | 天命人，来聊聊你的《黑神话：悟空》游玩体验', 'url': 'https://sspai.com/post/91709', 'created_time': '2024-08-22T07:20:54', 'summary': '「上香」之余想来点新鲜空气？本文所述的游戏体验应该能为你提供一些参考。<a href="https://sspai.com/post/91709" target="_blank">查看全文</a>', 'content': '', 'tags': [], 'published': 'Thu, 22 Aug 2024 15:20:54 +0800', 'published_key': '2024-08-22', '__relevant_score': -1}]
[2024-08-22, 18:00:41 CST] {subprocess.py:93} INFO - Filter output size: 1
[2024-08-22, 18:00:41 CST] {subprocess.py:93} INFO - #####################################################
[2024-08-22, 18:00:41 CST] {subprocess.py:93} INFO - # Summarize RSS Articles
[2024-08-22, 18:00:41 CST] {subprocess.py:93} INFO - #####################################################
[2024-08-22, 18:00:41 CST] {subprocess.py:93} INFO - Number of pages: 1
[2024-08-22, 18:00:41 CST] {subprocess.py:93} INFO - Summary max length: 20000
[2024-08-22, 18:00:41 CST] {subprocess.py:93} INFO - [LLMAgentSummary] translation language: Chinese, translation_enabled: True
[2024-08-22, 18:00:41 CST] {subprocess.py:93} INFO - [LLMAgentSummary] Initialized prompt: input_variables=['text'] template="\nWrite a concise and precise numbered list summary of the following text without losing any numbers and key points (English numbers need to be converted to digital numbers):\n```{text}```\n\nNUMBERED LIST SUMMARY IN BOTH ENGLISH AND Chinese, AFTER FINISHING ALL ENGLISH PART, THEN FOLLOW BY Chinese PART, USE '===' AS THE SEPARATOR:\n"
[2024-08-22, 18:00:41 CST] {subprocess.py:93} INFO - LLM chain initalized, provider: ollama, model_name: llama3.1, temperature: 0
[2024-08-22, 18:00:41 CST] {subprocess.py:93} INFO - [LLMAgentSummary] LLM chain initalized, provider: None, model_name: None, temperature: 0, chain_type: map_reduce
[2024-08-22, 18:00:41 CST] {subprocess.py:93} INFO - [INFO] Initialized default DB driver (Redis)
[2024-08-22, 18:00:41 CST] {subprocess.py:93} INFO - Summarying page, title: 把女朋友拍好看，技巧可能没有你以为的那么重要, list_name: 少数派
[2024-08-22, 18:00:41 CST] {subprocess.py:93} INFO - page content is empty, fallback to load web page via WebBaseLoader
[2024-08-22, 18:00:42 CST] {subprocess.py:93} INFO - [load_web] origin url: https://sspai.com/post/90666, landing page: https://sspai.com/post/90666
[2024-08-22, 18:00:42 CST] {subprocess.py:93} INFO - [load_web] finished, content (post refinement): 把女朋友拍好看，技巧可能没有你以为的那么重要 - 少数派
[2024-08-22, 18:00:42 CST] {subprocess.py:93} INFO - PRIMEMatrix栏目Pi Store更多 无需申请，自由写作 任何用户都可使用写作功能。成功发布 3 篇符合基本规则的内容，可成为正式作者。了解更多PRIMEMatrix栏目Pi Store更多 把女朋友拍好看，技巧可能没有你以为的那么重要主作者关注Hermanchannn少数派作者 一个拍照的 Hermanchannn关注Herma...
[2024-08-22, 18:00:42 CST] {subprocess.py:93} INFO - Page content (5587 chars)
[2024-08-22, 18:00:42 CST] {subprocess.py:93} INFO - [LLM] input text (5587 chars), chunk_size: 10240, chunk_overlap: 256, text: 把女朋友拍好看，技巧可能没有你以为的那么重要 - 少数派
[2024-08-22, 18:00:42 CST] {subprocess.py:93} INFO - PRIMEMatrix栏目Pi Store更多 无需申请，自由写作 任何用户都可使用写作功能。成功发布 3 篇符合基本规则的内容，可成为正式作者。了解更多PRIMEMatrix栏目Pi Store更多 把女朋友拍好看，技巧可能没有你以为的那么重要主作者关注Hermanchannn少数派作者 一个拍照的 Hermanchannn关注Herma
[2024-08-22, 18:01:53 CST] {subprocess.py:93} INFO - Traceback (most recent call last):
[2024-08-22, 18:01:53 CST] {subprocess.py:93} INFO -   File "/opt/***/run/auto-news/src/af_save.py", line 260, in <module>
[2024-08-22, 18:01:53 CST] {subprocess.py:93} INFO -     run(args)
[2024-08-22, 18:01:53 CST] {subprocess.py:93} INFO -   File "/opt/***/run/auto-news/src/af_save.py", line 241, in run
[2024-08-22, 18:01:53 CST] {subprocess.py:93} INFO -     stat = process_rss(args)
[2024-08-22, 18:01:53 CST] {subprocess.py:93} INFO -            ^^^^^^^^^^^^^^^^^
[2024-08-22, 18:01:53 CST] {subprocess.py:93} INFO -   File "/opt/***/run/auto-news/src/af_save.py", line 162, in process_rss
[2024-08-22, 18:01:53 CST] {subprocess.py:93} INFO -     data_summarized = op.summarize(data_filtered)
[2024-08-22, 18:01:53 CST] {subprocess.py:93} INFO -                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^
[2024-08-22, 18:01:53 CST] {subprocess.py:93} INFO -   File "/opt/***/run/auto-news/src/ops_rss.py", line 298, in summarize
[2024-08-22, 18:01:53 CST] {subprocess.py:93} INFO -     summary = llm_agent.run(content)
[2024-08-22, 18:01:53 CST] {subprocess.py:93} INFO -               ^^^^^^^^^^^^^^^^^^^^^^
[2024-08-22, 18:01:53 CST] {subprocess.py:93} INFO -   File "/opt/***/run/auto-news/src/llm_agent.py", line 344, in run
[2024-08-22, 18:01:53 CST] {subprocess.py:93} INFO -     tokens = self.get_num_tokens(text)
[2024-08-22, 18:01:53 CST] {subprocess.py:93} INFO -              ^^^^^^^^^^^^^^^^^^^^^^^^^
[2024-08-22, 18:01:53 CST] {subprocess.py:93} INFO -   File "/opt/***/run/auto-news/src/llm_agent.py", line 245, in get_num_tokens
[2024-08-22, 18:01:53 CST] {subprocess.py:93} INFO -     return self.llm.get_num_tokens(text)
[2024-08-22, 18:01:53 CST] {subprocess.py:93} INFO -            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[2024-08-22, 18:01:53 CST] {subprocess.py:93} INFO -   File "/home/***/.local/lib/python3.11/site-packages/langchain_core/language_models/base.py", line 273, in get_num_tokens
[2024-08-22, 18:01:53 CST] {subprocess.py:93} INFO -     return len(self.get_token_ids(text))
[2024-08-22, 18:01:53 CST] {subprocess.py:93} INFO -                ^^^^^^^^^^^^^^^^^^^^^^^^
[2024-08-22, 18:01:53 CST] {subprocess.py:93} INFO -   File "/home/***/.local/lib/python3.11/site-packages/langchain_core/language_models/base.py", line 260, in get_token_ids
[2024-08-22, 18:01:53 CST] {subprocess.py:93} INFO -     return _get_token_ids_default_method(text)
[2024-08-22, 18:01:53 CST] {subprocess.py:93} INFO -            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[2024-08-22, 18:01:53 CST] {subprocess.py:93} INFO -   File "/home/***/.local/lib/python3.11/site-packages/langchain_core/language_models/base.py", line 51, in _get_token_ids_default_method
[2024-08-22, 18:01:53 CST] {subprocess.py:93} INFO -     tokenizer = get_tokenizer()
[2024-08-22, 18:01:53 CST] {subprocess.py:93} INFO -                 ^^^^^^^^^^^^^^^
[2024-08-22, 18:01:53 CST] {subprocess.py:93} INFO -   File "/home/***/.local/lib/python3.11/site-packages/langchain_core/language_models/base.py", line 45, in get_tokenizer
[2024-08-22, 18:01:53 CST] {subprocess.py:93} INFO -     return GPT2TokenizerFast.from_pretrained("gpt2")
[2024-08-22, 18:01:53 CST] {subprocess.py:93} INFO -            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[2024-08-22, 18:01:53 CST] {subprocess.py:93} INFO -   File "/home/***/.local/lib/python3.11/site-packages/transformers/tokenization_utils_base.py", line 2147, in from_pretrained
[2024-08-22, 18:01:53 CST] {subprocess.py:93} INFO -     raise EnvironmentError(
[2024-08-22, 18:01:53 CST] {subprocess.py:93} INFO - OSError: Can't load tokenizer for 'gpt2'. If you were trying to load it from 'https://huggingface.co/models', make sure you don't have a local directory with the same name. Otherwise, make sure 'gpt2' is the correct path to a directory containing all relevant files for a GPT2TokenizerFast tokenizer.
[2024-08-22, 18:01:54 CST] {subprocess.py:97} INFO - Command exited with return code 1
[2024-08-22, 18:01:54 CST] {taskinstance.py:2731} ERROR - Task failed with exception
Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.11/site-packages/airflow/models/taskinstance.py", line 444, in _execute_task
    result = _execute_callable(context=context, **execute_callable_kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.11/site-packages/airflow/models/taskinstance.py", line 414, in _execute_callable
    return execute_callable(context=context, **execute_callable_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.11/site-packages/airflow/operators/bash.py", line 212, in execute
    raise AirflowException(
airflow.exceptions.AirflowException: Bash command failed. The command returned a non-zero exit code 1.
[2024-08-22, 18:01:54 CST] {taskinstance.py:1149} INFO - Marking task as UP_FOR_RETRY. dag_id=news_pulling, task_id=save, execution_date=20240822T100018, start_date=20240822T100035, end_date=20240822T100154
[2024-08-22, 18:01:54 CST] {standard_task_runner.py:107} ERROR - Failed to execute job 267 for task save (Bash command failed. The command returned a non-zero exit code 1.; 1449)
[2024-08-22, 18:01:54 CST] {local_task_job_runner.py:234} INFO - Task exited with return code 1
[2024-08-22, 18:01:54 CST] {taskinstance.py:3312} INFO - 0 downstream tasks scheduled from follow-on schedule check
finaldie commented 2 months ago

It is related to the langchain library of the tokenizer api.

The error means that part of the GPT2 tokenizer files is missing/incomplete from the huggingFace model hub. It looks like your application environment cannot access huggingface. Can you double-check this path?

source: https://github.com/langchain-ai/langchain/issues/9273

WiiCoder commented 2 months ago

Thank you. This was caused by my network problem. It has been resolved.