QuivrHQ / quivr

Open-source RAG Framework for building GenAI Second Brains šŸ§  Build productivity assistant (RAG) āš”ļøšŸ¤– Chat with your docs (PDF, CSV, ...) & apps using Langchain, GPT 3.5 / 4 turbo, Private, Anthropic, VertexAI, Ollama, LLMs, Groq that you can share with users ! Efficient retrieval augmented generation framework
https://quivr.com
Other
33.99k stars 3.33k forks source link

[Bug]: Error occurred while fetching your chats #2334

Closed plaethosai closed 2 weeks ago

plaethosai commented 3 months ago

What happened?

A bug happened! unable to create brain

Relevant log output

redis         | 1:C 10 Mar 2024 12:33:17.964 * Redis version=7.2.3, bits=64, commit=00000000, modified=0, pid=1, just started
redis         | 1:C 10 Mar 2024 12:33:17.964 # Warning: no config file specified, using the default config. In order to specify a config file use redis-server /path/to/redis.conf
redis         | 1:M 10 Mar 2024 12:33:17.964 * monotonic clock: POSIX clock_gettime
redis         | 1:M 10 Mar 2024 12:33:17.965 * Running mode=standalone, port=6379.
redis         | 1:M 10 Mar 2024 12:33:17.965 * Server initialized
redis         | 1:M 10 Mar 2024 12:33:17.965 * Loading RDB produced by version 7.2.3
redis         | 1:M 10 Mar 2024 12:33:17.965 * RDB age 2066 seconds
redis         | 1:M 10 Mar 2024 12:33:17.965 * RDB memory usage when created 1.43 Mb
redis         | 1:M 10 Mar 2024 12:33:17.965 * Done loading RDB, keys loaded: 9, keys expired: 0.
redis         | 1:M 10 Mar 2024 12:33:17.965 * DB loaded from disk: 0.000 seconds
redis         | 1:M 10 Mar 2024 12:33:17.965 * Ready to accept connections tcp
backend-core  | INFO:     Will watch for changes in these directories: ['/code']
backend-core  | WARNING:  "workers" flag is ignored when reloading is enabled.
backend-core  | INFO:     Uvicorn running on http://0.0.0.0:5050 (Press CTRL+C to quit)
backend-core  | INFO:     Started reloader process [1] using StatReload
web           |   ā–² Next.js 13.5.6
web           |   - Local:        http://localhost:3000
web           |   - Network:      http://0.0.0.0:3000
web           | 
web           |  āœ“ Ready in 83ms
backend-core  | Process SpawnProcess-1:
backend-core  | Traceback (most recent call last):
backend-core  |   File "/usr/local/lib/python3.11/multiprocessing/process.py", line 314, in _bootstrap
backend-core  |     self.run()
backend-core  |   File "/usr/local/lib/python3.11/multiprocessing/process.py", line 108, in run
backend-core  |     self._target(*self._args, **self._kwargs)
backend-core  |   File "/usr/local/lib/python3.11/site-packages/uvicorn/_subprocess.py", line 76, in subprocess_started
backend-core  |     target(sockets=sockets)
backend-core  |   File "/usr/local/lib/python3.11/site-packages/uvicorn/server.py", line 61, in run
backend-core  |     return asyncio.run(self.serve(sockets=sockets))
backend-core  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
backend-core  |   File "/usr/local/lib/python3.11/asyncio/runners.py", line 190, in run
backend-core  |     return runner.run(main)
backend-core  |            ^^^^^^^^^^^^^^^^
backend-core  |   File "/usr/local/lib/python3.11/asyncio/runners.py", line 118, in run
backend-core  |     return self._loop.run_until_complete(task)
backend-core  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
backend-core  |   File "/usr/local/lib/python3.11/asyncio/base_events.py", line 653, in run_until_complete
backend-core  |     return future.result()
backend-core  |            ^^^^^^^^^^^^^^^
backend-core  |   File "/usr/local/lib/python3.11/site-packages/uvicorn/server.py", line 68, in serve
backend-core  |     config.load()
backend-core  |   File "/usr/local/lib/python3.11/site-packages/uvicorn/config.py", line 473, in load
backend-core  |     self.loaded_app = import_from_string(self.app)
backend-core  |                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
backend-core  |   File "/usr/local/lib/python3.11/site-packages/uvicorn/importer.py", line 24, in import_from_string
backend-core  |     raise exc from None
backend-core  |   File "/usr/local/lib/python3.11/site-packages/uvicorn/importer.py", line 21, in import_from_string
backend-core  |     module = importlib.import_module(module_str)
backend-core  |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
backend-core  |   File "/usr/local/lib/python3.11/importlib/__init__.py", line 126, in import_module
backend-core  |     return _bootstrap._gcd_import(name[level:], package, level)
backend-core  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
backend-core  |   File "<frozen importlib._bootstrap>", line 1204, in _gcd_import
backend-core  |   File "<frozen importlib._bootstrap>", line 1176, in _find_and_load
backend-core  |   File "<frozen importlib._bootstrap>", line 1147, in _find_and_load_unlocked
backend-core  |   File "<frozen importlib._bootstrap>", line 690, in _load_unlocked
backend-core  |   File "<frozen importlib._bootstrap_external>", line 940, in exec_module
backend-core  |   File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
backend-core  |   File "/code/main.py", line 14, in <module>
backend-core  |     from modules.api_key.controller import api_key_router
backend-core  |   File "/code/modules/api_key/controller/__init__.py", line 1, in <module>
backend-core  |     from .api_key_routes import api_key_router
backend-core  |   File "/code/modules/api_key/controller/api_key_routes.py", line 7, in <module>
backend-core  |     from middlewares.auth import AuthBearer, get_current_user
backend-core  |   File "/code/middlewares/auth/__init__.py", line 1, in <module>
backend-core  |     from .auth_bearer import AuthBearer, get_current_user
backend-core  |   File "/code/middlewares/auth/auth_bearer.py", line 7, in <module>
backend-core  |     from modules.api_key.service.api_key_service import ApiKeyService
backend-core  |   File "/code/modules/api_key/service/api_key_service.py", line 6, in <module>
backend-core  |     from modules.api_key.repository.api_keys import ApiKeys
backend-core  |   File "/code/modules/api_key/repository/api_keys.py", line 5, in <module>
backend-core  |     from models.settings import get_supabase_client
backend-core  |   File "/code/models/__init__.py", line 2, in <module>
backend-core  |     from .files import File
backend-core  |   File "/code/models/files.py", line 10, in <module>
backend-core  |     from models.settings import get_supabase_db
backend-core  |   File "/code/models/settings.py", line 8, in <module>
backend-core  |     from pydantic import BaseSettings
backend-core  |   File "/usr/local/lib/python3.11/site-packages/pydantic/__init__.py", line 374, in __getattr__
backend-core  |     return _getattr_migration(attr_name)
backend-core  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
backend-core  |   File "/usr/local/lib/python3.11/site-packages/pydantic/_migration.py", line 296, in wrapper
backend-core  |     raise PydanticImportError(
backend-core  | pydantic.errors.PydanticImportError: `BaseSettings` has been moved to the `pydantic-settings` package. See https://docs.pydantic.dev/2.6/migration/#basesettings-has-moved-to-pydantic-settings for more details.
backend-core  | 
backend-core  | For further information visit https://errors.pydantic.dev/2.6/u/import-error
worker        | /usr/local/lib/python3.11/site-packages/celery/platforms.py:829: SecurityWarning: You're running the worker with superuser privileges: this is
worker        | absolutely not recommended!
worker        | 
worker        | Please specify a different user using the --uid option.
worker        | 
worker        | User information: uid=0 euid=0 gid=0 egid=0
worker        | 
worker        |   warnings.warn(SecurityWarning(ROOT_DISCOURAGED.format(
beat          | [2024-03-10 12:33:19,794: INFO/MainProcess] beat: Starting...
beat          | [2024-03-10 12:33:19,807: INFO/MainProcess] Scheduler: Sending due task process_integration_brain_sync (celery_worker.process_integration_brain_sync)
worker        |  
worker        |  -------------- celery@3e2c9847efcb v5.3.6 (emerald-rush)
worker        | --- ***** ----- 
worker        | -- ******* ---- Linux-6.6.16-linuxkit-aarch64-with-glibc2.31 2024-03-10 12:33:19
worker        | - *** --- * --- 
worker        | - ** ---------- [config]
worker        | - ** ---------- .> app:         celery_config:0xffffaca1f450
worker        | - ** ---------- .> transport:   redis://redis:6379/0
worker        | - ** ---------- .> results:     redis://redis:6379/0
worker        | - *** --- * --- .> concurrency: 8 (prefork)
worker        | -- ******* ---- .> task events: OFF (enable -E to monitor tasks in this worker)
worker        | --- ***** ----- 
worker        |  -------------- [queues]
worker        |                 .> celery           exchange=celery(direct) key=celery
worker        |                 
worker        | 
worker        | [tasks]
worker        |   . NotionConnectorLoad
worker        |   . celery_worker.process_integration_brain_sync
worker        |   . celery_worker.process_integration_brain_sync_user_brain
worker        |   . celery_worker.remove_onboarding_more_than_x_days_task
worker        |   . process_crawl_and_notify
worker        |   . process_file_and_notify
worker        | 
flower        | 2024-03-10 12:33:19,841:INFO - Visit me at http://0.0.0.0:5555
flower        | 2024-03-10 12:33:19,844:INFO - Broker: redis://redis:6379/0
flower        | 2024-03-10 12:33:19,846:INFO - Registered tasks: 
flower        | ['NotionConnectorLoad',
flower        |  'celery.accumulate',
flower        |  'celery.backend_cleanup',
flower        |  'celery.chain',
flower        |  'celery.chord',
flower        |  'celery.chord_unlock',
flower        |  'celery.chunks',
flower        |  'celery.group',
flower        |  'celery.map',
flower        |  'celery.starmap',
flower        |  'celery_worker.process_integration_brain_sync',
flower        |  'celery_worker.process_integration_brain_sync_user_brain',
flower        |  'celery_worker.remove_onboarding_more_than_x_days_task',
flower        |  'process_crawl_and_notify',
flower        |  'process_file_and_notify']
flower        | 2024-03-10 12:33:19,854:INFO - Connected to redis://redis:6379/0
worker        | [2024-03-10 12:33:20,438: WARNING/MainProcess] /usr/local/lib/python3.11/site-packages/celery/worker/consumer/consumer.py:507: CPendingDeprecationWarning: The broker_connection_retry configuration setting will no longer determine
worker        | whether broker connection retries are made during startup in Celery 6.0 and above.
worker        | If you wish to retain the existing behavior for retrying connections on startup,
worker        | you should set broker_connection_retry_on_startup to True.
worker        |   warnings.warn(
worker        | 
worker        | [2024-03-10 12:33:20,444: INFO/MainProcess] Connected to redis://redis:6379/0
worker        | [2024-03-10 12:33:20,445: WARNING/MainProcess] /usr/local/lib/python3.11/site-packages/celery/worker/consumer/consumer.py:507: CPendingDeprecationWarning: The broker_connection_retry configuration setting will no longer determine
worker        | whether broker connection retries are made during startup in Celery 6.0 and above.
worker        | If you wish to retain the existing behavior for retrying connections on startup,
worker        | you should set broker_connection_retry_on_startup to True.
worker        |   warnings.warn(
worker        | 
worker        | [2024-03-10 12:33:20,446: INFO/MainProcess] mingle: searching for neighbors
flower        | 2024-03-10 12:33:20,887:WARNING - Inspect method scheduled failed
flower        | 2024-03-10 12:33:20,887:WARNING - Inspect method registered failed
flower        | 2024-03-10 12:33:20,889:WARNING - Inspect method stats failed
flower        | 2024-03-10 12:33:20,889:WARNING - Inspect method active_queues failed
flower        | 2024-03-10 12:33:20,890:WARNING - Inspect method active failed
flower        | 2024-03-10 12:33:20,890:WARNING - Inspect method revoked failed
flower        | 2024-03-10 12:33:20,906:WARNING - Inspect method reserved failed
flower        | 2024-03-10 12:33:20,906:WARNING - Inspect method conf failed
worker        | [2024-03-10 12:33:21,454: INFO/MainProcess] mingle: all alone
worker        | [2024-03-10 12:33:21,461: INFO/MainProcess] celery@3e2c9847efcb ready.
worker        | [2024-03-10 12:33:21,464: INFO/MainProcess] Task celery_worker.process_integration_brain_sync[a9650969-587b-4138-8a12-ba04182e9db1] received
worker        | [2024-03-10 12:33:21,498: INFO/ForkPoolWorker-2] HTTP Request: GET http://host.docker.internal:54321/rest/v1/integrations_user?select=%2A%2C%20integrations%20%28%29&integrations.integration_name=eq.notion "HTTP/1.1 200 OK"
worker        | [2024-03-10 12:33:21,504: ERROR/ForkPoolWorker-2] Task celery_worker.process_integration_brain_sync[a9650969-587b-4138-8a12-ba04182e9db1] raised unexpected: TypeError("'NoneType' object is not iterable")
worker        | Traceback (most recent call last):
worker        |   File "/usr/local/lib/python3.11/site-packages/celery/app/trace.py", line 477, in trace_task
worker        |     R = retval = fun(*args, **kwargs)
worker        |                  ^^^^^^^^^^^^^^^^^^^^
worker        |   File "/usr/local/lib/python3.11/site-packages/celery/app/trace.py", line 760, in __protected_call__
worker        |     return self.run(*args, **kwargs)
worker        |            ^^^^^^^^^^^^^^^^^^^^^^^^^
worker        |   File "/code/celery_worker.py", line 192, in process_integration_brain_sync
worker        |     for integration in integrations:
worker        | TypeError: 'NoneType' object is not iterable
worker        | [2024-03-10 12:33:24,853: INFO/MainProcess] Events of group {task} enabled by remote.
beat          | [2024-03-10 12:35:00,001: INFO/MainProcess] Scheduler: Sending due task process_integration_brain_sync (celery_worker.process_integration_brain_sync)
worker        | [2024-03-10 12:35:00,016: INFO/MainProcess] Task celery_worker.process_integration_brain_sync[54d01da2-4680-4366-82fa-b8e0d1c55a6e] received
worker        | [2024-03-10 12:35:00,080: INFO/ForkPoolWorker-2] HTTP Request: GET http://host.docker.internal:54321/rest/v1/integrations_user?select=%2A%2C%20integrations%20%28%29&integrations.integration_name=eq.notion "HTTP/1.1 200 OK"
worker        | [2024-03-10 12:35:00,095: ERROR/ForkPoolWorker-2] Task celery_worker.process_integration_brain_sync[54d01da2-4680-4366-82fa-b8e0d1c55a6e] raised unexpected: TypeError("'NoneType' object is not iterable")
worker        | Traceback (most recent call last):
worker        |   File "/usr/local/lib/python3.11/site-packages/celery/app/trace.py", line 477, in trace_task
worker        |     R = retval = fun(*args, **kwargs)
worker        |                  ^^^^^^^^^^^^^^^^^^^^
worker        |   File "/usr/local/lib/python3.11/site-packages/celery/app/trace.py", line 760, in __protected_call__
worker        |     return self.run(*args, **kwargs)
worker        |            ^^^^^^^^^^^^^^^^^^^^^^^^^
worker        |   File "/code/celery_worker.py", line 192, in process_integration_brain_sync
worker        |     for integration in integrations:
worker        | TypeError: 'NoneType' object is not iterable
web           | react-i18next:: You will need to pass in an i18next instance by using initReactI18next
web           | i18next: languageChanged en
web           | i18next: initialized {
web           |   debug: true,
web           |   initImmediate: true,
web           |   ns: [ 'translation' ],
web           |   defaultNS: 'translation',
web           |   fallbackLng: [ 'en' ],
web           |   fallbackNS: false,
web           |   supportedLngs: false,
web           |   nonExplicitSupportedLngs: false,
web           |   load: 'all',
web           |   preload: false,
web           |   simplifyPluralSuffix: true,
web           |   keySeparator: '.',
web           |   nsSeparator: ':',
web           |   pluralSeparator: '_',
web           |   contextSeparator: '_',
web           |   partialBundledLanguages: false,
web           |   saveMissing: false,
web           |   updateMissing: false,
web           |   saveMissingTo: 'fallback',
web           |   saveMissingPlurals: true,
web           |   missingKeyHandler: false,
web           |   missingInterpolationHandler: false,
web           |   postProcess: false,
web           |   postProcessPassResolved: false,
web           |   returnNull: false,
web           |   returnEmptyString: true,
web           |   returnObjects: false,
web           |   joinArrays: false,
web           |   returnedObjectHandler: false,
web           |   parseMissingKeyHandler: false,
web           |   appendNamespaceToMissingKey: false,
web           |   appendNamespaceToCIMode: false,
web           |   overloadTranslationOptionHandler: [Function: overloadTranslationOptionHandler],
web           |   interpolation: {
web           |     escapeValue: false,
web           |     format: [Function: bound format],
web           |     prefix: '{{',
web           |     suffix: '}}',
web           |     formatSeparator: ',',
web           |     unescapePrefix: '-',
web           |     nestingPrefix: '$t(',
web           |     nestingSuffix: ')',
web           |     nestingOptionsSeparator: ',',
web           |     maxReplaces: 1000,
web           |     skipOnVariables: true
web           |   },
web           |   lng: 'en',
web           |   resources: {
web           |     en: {
web           |       brain: [Object],
web           |       chat: [Object],
web           |       config: [Object],
web           |       contact: [Object],
web           |       explore: [Object],
web           |       home: [Object],
web           |       invitation: [Object],
web           |       login: [Object],
web           |       logout: [Object],
web           |       monetization: [Object],
web           |       translation: [Object],
web           |       upload: [Object],
web           |       user: [Object],
web           |       delete_or_unsubscribe_from_brain: [Object],
web           |       knowledge: {},
web           |       external_api_definition: [Object]
web           |     },
web           |     es: {
web           |       brain: [Object],
web           |       chat: [Object],
web           |       config: [Object],
web           |       contact: [Object],
web           |       explore: [Object],
web           |       home: [Object],
web           |       invitation: [Object],
web           |       login: [Object],
web           |       logout: [Object],
web           |       monetization: [Object],
web           |       translation: [Object],
web           |       upload: [Object],
web           |       user: [Object],
web           |       delete_or_unsubscribe_from_brain: [Object],
web           |       knowledge: {},
web           |       external_api_definition: [Object]
web           |     },
web           |     fr: {
web           |       brain: [Object],
web           |       chat: [Object],
web           |       config: [Object],
web           |       contact: [Object],
web           |       explore: [Object],
web           |       home: [Object],
web           |       invitation: [Object],
web           |       login: [Object],
web           |       logout: [Object],
web           |       monetization: [Object],
web           |       translation: [Object],
web           |       upload: [Object],
web           |       user: [Object],
web           |       delete_or_unsubscribe_from_brain: [Object],
web           |       knowledge: {},
web           |       external_api_definition: [Object]
web           |     },
web           |     ptbr: {
web           |       brain: [Object],
web           |       chat: [Object],
web           |       config: [Object],
web           |       contact: [Object],
web           |       explore: [Object],
web           |       home: [Object],
web           |       invitation: [Object],
web           |       login: [Object],
web           |       logout: [Object],
web           |       monetization: [Object],
web           |       translation: [Object],
web           |       upload: [Object],
web           |       user: [Object],
web           |       delete_or_unsubscribe_from_brain: [Object],
web           |       knowledge: {},
web           |       external_api_definition: [Object]
web           |     },
web           |     ru: {
web           |       brain: [Object],
web           |       chat: [Object],
web           |       config: [Object],
web           |       contact: [Object],
web           |       explore: [Object],
web           |       home: [Object],
web           |       invitation: [Object],
web           |       login: [Object],
web           |       logout: [Object],
web           |       monetization: [Object],
web           |       translation: [Object],
web           |       upload: [Object],
web           |       user: [Object],
web           |       delete_or_unsubscribe_from_brain: [Object],
web           |       knowledge: {},
web           |       external_api_definition: [Object]
web           |     },
web           |     zh_cn: {
web           |       brain: [Object],
web           |       chat: [Object],
web           |       config: [Object],
web           |       contact: [Object],
web           |       explore: [Object],
web           |       home: [Object],
web           |       invitation: [Object],
web           |       login: [Object],
web           |       logout: [Object],
web           |       monetization: [Object],
web           |       translation: [Object],
web           |       upload: [Object],
web           |       user: [Object],
web           |       delete_or_unsubscribe_from_brain: [Object],
web           |       knowledge: {},
web           |       external_api_definition: [Object]
web           |     }
web           |   },
web           |   ignoreJSONStructure: true
web           | }

Twitter / LinkedIn details

No response

dosubot[bot] commented 3 months ago

Hey @plaethosai! Great to see you back. Hope you're doing well!

The TypeError: 'NoneType' object is not iterable error in the celery_worker.process_integration_brain_sync task likely occurs because the integration.get_integration_brain_by_type_integration("notion") call returns None instead of an iterable object (like a list). This can happen if the method get_integration_brain_by_type_integration does not find any integrations of type "notion" and returns None instead of an empty list or if there's an issue within the method itself that leads to an unexpected None return.

To resolve this issue, ensure that get_integration_brain_by_type_integration always returns an iterable object, even when no integrations are found. This can be done by returning an empty list if no results are found or by adding a check before iterating over integrations to ensure it is not None. Here's a quick example of how you might modify the process_integration_brain_sync task to include such a check:

@celery.task
def process_integration_brain_sync():
    integration = IntegrationBrain()
    integrations = integration.get_integration_brain_by_type_integration("notion")

    if integrations is None:
        logger.info("No integrations found.")
        return

    time = datetime.now(timezone.utc)
    for integration in integrations:
        print(f"last_synced: {integration.last_synced}")
        last_synced = datetime.strptime(
            integration.last_synced, "%Y-%m-%dT%H:%M:%S.%f%z"
        )
        if last_synced < time - timedelta(hours=12):
            process_integration_brain_sync_user_brain.delay(
                brain_id=integration.brain_id, user_id=integration.user_id
            )

Regarding the issue of the backend core failing to import BaseSettings from pydantic, this could be due to several reasons such as an incorrect Pydantic version where BaseSettings is not available, or a misconfiguration in the environment. To resolve this, ensure that you have the correct version of Pydantic installed where BaseSettings is available. BaseSettings is a class provided by Pydantic for settings management, so you should ensure that your Pydantic installation is not corrupted and is up to date. You can update Pydantic using pip:

pip install -U pydantic

If the issue persists, check your import statements to ensure they are correct and that there are no circular dependencies or other import-related issues in your project.

The celery_worker.process_integration_brain_sync task in the quivr codebase is designed to synchronize user brains with their Notion integrations. It retrieves all integrations of type "notion" from the database and checks if the last synchronization time for each integration is more than 12 hours ago. If so, it triggers the process_integration_brain_sync_user_brain task for each of those integrations, passing the brain_id and user_id as arguments. This task is scheduled to run every 5 minutes. You can find more details in the backend/celery_worker.py file.

Sources

#### About Dosu This response is meant to be useful and save you time. It is **not meant to be a precise solution**, but rather a starting point for your own research. Help me be more useful! Please leave a šŸ‘ if this is helpful and šŸ‘Ž if it is irrelevant. If you want to continue the conversation, start your reply with @dosu-bot.
github-actions[bot] commented 3 weeks ago

Thanks for your contributions, we'll be closing this issue as it has gone stale. Feel free to reopen if you'd like to continue the discussion.