standardnotes / server

Server ecosystem for Standard Notes; fully self-hostable.
https://standardnotes.com/help/self-hosting/docker
GNU General Public License v3.0
262 stars 62 forks source link

Issue with notes duplicating #1058

Open AdamXweb opened 5 months ago

AdamXweb commented 5 months ago

I think the issue is narrowed down to localstack, as this is the error i'm getting:

LocalStack version: 1.3.1
LocalStack build date: 2022-12-16
LocalStack build git hash: 0c095708

Waiting for all LocalStack services to be ready
Waiting for all LocalStack services to be ready
Waiting for all LocalStack services to be ready
Error starting infrastructure: gave up waiting for edge server on 0.0.0.0:4566 Traceback (most recent call last):
  File "/opt/code/localstack/localstack/services/infra.py", line 421, in start_infra
    thread = do_start_infra(asynchronous, apis, is_in_docker)
  File "/opt/code/localstack/localstack/services/infra.py", line 514, in do_start_infra
    thread = start_runtime_components()
  File "/opt/code/localstack/localstack/utils/bootstrap.py", line 77, in wrapped
    return f(*args, **kwargs)
  File "/opt/code/localstack/localstack/services/infra.py", line 507, in start_runtime_components
    raise TimeoutError(
TimeoutError: gave up waiting for edge server on 0.0.0.0:4566

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/local/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/opt/code/localstack/localstack/cli/main.py", line 15, in <module>
    main()
  File "/opt/code/localstack/localstack/cli/main.py", line 11, in main
    cli()
  File "/opt/code/localstack/localstack/cli/plugin.py", line 15, in __call__
    self.group(*args, **kwargs)
  File "/opt/code/localstack/.venv/lib/python3.10/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/opt/code/localstack/.venv/lib/python3.10/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/opt/code/localstack/.venv/lib/python3.10/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/opt/code/localstack/.venv/lib/python3.10/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/opt/code/localstack/.venv/lib/python3.10/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/opt/code/localstack/localstack/utils/analytics/cli.py", line 66, in publisher_wrapper
    return fn(*args, **kwargs)
  File "/opt/code/localstack/localstack/cli/localstack.py", line 147, in cmd_start
    bootstrap.start_infra_locally()
  File "/opt/code/localstack/localstack/utils/bootstrap.py", line 260, in start_infra_locally
    return infra.start_infra()
  File "/opt/code/localstack/localstack/services/infra.py", line 435, in start_infra
    raise e
  File "/opt/code/localstack/localstack/services/infra.py", line 421, in start_infra
    thread = do_start_infra(asynchronous, apis, is_in_docker)
  File "/opt/code/localstack/localstack/services/infra.py", line 514, in do_start_infra
    thread = start_runtime_components()
  File "/opt/code/localstack/localstack/utils/bootstrap.py", line 77, in wrapped
    return f(*args, **kwargs)
  File "/opt/code/localstack/localstack/services/infra.py", line 507, in start_runtime_components
    raise TimeoutError(
TimeoutError: gave up waiting for edge server on 0.0.0.0:4566
2024-05-30 00:14:27,727 INFO exited: infra (exit status 1; not expected)
2024-05-30 00:14:28,729 INFO spawned: 'infra' with pid 379877
2024-05-30 00:14:29,731 INFO success: infra entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
Waiting for all LocalStack services to be ready
SERVICES variable is ignored if EAGER_SERVICE_LOADING=0.

The latest note has been duplicated thousands of times, and i've now run into thousands of items.

Screenshot 2024-06-05 at 9 25 24 PM

Not sure what else I can provide, but it's causing the app to crash and display a white box. I also can't delete them as selecting too many notes is crashing the app.

The console log shows these errors.

Screenshot 2024-06-05 at 9 28 40 PM

In network, there are a constant stream of requests to {{URL}}/v1/items with about 149 items of the same entry

auth_hash
: 
null
content
: 
"<<redacted>>"
content_type
: 
"Note"
created_at
: 
"2024-05-28T10:58:47.736000Z"
created_at_timestamp
: 
1716893927736000
deleted
: 
false
duplicate_of
: 
"e2f22b98-fa38-4c13-b7fb-5f3a9f5f849b"
enc_item_key
: 
"<<redacted>>"
items_key_id
: 
"dbc3cefc-1e97-458c-b311-8113a3aa0022"
key_system_identifier
: 
null
last_edited_by_uuid
: 
null
shared_vault_uuid
: 
null
updated_at
: 
"2024-06-02T10:43:21.479634Z"
updated_at_timestamp
: 
1717325001479634
updated_with_session
: 
"07f1f2b9-7752-49f2-a806-96705a8859b9"
user_uuid
: 
"<<redacted>>"
uuid
: 
"68ed15b9-aa7c-4052-8da1-4ff61358dc33"

While writing this, it has continued to increase. It looks like its adding entries to the trash?

Screenshot 2024-06-05 at 9 35 36 PM Screenshot 2024-06-05 at 9 36 11 PM

My client is running standardnotes on MacOS Version 3.194.0

Please let me know what other information I can provide

AdamXweb commented 5 months ago

Docker-compose.yml contains:

services:
  server:
    image: standardnotes/server
    env_file: .env
    container_name: server_self_hosted
    restart: unless-stopped
    ports:
      - 3088:3000
      - 3125:3104
    volumes:
      - ./logs:/var/lib/server/logs
      - ./uploads:/opt/server/packages/files/dist/uploads
    networks:
      - standardnotes_self_hosted

  localstack:
    image: localstack/localstack:1.3
    container_name: localstack_self_hosted
    expose:
      - 4566
    restart: unless-stopped
    environment:
      - SERVICES=sns,sqs
      - HOSTNAME_EXTERNAL=localstack
      - LS_LOG=warn
    volumes:
      - ./localstack_bootstrap.sh:/etc/localstack/init/ready.d/localstack_bootstrap.sh
    networks:
      - standardnotes_self_hosted

  db:
    image: mysql:8
    container_name: db_self_hosted
    environment:
      - MYSQL_DATABASE=standard_notes_db
      - MYSQL_USER=std_notes_user
      - MYSQL_ROOT_PASSWORD=<<redacted>>
      - MYSQL_PASSWORD=<<redacted>>
    expose:
      - 3306
    restart: unless-stopped
    command: --mysql-native-password=ON --character-set-server=utf8mb4 --collation-server=utf8mb4_general_ci
    volumes:
      - ./data/mysql:/var/lib/mysql
      - ./data/import:/docker-entrypoint-initdb.d
    networks:
      - standardnotes_self_hosted
 cache:
    image: redis:6.0-alpine
    container_name: cache_self_hosted
    volumes:
      - ./data/redis/:/data
    expose:
      - 6379
    restart: unless-stopped
    networks:
      - standardnotes_self_hosted

networks:
  standardnotes_self_hosted:
    name: standardnotes_self_hosted

and .env


######
# DB #
######

DB_HOST=db
DB_PORT=3306
DB_USERNAME=<<redacted>>
DB_PASSWORD=<<redacted>>
DB_DATABASE=standard_notes_db
DB_TYPE=mysql

#########
# CACHE #
#########

REDIS_PORT=6379
REDIS_HOST=cache
CACHE_TYPE=redis

########
# KEYS #
########

AUTH_JWT_SECRET=<<redacted>>
AUTH_SERVER_ENCRYPTION_SERVER_KEY=<<redacted>>
VALET_TOKEN_SECRET=<<redacted>>

PUBLIC_FILES_SERVER_URL=http://<<hostname>>:3125
AdamXweb commented 5 months ago

Revisions-worker.log

{"level":"error","message":"Error occured while handling SQS message: SQSError: SQS receive message failed: getaddrinfo EAI_AGAIN localstack\n    at toSQSError (/opt/server/.yarn/__virtual__/sqs-consumer-virtual-a292e446e4/0/cache/sqs-consumer-npm-8.1.0-acaed020b6-7fe1c17b9f.zip/node_modules/sqs-consumer/dist/errors.js:47:22)\n    at Consumer.receiveMessage (/opt/server/.yarn/__virtual__/sqs-consumer-virtual-a292e446e4/0/cache/sqs-consumer-npm-8.1.0-acaed020b6-7fe1c17b9f.zip/node_modules/sqs-consumer/dist/consumer.js:189:43)\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5) {\n  code: 'Error',\n  statusCode: undefined,\n  retryable: undefined,\n  service: undefined,\n  fault: undefined,\n  time: 2024-06-05T11:37:41.275Z\n}","service":"revisions:worker"}

syncing-server-worker.log

{"level":"error","message":"Error occured while handling SQS message: SQSError: SQS receive message failed: connect ECONNREFUSED 172.24.0.5:4566\n    at toSQSError (/opt/server/.yarn/__virtual__/sqs-consumer-virtual-a292e446e4/0/cache/sqs-consumer-npm-8.1.0-acaed020b6-7fe1c17b9f.zip/node_modules/sqs-consumer/dist/errors.js:47:22)\n    at Consumer.receiveMessage (/opt/server/.yarn/__virtual__/sqs-consumer-virtual-a292e446e4/0/cache/sqs-consumer-npm-8.1.0-acaed020b6-7fe1c17b9f.zip/node_modules/sqs-consumer/dist/consumer.js:189:43)\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5) {\n  code: 'Error',\n  statusCode: undefined,\n  retryable: undefined,\n  service: undefined,\n  fault: undefined,\n  time: 2024-06-05T08:58:02.305Z\n}","service":"syncing-server:worker"}

files-worker.log

{"level":"error","message":"Error occured while handling SQS message: SQSError: SQS receive message failed: connect ECONNREFUSED 172.24.0.5:4566\n    at toSQSError (/opt/server/.yarn/__virtual__/sqs-consumer-virtual-a292e446e4/0/cache/sqs-consumer-npm-8.1.0-acaed020b6-7fe1c17b9f.zip/node_modules/sqs-consumer/dist/errors.js:47:22)\n    at Consumer.receiveMessage (/opt/server/.yarn/__virtual__/sqs-consumer-virtual-a292e446e4/0/cache/sqs-consumer-npm-8.1.0-acaed020b6-7fe1c17b9f.zip/node_modules/sqs-consumer/dist/consumer.js:189:43)\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5) {\n  code: 'Error',\n  statusCode: undefined,\n  retryable: undefined,\n  service: undefined,\n  fault: undefined,\n  time: 2024-06-05T11:05:01.489Z\n}","service":"files:worker"}

auth.log

query: SELECT `session`.`uuid` AS `session_uuid`, `session`.`user_uuid` AS `session_user_uuid`, `session`.`hashed_access_token` AS `session_hashed_access_token`, `session`.`hashed_refresh_token` AS `session_hashed_refresh_token`, `session`.`access_expiration` AS `session_access_expiration`, `session`.`refresh_expiration` AS `session_refresh_expiration`, `session`.`api_version` AS `session_api_version`, `session`.`user_agent` AS `session_user_agent`, `session`.`created_at` AS `session_created_at`, `session`.`updated_at` AS `session_updated_at`, `session`.`readonly_access` AS `session_readonly_access` FROM `sessions` `session` WHERE `session`.`uuid` = ? -- PARAMETERS: ["<<redacted>>"]
query: SELECT `user`.`uuid` AS `user_uuid`, `user`.`version` AS `user_version`, `user`.`email` AS `user_email`, `user`.`pw_nonce` AS `user_pw_nonce`, `user`.`encrypted_server_key` AS `user_encrypted_server_key`, `user`.`server_encryption_version` AS `user_server_encryption_version`, `user`.`kp_created` AS `user_kp_created`, `user`.`kp_origination` AS `user_kp_origination`, `user`.`pw_cost` AS `user_pw_cost`, `user`.`pw_key_size` AS `user_pw_key_size`, `user`.`pw_salt` AS `user_pw_salt`, `user`.`pw_alg` AS `user_pw_alg`, `user`.`pw_func` AS `user_pw_func`, `user`.`encrypted_password` AS `user_encrypted_password`, `user`.`created_at` AS `user_created_at`, `user`.`updated_at` AS `user_updated_at`, `user`.`locked_until` AS `user_locked_until`, `user`.`num_failed_attempts` AS `user_num_failed_attempts` FROM `users` `user` WHERE `user`.`uuid` = ? -- PARAMETERS: ["<<redacted>>"]
query: SELECT `roles`.`uuid` AS `roles_uuid`, `roles`.`name` AS `roles_name`, `roles`.`version` AS `roles_version`, `roles`.`created_at` AS `roles_created_at`, `roles`.`updated_at` AS `roles_updated_at` FROM `roles` `roles` INNER JOIN `user_roles` `user_roles` ON `user_roles`.`user_uuid` IN (?) AND `user_roles`.`role_uuid`=`roles`.`uuid` -- PARAMETERS: ["6bac8da5-2d5d-451a-888c-ca2ce4d31637"]
query: SELECT `shared_vault_user`.`uuid` AS `shared_vault_user_uuid`, `shared_vault_user`.`shared_vault_uuid` AS `shared_vault_user_shared_vault_uuid`, `shared_vault_user`.`user_uuid` AS `shared_vault_user_user_uuid`, `shared_vault_user`.`permission` AS `shared_vault_user_permission`, `shared_vault_user`.`is_designated_survivor` AS `shared_vault_user_is_designated_survivor`, `shared_vault_user`.`created_at_timestamp` AS `shared_vault_user_created_at_timestamp`, `shared_vault_user`.`updated_at_timestamp` AS `shared_vault_user_updated_at_timestamp` FROM `auth_shared_vault_users` `shared_vault_user` WHERE `shared_vault_user`.`user_uuid` = ? -- PARAMETERS: ["<<redacted>>"]
mannp commented 3 weeks ago

@AdamXweb did you ever get this resolved? I have the same issue of duplicating notes, but LocalStack has no errors.

I do though have the same errors in Revisions-worker.log

AdamXweb commented 2 weeks ago

I ended up switching away from StandardNotes to FSnotes which simplified everything