Migrating from a keyv backend to another (redis -> postgres)

[GuillaumeV-cemea]

Hi,

I'm using excalidraw-storage-backend with redis backend for keyv (which was the preferred solution when I installed it), but I'm having some issue with "old" (couple of weeks/months) data disappearing, and read-only link / collab link beeing broken.

From what little I understand about redis, this might be coming from redis deleting old keys. I would like to migrate to another keyv backend (postgres), which I have a far better understanding of, however I couldn't find any useful resource on this.

I have 2 questions :

1. Would migrating to postgres actually solve my disappearing data issue, or is this coming from another thing entirely ?
2. Do you have any resource to migrate this data ? If not, I'll try a simple npm program : connect to old keyv, connect to new keyv, get all keys from redis, insert all keys into postgres

[JannikStreek]

HI @GuillaumeV-cemea (and sorry for the late reply), I guess thats the preconfigured TTL that is having an effect on you. The backend automatically deletes data after some time. You can configure it with STORAGE_TTL (using a large value). What's the value you have configure there?

It's quite easy to migrate keyv to Postgres. In fact I already changed the docker compose file and other files accordingly: https://github.com/kitsteam/excalidraw-storage-backend/blob/main/docker-compose.yml

Our default is Postgres for now. But the TTL does also have an effect there, so its the wrong solution for your problem.

[GuillaumeV-cemea]

No issue :-)

My STORAGE_TTL seems large enough : 2592000000. My complete docker-compose is as follow :

version: "3.8"

services:
  excalidraw:
    build:
      context: .
      target: production
    container_name: excalidraw
    ports:
      - "5010:80"
    restart: always
    stdin_open: true
    healthcheck:
      disable: true
    environment:
      - NODE_ENV=production

  excalidraw-storage-backend:
    build: https://github.com/kitsteam/excalidraw-storage-backend.git#main
    ports:
      - "5011:8080"
    restart: always
    environment:
      STORAGE_URI: redis://:${REDIS_PASSWORD}@redis:6379
      STORAGE_TTL: 2592000000

  excalidraw-room:
    image: excalidraw/excalidraw-room
    restart: always
    ports:
      - "5012:80"

  redis:
    image: redis
    command: redis-server --requirepass ${REDIS_PASSWORD}
    restart: always
    volumes:
      - redis_data:/data

volumes:
  notused:
  redis_data:

[JannikStreek]

2592000000 means 1 month. It's given in milli seconds, as far as I know. See:

 await keyv.set('foo', 'expires in 1 second', 1000); // true

So reading your issue again:

I'm having some issue with "old" (couple of weeks/months) data disappearing

Sounds like thats the issue.

[GuillaumeV-cemea]

I feel very stupid, I thought in was in seconds :) I updated the TTL to something long-term, and I'll stay with redis then (this way I don't have to migrate the data).

Thanks !