search

FalkorDB / FalkorDB

A super fast Graph Database uses GraphBLAS under the hood for its sparse adjacency matrix graph representation. Our goal is to provide the best Knowledge Graph for LLM (GraphRAG).
https://www.falkordb.com/
Other
601 stars 21 forks source link

Export from Redis Graph and Import into Falkor #660

Open chpatel2 opened 4 months ago

chpatel2 commented 4 months ago

We have a few instances of Redis graph in production. trying to experiment porting over to Falkor. Is there a way to export from redis graph and then import into Falkor? thanks.

swilly22 commented 4 months ago

Yes, you can export your data via RDB, using redis-cli issue the BGSAVE command to create an RDB from RedisGraph. Once the RDB file is created place it in the same directory from which you'll run FalkorDB and it will pick it up automatically.

chpatel2 commented 2 months ago

I made a dump.rdb file using the BGSAVE command. Attempting to import this into falkor failed, with the error below:

11:C 12 Jun 2024 20:24:57.888 oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo 11:C 12 Jun 2024 20:24:57.888 Redis version=7.2.4, bits=64, commit=00000000, modified=0, pid=11, just started 11:C 12 Jun 2024 20:24:57.888 Configuration loaded 11:M 12 Jun 2024 20:24:57.888 monotonic clock: POSIX clockgettime . .-__ ''-._ _.- .. ''-. Redis 7.2.4 (00000000/0) 64 bit .-.-```. ```\/ _.,_ ''-._ ( ' , .-` | `, ) Running in standalone mode |`-._`-...-` __...-.-.|'` .-'| Port: 6379 | -._. / .-' | PID: 11 -._-. `-./ .-' .-' |`-.-._-..-' .-'.-'| | -._-. .-'.-' | https://redis.io/ `-. -._-..-'.-' .-' |-._-._ -.__.-' _.-'_.-'| |-.`-. .-'.-' | -._-._-.__.-'_.-' _.-' -._ -.__.-' _.-' -. .-' `-.__.-' 11:M 12 Jun 2024 20:24:57.893 Enabled role change notification 11:M 12 Jun 2024 20:24:57.893 Starting up FalkorDB version 4.0.10. 11:M 12 Jun 2024 20:24:57.895 Thread pool created, using 16 threads. 11:M 12 Jun 2024 20:24:57.895 Maximum number of OpenMP threads set to 16 11:M 12 Jun 2024 20:24:57.895 Query backlog size: 1000 11:M 12 Jun 2024 20:24:57.895 Module 'graph' loaded from /FalkorDB/bin/src/falkordb.so 11:M 12 Jun 2024 20:24:57.896 Server initialized 11:M 12 Jun 2024 20:24:57.897 Loading RDB produced by version 7.0.11 11:M 12 Jun 2024 20:24:57.897 RDB age 745 seconds 11:M 12 Jun 2024 20:24:57.897 RDB memory usage when created 5008.62 Mb 11:M 12 Jun 2024 20:24:57.897 # The RDB file contains AUX module data I can't load: no matching module 'scdtype00'

swilly22 commented 2 months ago

Seems like the Redis server you're trying to migrate from runs additional modules. Can you please run the following command: MODULE LIST and share its output?

chpatel2 commented 2 months ago

Not sure if this helps. Thanks. 1) "name" 2) "graph" 3) "ver" 4) "21210" 5) "path" 6) "/var/opt/redislabs/lib/modules/redisgraph.so" 7) "args" 8) (empty list or set) 2) 1) "name" 2) "ReJSON" 3) "ver" 4) "20407" 5) "path" 6) "/var/opt/redislabs/lib/modules/rejson.so" 7) "args" 8) (empty list or set) 3) 1) "name" 2) "ai" 3) "ver" 4) "10205" 5) "path" 6) "/var/opt/redislabs/lib/modules/redisai.so" 7) "args" 8) (empty list or set) 4) 1) "name" 2) "search" 3) "ver" 4) "20609" 5) "path" 6) "/var/opt/redislabs/lib/modules/redisearch.so" 7) "args" 8) (empty list or set) 5) 1) "name" 2) "bf" 3) "ver" 4) "20209" 5) "path" 6) "/var/opt/redislabs/lib/modules/redisbloom.so" 7) "args" 8) (empty list or set)

swilly22 commented 2 months ago

It does, thanks!

The issue here is clear, there's an additional module which saves data into the RDB file, that module must be present in FalkorDB for the RDB to load.

Although there are other ways to import/export the data, do you know how big is your graph? Would you mind getting the graph config? share the output from the command: GRAPH.CONFIG GET *

jmaness commented 2 months ago

I've been investigating this along with @chpatel2, and the missing modules previously used have been added now. However, the following error on redis startup is occurring. 

1:M 14 Jun 2024 22:07:27.191 * Server initialized
1:M 14 Jun 2024 22:07:27.200 * Reading RDB base file on AOF loading...
1:M 14 Jun 2024 22:07:27.200 * Loading RDB produced by version 7.0.11
1:M 14 Jun 2024 22:07:27.200 * RDB age 612091 seconds
1:M 14 Jun 2024 22:07:27.200 * RDB memory usage when created 5009.46 Mb
1:M 14 Jun 2024 22:07:27.200 * RDB is base AOF
1:M 14 Jun 2024 22:07:27.200 # The RDB file contains AUX module data I can't load: no matching module 'graphdata'

I found https://stackoverflow.com/questions/76928167/how-to-migrate-redis-rdb-file-created-with-a-module-to-a-deployment-without-that and https://github.com/redis-stack/redis-stack/releases/tag/v7.2.0-v5 that appear to related to the same error we are seeing.

Offhand, it seems like FalkorDB might not be providing a module named graphdata.

swilly22 commented 2 months ago

Hi @jmaness, @chpatel2 FalkorDB offers only the graphdata datatype as can be seen in the following:

127.0.0.1:6379> keys *
1) "telemetry{falkordb-py}"
2) "falkordb-py"
127.0.0.1:6379> type falkordb-py
graphdata

If we try to load an RDB which contains data related to any other module e.g. scdtype00 then the loading procedure would fail, as it is missing a required module, this is possible when migrating from RedisEnterprise to Falkor as RedisEnterprise runs multiple modules on their servers.

This is why I would like to suggest migrating your graph(s) using a different approach, using Redis DUMP command to export an individual key and importing it using Redis RESTORE command. For that to work I need additional information from your end, specifically I would like to understand how big are your graphs (in terms of number of nodes and edges).

Thank you.