search

singnet / das-serverless-functions

0 stars 0 forks source link

Testing memorydb for redis #46

Closed levisingularity closed 8 months ago

levisingularity commented 9 months ago

https://docs.aws.amazon.com/AmazonElastiCache/latest/red-ug/related-services-choose-between-memorydb-and-redis.html

levisingularity commented 9 months ago

2024-01-10 14:29:02 INFO Added 9600000 keys (line count = 28077511) 2024-01-10 14:30:31 INFO Added 9700000 keys (line count = 28369885) 2024-01-10 14:32:02 INFO Added 9800000 keys (line count = 28662563) 2024-01-10 14:33:20 INFO Populating collection incomming_set 2024-01-10 14:33:29 ERROR Error in key-value file incomming_set on line 36853```

This error happened yesterday and today morning while trying to load the database

levisingularity commented 9 months ago

https://www.youtube.com/watch?v=Jbq_XZMZEKY

levisingularity commented 9 months ago

2024-01-19 00:38:13 INFO Parsing /data/netact_nov27/SO_with_definition_2023-11-08.metta 2024-01-19 00:38:13 INFO Skipping parser 2024-01-19 00:38:13 INFO Parsing /data/netact_nov27/PSI_MI_with_definition_2023-11-26.metta 2024-01-19 00:38:13 INFO Skipping parser 2024-01-19 00:38:13 INFO Parsing /data/netact_nov27/Go-Plus-UBERON_with_definition_2023-11-26.metta 2024-01-19 00:38:13 INFO Skipping parser 2024-01-19 00:38:13 INFO Parsing /data/netact_nov27/Go-Plus-GO_with_definition_2023-11-26.metta 2024-01-19 00:38:13 INFO Skipping parser 2024-01-19 00:38:13 INFO Parsing /data/netact_nov27/Go-Plus-CL_with_definition_2023-11-26.metta 2024-01-19 00:38:13 INFO Skipping parser 2024-01-19 00:38:13 INFO Parsing /data/netact_nov27/Go-Plus-CHEBI_with_definition_2023-11-26.metta 2024-01-19 00:38:13 INFO Skipping parser 2024-01-19 00:38:13 INFO Parsing /data/netact_nov27/FBdv_with_definition_2023-11-26.metta 2024-01-19 00:38:13 INFO Skipping parser 2024-01-19 00:38:13 INFO Parsing /data/netact_nov27/FBcv_with_definition_2023-11-26.metta 2024-01-19 00:38:13 INFO Skipping parser 2024-01-19 00:38:13 INFO Parsing /data/netact_nov27/FBbt_with_definition_2023-11-26.metta 2024-01-19 00:38:13 INFO Skipping parser 2024-01-19 00:38:13 INFO Parsing /data/netact_nov27/ECO_with_definition_2023-11-26.metta 2024-01-19 00:38:13 INFO Skipping parser 2024-01-19 00:38:13 INFO Parsing /data/netact_nov27/DO_with_definition_2023-11-26.metta 2024-01-19 00:38:13 INFO Skipping parser 2024-01-19 00:38:13 INFO Populating Redis 2024-01-19 00:38:13 INFO Processing key-value files 2024-01-19 00:38:13 INFO Populating collection incomming_set The load still very slow, after a few minutes I was stuck in this step, nothing else was shown.

levisingularity commented 9 months ago

During the research, I found a relevant piece of information: Amazon ElastiCache proved to be faster than MemoryDB (I posted the explanation with a screenshot of the information earlier in the channel). After discussing with André, he suggested testing the architectures using a smaller database.

levisingularity commented 9 months ago

I ran a test by uploading a reduced bios database to both Elasticache and MemoryDB, and indeed, MemoryDB is taking much longer. I used the same setup, an r7g.xlarge instance with 4 vCPUs and 30 GB of RAM.

das-memorydb.log das-elasticache.log