Cannot set a 250mb string

[Tomato6966]

I get [ErrorReply: ERR Protocol error: invalid multibulk length]

Redis can do it

[romange]

First of all, Redis is great!

Secondly, I chose sane defaults for the initial launch. I prefer having tighter bounds and release them gradually as DF matures.

[Tomato6966]

Hey thanks for your fast response

[Tomato6966]

you claim to be 25x faster then redis, however i did string set del and get tests for 0.05mb and 25mb datas and on high frequently datas such as 10 connections spamming the "db"

redis was 2.1 milliseconds faster overall then dragonfly

i am using an enterpries amd epyc with ddr4 ecc ram

[romange]

I would love to hear more about your benchmark. Do you have a reproducible script? what hardware did you run it on? I am going to release a version with debug symbols. Will you be able to send me the profile data? I will show you how to profile dragonfly.

[Tomato6966]

I would love to hear more about your benchmark. Do you have a reproducible script? what hardware did you ri

I posted all stats in a discord channel for discord-creators may I invite you?

[romange]

yes, sure. Did not know there is discord channel for DF

[romange]

shoot me the invite to roman@dragonflydb.io

[Tomato6966]

no i didnt, can u send the invite of your dc in here, since there is eitherway no admin online for you to verify you... sadge

[Tomato6966]

or here is the link to join, then click on the verify ticket https://discord.gg/pMc6jFZe

[romange]

@Tomato6966 https://discord.gg/HfertDn

[romange]

For the protocol - to test the following scenario on x86 machine:

redis-benchmark  -c 50 --threads 4 -t set -n 10000 -d 512000

[Tomato6966]

const redisOptionsdragonflydb = {
  password: redis.password, 
  url: redis.url // "redis://HOSTNAME:PORT",
  retry_strategy: () => 1000
}
const redisClient = redis.createClient(RedisSettings)
redisClient.connect().then(() => {
  redisClient.set("KEY", "DATA");

  // tests
  let averages = [];
  const key = "settings_773668217163218944";
  const Data = JSON.stringify(getDataSomehow("https://sourceb.in/I7ChCQGwqg"))
  let DateTime = Date.now();
  for(let i = 0; i<=100;i++) {
    console.log(`#${i}`)
    let Da = Date.now();

    await client.set(key+"_"+DateTime+"_"+i, Data)
    await client.get(key+"_"+DateTime+"_"+i)

    averages.push({
      time: Date.now() - Da,
      used: Math.round(process.memoryUsage().heapUsed / 1024 / 1024 * 100) / 100,
      total: Math.round(process.memoryUsage().heapTotal / 1024 / 1024 * 100) / 100,
      rss: Math.round(process.memoryUsage().rss / 1024 / 1024 * 100) / 100
    })
  }
  console.table({
    time: averages.map(d => d.time).reduce((a,b) => a+b,0)/averages.length,
    used: averages.map(d => d.used).reduce((a,b) => a+b,0) / averages.length,
    total: averages.map(d => d.total).reduce((a,b) => a+b,0) / averages.length,
    rss: averages.map(d => d.rss).reduce((a,b) => a+b,0) / averages.length
  })
});

npm install redis

[romange]

@Tomato6966 it was great chatting with you today. I was impressed by how you record everything you do at online. Any tips on how to do this easily? :)

I will benchmark large keys operations in the coming weeks. In any case, the main key takeaway is that throughput won't increase with single connection no matter how strong the server is because it's only determined by latency. Throughput = 1/latency

in order to utilize the server efficiently you should use multiple connections - I've sent you a pointer to some nodejs candidates but I am not sure how good they are.

One final thing. I did not completely understand your use-case but usually storing huge blobs are signs of suboptimal design of an app developer. It may be easier with the PoC but later it can bite you due to unreasonable bandwidth and memory requirements. If you update json objects, I suggest that you check if it's possible to know what fields have changed and then use hashes (hset) to update specific fields.
See here for a SO discussion regarding this subject.

Please tell me if I can close this issue.