Example of getting messages with blobs

[nichoth]

If we travel back in time, ssb-server includes this example:

 // monitor the feed for new links to blobs
  pull(
    sbot.links({dest: '&', live: true}),
    pull.drain(function (data) {
      // do we have the referenced blob yet?
      sbot.blobs.has(data.dest, function (_, has) {
        if(!has) createJob(data.dest, data.source)
      })
    })
  )

How would you do the same thing in ssb-db2? "monitor the feed for new links to blobs"?

[staltz]

You could use jitdb's predicate to check that a mention starts with the character & and run it with live(), or you could use the fullMentions leveldb in ssb-db2 but you'd have the add more code to the source code for doing this live logic.

[nichoth]

you could use the fullMentions leveldb in ssb-db2

The fullMentions example in the readme is like this:

sbot.db.query(
  where(and(type('post'), fullMentions(alice.id)))),
  toCallback((err, msgs) => {
    console.log('There are ' + msgs.length + ' messages')
    sbot.close()
  })
)

But it can filter just based on the prefix of the string that you pass to it also?

[staltz]

As I said, you will have to add more code to the source code at ssb-db2/indexes/full-mention.js, using pull-level to query the leveldb index, example: https://github.com/ssb-ngi-pointer/ssb-db2/blob/7f23695f8bd59fc8f3ee3f476e1b0f5da69ec6f9/indexes/full-mentions.js#L60-L61

[nichoth]

thanks @staltz

could use jitdb's predicate to check that a mention starts with the character & and run it with live()

So something that is not totally clear to me about the new DB — flume was basically a kappa structure DB, where everything was a stream, and you would process messages in order.

The thing I want to avoid is my server having to re-process the same messages each time you restart the server. It sounds like for that you would need to create some kind of plugin that stores its last state on the hard drive. Is that right?

[staltz]

Jitdb doesn't re-process old messages upon restarting, because jitdb indexes are persisted to disk.

[nichoth]

thanks @staltz

It's a little bit unclear how the seek function works with the predicate operator

            // The `seekType` function takes a buffer and uses `bipf` APIs to search for
            // the fields we want.
            const bValue = Buffer.from('value') // better for performance if defined outside
            const bContent = Buffer.from('content')
            const bType = Buffer.from('type')
            function seekBlob(buffer) {
                var p = 0 // p stands for "position" in the buffer, offset from start
                p = bipf.seekKey(buffer, p, bValue)
                if (p < 0) return
                p = bipf.seekKey(buffer, p, bContent)
                if (p < 0) return
                return bipf.seekKey(buffer, p, bType)
            }

            sbot.db.query(
                where(
                    and(
                        type('post'),
                        predicate(seekBlob, arg => {  // what is passed to this function?
                            console.log('***arg***', arg)
                        }, { name: 'blobs' })
                    )
                ),
                descending(),
                paginate(10),
                toCallback((err, res) => {
                    if (err) return console.log('arrrrr', err)
                    console.log('res', res)
                })
            )

The predicate fn has a signature like predicate(seek, fn, opts). What is passed to the middle function argument?

We want to gossip blobs in this case along with the messages.

[staltz]

@nichoth these things are in the docs (readme) for jitdb already. See again:

predicate(seek, fn, opts) filters for messages where a seeked field is passed to a predicate function fn and the fn returns true

seek is already explained in the jitdb readme, it "seeks" for a field's value in a message. That value is passed to fn and you're supposed to return a boolean indicating whether that value is desirable for your query results or not.