support ssb-tribes

[staltz]

@arj03 @cryptix @mixmix I think fixtures now can generate: (1) tribe creation, (2) tribe invitation to other feeds, (3) post a message inside tribes, all using ssb-tribes. I think it's working, but it would be good to be sure.

How to test this out:

• git clone
• checkout branch tribes
• npm install
• npm run compile
• node lib/bin.js
• check ./data dir when it's done

Note, you can get the secret of the 3 most popular feeds, they are in the generated ./data/.

[mixmix]

I guess the test of this is if you try and use the fixtures. Might be good to list what you expect a person to be able to do with these fixtures (and what they won't be able to do /see if anything)

Eg are there feeds in here which aren't in group and so cannot decrypt? Which ones are they

[staltz]

Eg are there feeds in here which aren't in group and so cannot decrypt?

Yes

Which ones are they

It's probabilistic

[mixmix]

I think what I'm wondering is if it would be good to have an output at the end which summarises who is in each group (and what dm groups exist) so you know if your tests are working when you decrypt. Do you have that? (sorry if you do, ts is a bit dense when read on a phone)

[staltz]

Great idea! We already have a "report" module, but it doesn't report on tribes, and it could

[arj03]

I was testing this using the instructions posted and ran into this problem:

UnhandledPromiseRejectionWarning: Error: data.recps has less items than allowed
    at groupInvite (/home/arj/dev/ssb-fixtures/node_modules/ssb-tribes/method/group/add-member.js:28:38)
    at Timeout._onTimeout (/home/arj/dev/ssb-fixtures/node_modules/ssb-tribes/method/index.js:32:54)

it was spitting out a ton of debug I could see.

[staltz]

@arj03 how many authors do you configure it to run with?

[arj03]

I just ran it with the defaults.

[staltz]

Weird, then I'm a bit lost what is the problem... could try to figure it out together in a screen share

[arj03]

I think you can merge this. Still need to test it properly with ssb-db2. It's up to you.

[staltz]

Well, when this is merged, it would be a breaking change, so leading to v3 fixtures. I'd rather make sure that what this PR does is correct, so not in a hurry of merging.

[staltz]

Maybe we could try getting this one merged this week?

[arj03]

Yes, I will schedule this for friday for me to test.

[arj03]

I'm testing this, I can't get my code to decode the groups but I see there is something wrong with the way they are created. I added some debug to tribe creation and what I see is this:

@V4UXI created tribe %CU2Qd // <-- ?
{
  key: '%jE3SudINIR0Ht9b5kzxAWFJ6uqwAyiWceLb4BUxHofA=.sha256',
  value: {
    previous: '%u4s07Uq7mMMtw6VBdB9xc0PYgZw1Sv2W508H5jMAnbo=.sha256',
    sequence: 769,
    author: '@YHiUEFOxW5nJr8XU2SsX1LsLVJThs/pq5sr5Uxf59HQ=.ed25519', // <-- ?
    timestamp: 1440904785000,
    hash: 'sha256',
    content: 'zNSpJxlgq2lB7m7EPDvbpJUeBxRRXOCMgTd5zDYTxKTWE6yAyeypspLgDWzCqLZn0gMUdmjE59C1Wno5Ka/f4cTxK2YTMbz44iDtLJqK9/3qM7IACpFU+UmdrqlLjeU169yB6gxrx0TIHQzbDg6S1Oqe80kHg+X5AY5+cEm3wnnfVeLnfQYwmDurU/iNVMM1JW0QDt1ufg==.box2',
    signature: 'EmzTwoZA6g4WdJdiHE895SuhSGbv1qtLM8k3KnQH1K8oTqpk0B1Z/uM+K3Dv4WIj+ENtmnBj1E2b5UrSgRdaAQ==.sig.ed25519'
  },
  timestamp: 1440904905000
}

So this means that all groups are creating by the hops0 user.

[staltz]

Oooh, I see the problem now. ssb-tribes does the sbot.publish see source code so it dictates who the author is. Otherwise in ssb-fixtures we use sbot.createFeed() then author.add.

Do you have any ideas how we could solve this without changes in ssb-tribes? Or do we need changes in ssb-tribes?

@mixmix ?

[mixmix]

I don't use createFeed (and I notice it's marked as deprecated but also in use in places?). Anyway no idea what it would do with tribes.

My approach to starting many different groups from many different authors is use scuttle-testbot to easily spin up many instances and perform operations from there. Might not be fastest, but it works well (and no semi - supported methods!)

Look in the tests of ssb-tribes for a helper test bot file which is the form you want

[staltz]

I updated this PR to be based on scuttle-testbot (pending on PR https://github.com/ssbc/scuttle-testbot/pull/13 being merged).

@arj03 can you test?

[arj03]

I ran the new code. Tried deleting the data dir to make sure everything was fresh, then I got a small error because there seems to be a missing mkdir? After creating the dir and running this I get an error:

node lib/bin.js
(node:438019) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 wakeup listeners added to [EventEmitter]. Use emitter.setMaxListeners() to increase limit
(Use `node --trace-warnings ...` to show where the warning was created)
Error: get-group-tangle: unknown groupId %OJwo29P3LvOW7U2KGWAdtHy59YZpThFTVBBrIdOiPb4=.cloaked
    at getGroupTangle (/home/arj/dev/ssb-fixtures/node_modules/ssb-tribes/lib/get-group-tangle.js:14:26)
    at /home/arj/dev/ssb-fixtures/node_modules/ssb-tribes/index.js:116:5
    at /home/arj/dev/ssb-fixtures/node_modules/hoox/index.js:3:17
    at hooked (/home/arj/dev/ssb-fixtures/node_modules/hoox/index.js:10:15)
    at /home/arj/dev/ssb-fixtures/node_modules/promisify-4loc/index.js:3:11
    at new Promise (<anonymous>)
    at /home/arj/dev/ssb-fixtures/node_modules/promisify-4loc/index.js:2:3
    at /home/arj/dev/ssb-fixtures/lib/index.js:87:43
    at Generator.next (<anonymous>)
    at fulfilled (/home/arj/dev/ssb-fixtures/lib/index.js:5:58)
(node:438019) UnhandledPromiseRejectionWarning: Error: unknown groupId %OJwo29P3LvOW7U2KGWAdtHy59YZpThFTVBBrIdOiPb4=.cloaked, cannot encrypt message
    at /home/arj/dev/ssb-fixtures/node_modules/ssb-tribes/envelope.js:32:29
    at Array.reduce (<anonymous>)
    at boxer (/home/arj/dev/ssb-fixtures/node_modules/ssb-tribes/envelope.js:29:40)
    at box (/home/arj/dev/ssb-fixtures/node_modules/ssb-db/autobox.js:18:18)
    at dbAppend (/home/arj/dev/ssb-fixtures/node_modules/ssb-db/minimal.js:149:23)
    at /home/arj/dev/ssb-fixtures/node_modules/ssb-db/minimal.js:36:37
    at AsyncJobQueue.runAll (/home/arj/dev/ssb-fixtures/node_modules/ssb-db/util.js:197:32)
    at /home/arj/dev/ssb-fixtures/node_modules/ssb-db/minimal.js:36:24
    at /home/arj/dev/ssb-fixtures/node_modules/ssb-db/minimal.js:46:33
    at AsyncJobQueue.runAll (/home/arj/dev/ssb-fixtures/node_modules/ssb-db/util.js:197:32)

I wonder if its the same error as Henry ran into.

[staltz]

Indeed, lots of errors. Gonna fix them.

[staltz]

Updated this branch to fix a bunch of errors, but now I'm getting a deterministic error, would need help from @mixmix maybe:

PS: I removed some console.logs from scuttle-testbot and ssb-tribes such as rebuild!!! (ﾉ´ヮ´)ﾉ*:･ﾟ✧ and 4zamJnOv: 33 --> @daV3I9mK (sees: vote) which by the way are annoying as I can't turn them off with an opt and they're on by default.

$ node lib/bin.js --seed=fofa

(node:24944) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 wakeup listeners added. Use emitter.setMaxListeners() to increase limit
@U0S0a created tribe %N9Toa
@hHwN2 created tribe %Cmuc1
@deroA created tribe %zse9M
@tzqcW created tribe %CAjt8
@R7BNB created tribe %ifAGG
@U0S0a invited @JjoXx,@AvdFd,@ufy9m,@ticdx,@ED33o,@VUXDz,@tZ3s7,@A/CDi,@XPVPm to tribe %N9Toa
@ticdx posted to tribe %N9Toa
@YSoCU created tribe %LXq/s
@2aVQd created tribe %I1V2K
@4zamJ created tribe %JEGUN
@ticdx posted to tribe %N9Toa
@+TRWd created tribe %iVmmE
@ticdx posted to tribe %N9Toa
@4zamJ invited @DYKYl,@tzqcW,@MHyhO,@a1RS/,@opada,@HB4JF,@ticdx,@daV3I to tribe %JEGUN
@4zamJ posted to tribe %JEGUN
@8c2+m created tribe %8vftO
@ticdx created tribe %qPV0W
@opada posted to tribe %JEGUN
@km3v4 created tribe %pOZR3
@YSoCU posted to tribe %LXq/s
@DYKYl posted to tribe %JEGUN
@ticdx invited @YSyDx,@qhQAJ,@nMB8t,@S9ah0,@vvXbo,@igXd5,@qtCZl,@7Bs+h,@UIIGm to tribe %JEGUN

/home/staltz/ssb-fixtures/node_modules/.pnpm/levelup@4.4.0/node_modules/levelup/lib/levelup.js:188
        err = new NotFoundError('Key not found in database [' + key + ']', err)
              ^
NotFoundError: Key not found in database [%ABkoVYXwv2+1VWWb7mWnwdFDdIkuaYgTzLTbmPj1Bfc=.sha256]
    at /home/staltz/ssb-fixtures/node_modules/.pnpm/levelup@4.4.0/node_modules/levelup/lib/levelup.js:188:15
    at /home/staltz/ssb-fixtures/node_modules/.pnpm/encoding-down@6.3.0/node_modules/encoding-down/index.js:75:21

[mixmix]

Did you make it so some of those are disabled or disableable? Many of those logs are quite important / useful for understand what's happening in tests.

The key not found in db is something I've seen before and don't know what it is. I'm in holiday at Bob's at the moment. Can pair on this if you'd like when I'm back in Wellington. Feel free to dm on signal

[arj03]

Pushed up the changes @mixmix and I paired on.

This now closer to working but is dependant on this bug in tribes.

And this bug:

# Fatal error in , line 0
# Check failed: result.second.
#
#
#
#FailureMessage Object: 0x7ffdda3eb7c0
 1: 0xa69c61  [node]
 2: 0x1999544 V8_Fatal(char const*, ...) [node]
 3: 0xe41af9 v8::internal::GlobalBackingStoreRegistry::Register(std::shared_ptr<v8::internal::BackingStore>) [node]
 4: 0xb9dc98 v8::ArrayBuffer::GetBackingStore() [node]
 5: 0x9bc360 napi_get_typedarray_info [node]
 6: 0x7f9504eddabd sn_randombytes_buf [/home/arj/dev/ssb-fixtures/node_modules/sodium-native/prebuilds/linux-x64/node.napi.node]
 7: 0x9b40df  [node]
 8: 0xbdc91b  [node]
 9: 0xbddec6  [node]
10: 0xbde546 v8::internal::Builtin_HandleApiCall(int, unsigned long*, v8::internal::Isolate*) [node]
11: 0x13cc9d9  [node]
Illegal instruction (core dumped)

and this one:

(node:97293) UnhandledPromiseRejectionWarning: Error: initial message must have previous: null, on feed:@p05LpBnUlkgg8e3bjhg26iXnvtNcDEXHdKCN6gAZGpU=.ed25519
    at Object.exports.checkInvalidCheap (/home/arj/dev/ssb-fixtures/node_modules/ssb-validate/index.js:155:20)
    at Object.exports.checkInvalid (/home/arj/dev/ssb-fixtures/node_modules/ssb-validate/index.js:168:21)
    at Object.exports.appendKVT (/home/arj/dev/ssb-fixtures/node_modules/ssb-validate/index.js:231:17)
    at Object.exports.append (/home/arj/dev/ssb-fixtures/node_modules/ssb-validate/index.js:262:18)
    at queue (/home/arj/dev/ssb-fixtures/node_modules/ssb-db/minimal.js:119:9)
    at /home/arj/dev/ssb-fixtures/node_modules/ssb-db/minimal.js:36:37
    at AsyncJobQueue.runAll (/home/arj/dev/ssb-fixtures/node_modules/ssb-db/util.js:197:32)
    at EventEmitter.queue (/home/arj/dev/ssb-fixtures/node_modules/ssb-db/minimal.js:36:24)
    at EventEmitter.db.add (/home/arj/dev/ssb-fixtures/node_modules/ssb-db/create.js:95:8)
    at apply (/home/arj/dev/ssb-fixtures/node_modules/muxrpc-validation/index.js:197:15)

[staltz]

Thanks for doing it! Cool to see you dabbling with TypeScript

[mixmix]

I have seen that core dump before. I think this solved it https://github.com/mixmix/ssb-tribes/blob/master/test/helpers/test-bot.js#L23

[mixmix]

The unhandled promise rejection... looks bad. Is it a seq 1 message with a non null previous? No idea how that would be possible

[staltz]

Bye bye PR