Unable to successfully retrieve my v1 secrets.

[ameba23]

So im talking here about our 'development' branch. Maybe the development PR is a better place for this conversation than an issue but whatever.

Although all tests pass, I can no longer recover any of my existing secrets, using patchbay-dark-crystal OR the command line client.

Two of them allow me to recombine but the secret appears as random characters, and one of them shows the error message 'not enough valid shards to recombine'.

From a bit of console logging, i was able to determine that not enough shards are making it to 'lib/secrets-wrapper/v1' (it is getting an array of 2 shares, when there are three valid 'invite-reply' messages with shares for the secret i am looking at right now.

Looking at 'recover/async/mend', if i output the shardsData argument which is passed to the function getRequestedShards, dump it to a json file and apply shardsData.map(m => m.requestsData) i get the following:

[ [ { request: [Object], reply: [Object] } ],
  [ { request: [Object] } ],  // <--- this should have a reply
  [ { request: [Object], reply: [Object] } ],
  [] ]

This is a secret with 4 shards and a quorum of 3. It is expected that the final shard has no requests or replies. To see what i mean look at the same secret viewed in patchbay:

I can't work out at what point this reply message is going missing.

Would be interested if anyone is having similar problems?

[mixmix]

Can't see this with v1 shards of my own....

[ameba23]

here is output of fetch. i had to add the txt extension for github to allow me to upload the file out2.json.txt

[mixmix]

Can you do some logging around here @ameba23 : https://github.com/blockades/scuttle-dark-crystal/blob/development/recover/async/fetch.js#L88-L99

[mixmix]

github gist makes this easier!

https://gist.github.com/mixmix/690d9d7704d68ff0e9a7a91100ec7e7e

[mixmix]

Right so in the new code I gather dialogue by pulling out the 'custodian' (not you) part of the message https://github.com/blockades/scuttle-dark-crystal/blob/development/recover/async/fetch.js#L88 Then I try finding a Reply for each Request. I noted in there that what I have written is brittle https://github.com/blockades/scuttle-dark-crystal/blob/development/recover/async/fetch.js#L132-L143 You might be hitting just this brittleness, can you have a look in there and see if it's where things are going wrong @ameba23 ?

In that older view code, I generate a "dialogueKey" which is your recps joined into a string https://github.com/blockades/patchbay-dark-crystal/blob/development/views/crystals/show.js#L77-L97

[ameba23]

yep you're right thats exactly where the problem is:

request.key:                        %rwVXHFs96LULSIFMe0n9nFQTahBzLuoB1B9hKlIJ7lk=.sha256
replies[0].value.content.branch:  [ '%rwVXHFs96LULSIFMe0n9nFQTahBzLuoB1B9hKlIJ7lk=.sha256',
  '%z0Spt7xUzQ4Qs16pgkmydqkcofj8VM4Btp6H5Eiw+Mw=.sha256' ]

request.key:                        %pRJEhXsao0bVPLzZA+3epcx1AonC/URlnCVYfMTFbqU=.sha256
replies[0].value.content.branch:  [ '%OX8zPyUO5fdux4hkkCcmCmdIk2b4hpr166cMtOUWLxo=.sha256',
  '%pRJEhXsao0bVPLzZA+3epcx1AonC/URlnCVYfMTFbqU=.sha256' ]

request.key:                        %aoC+01sWRoHKfB9lN/Mtlm2b8hKt2HQG3Pv7NBL5gjk=.sha256
replies[0].value.content.branch:  [ '%aoC+01sWRoHKfB9lN/Mtlm2b8hKt2HQG3Pv7NBL5gjk=.sha256',
  '%uRDHJeYfMBWBxiJT0OSNC3dWGAY4TfGxK//Lfv2Ri8E=.sha256' ]

notice that with the middle one (which is causing the problem) the the invite is not the first item in the branch array. So 'getReply' needs an extra .find, one to find the reply, and one to find the right branch.

i just made a fix but check it (and maybe make prettier)

[ameba23]

so now my first two secrets are now combining just fine. but the third one (which was giving an error message before) is a collection of random characters.

although it might have been doing that before, i can't remember. will investigate...

[ameba23]

Wierd. for this secret i have 3 replies with shards, but only one of them is making it to secretsWrapper combine.

lets take a look:

request.key:  %fTXgajRItGEL//44ku6EwtckRe4KtqeA6RIp8BzGwX4=.sha256
replies branch:  [ [ '%/2SlJyXPoteq++ztDzd+TihH9bXLeJB5qTblRxlWWx8=.sha256' ] ]
request.key:  %pxg9KPNcPBAj/nVxqPa2mZqKxSBV3hrhDTIk7emX1r8=.sha256
replies branch:  [ [ '%pDXW8kdxMwU1Rcjs0m4Zq9dLM26WldOlOCPXm9J32PQ=.sha256' ] ]
request.key:  %F3s7JSlVBXP02770qrCclN/ZKakqx4cT+qM/VfSJql8=.sha256
replies branch:  [ [ '%+Y/8Dj/3yfAt8QoEambdgNtj4B/FRSOMrKnt8npalKY=.sha256',
    '%F3s7JSlVBXP02770qrCclN/ZKakqx4cT+qM/VfSJql8=.sha256' ] ]

the last one is fine, but the first two have a branch of a request we dont know about. multiple requests?

i should mention that this is my oldest secret, quite possibly it was put together when things were in a state of not totally working yet.

[ameba23]

even wierder. i looked up one of the 'wrong' items in branch:

key: '%/2SlJyXPoteq++ztDzd+TihH9bXLeJB5qTblRxlWWx8=.sha256',
  value: {
    previous: '%rzLfBaSyR492ral9YyVde0fW66lWS7oauKW9YUFVlcQ=.sha256',
    sequence: 1540,
    author: '@MpDd66GPXgN1+eMNrZInHkWq1THMurWwLdMx8BZ1ncw=.ed25519',
    timestamp: 1534252041719,
    hash: 'sha256',
    content: {
      root: '%dmCKvxJeUMqQItA5pneX44lQokGilUVoBS9l/KcF6lE=.sha256',
      branch: [
        '%fTXgajRItGEL//44ku6EwtckRe4KtqeA6RIp8BzGwX4=.sha256',
        '%ry93VL+Vjymk/oKkoajjwzPMHBJrfmA88JCvi4seU70=.sha256',
      ],
      accept: true,
      recps: [
        '@MpDd66GPXgN1+eMNrZInHkWq1THMurWwLdMx8BZ1ncw=.ed25519',
        '@vEJe4hdnbHJl549200IytOeA3THbnP0oM+JQtS1u+8o=.ed25519',
      ],
      type: 'invite-reply',
      version: '1',
    },

its another invite-reply message. with no body. and the correct request in the branch array. but whats the other item in the branch array? a 'dark-crystal/shard' message. am i going crazy?

im pretty confused but my only explanation is that this was made in an early stage of development...

[mixmix]

the branch array can have more than one entry - it's not actually a guarentee that it points to the "heads" of the thread. Which can be 2 messages ... if they were posted by different people before they were able to sync. But if it's from the same person this is just bad code ...

This is annoying huh. I think we should tighten up the code some (I'll go look at your PR) then make some fresh v1 shards or get others like Dan and Kieran to test recombining all their old shard

[mixmix]

yeah that PR bugfix seems good to me @ameba23

I don't have mental power left to dig into your logs above closely right now. Perhaps we can assess if this is urgent to do more work on verbally at some point

[m4gpi]

@ameba23 did this get resolved?

[ameba23]

@KGibb8 i would say its as resolved as it ever will be, so im closing the issue.

we still have a few broken mesages in our feeds from early testing, but we are now dealing with them as gracefully as we can.