hypercore doesn't fire `.ready(cb)` when `random-access-memory`

[serapath]

in our project

1. when we run npm start and no files have been created yet:
   1. it needs one start which doesn't work, but creates the first hypercore files
   2. it needs a restart which doesn't work, but creates the second hypercore files
   3. from then on - every additional restart makes it work finally
2. if we start using random-access-memory for creating hypercores here and here it stops working altogether and it is not calling the .ready(cb) callback here and here anymore no matter how often we restart.

[RangerMauve]

Looking into this

[RangerMauve]

Err, can't run it on Windows. 😩 Gonna see if there's anything obvious in the source code.

[RangerMauve]

Could you elaborate on what the side effects of "doesn't work" are the first two times? Are there any stack traces happening?

When you say it stops working with random-access-memory, are the various services running continuously for the duration of the test?

For the substrate simulator, did you intend to omit the response containing the originalFeed [on this line]? That might be causing issues if your code is expecting a key to be returned.

Outside of those questions, you might want to set some console logs or breakpoints inside hypercore to see exactly where ready gets stopped. Add a bunch inside the _open function and lemme know what part seems to get stuck.

[RangerMauve]

 code ELIFECYCLE
npm ERR! errno 1
npm ERR![C:\USERS\MAUVE\PROGRAMMING\DATDOT-SERVICE\LAB\REQUESTER-ENCODER-SEEDER\SUBSTRATE-SIMULATOR] listening on  http://localhost:8989
 service@1.0.0 scenario1:feed2: `node ./lab/requester-encoder-seeder/encoder.js`
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the service@1.0.0 scenario1:feed2 script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     C:\Users\Mauve\AppData\Roaming\npm-cache\_logs\2020-01-22T20_12_13_228Z-debug.log
internal/modules/cjs/loader.js:857
  return process.dlopen(module, path.toNamespacedPath(filename));
                 ^

Error: \\?\C:\Users\Mauve\programming\datdot-service\node_modules\iltorb\build\bindings\iltorb.node is not a valid Win32 application.
\\?\C:\Users\Mauve\programming\datdot-service\node_modules\iltorb\build\bindings\iltorb.node
    at Object.Module._extensions..node (internal/modules/cjs/loader.js:857:18)
    at Module.load (internal/modules/cjs/loader.js:685:32)
    at Function.Module._load (internal/modules/cjs/loader.js:620:12)
    at Module.require (internal/modules/cjs/loader.js:723:19)
    at require (internal/modules/cjs/helpers.js:14:16)
    at Object.<anonymous> (C:\Users\Mauve\programming\datdot-service\node_modules\iltorb\index.js:10:40)
    at Module._compile (internal/modules/cjs/loader.js:816:30)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:827:10)
    at Module.load (internal/modules/cjs/loader.js:685:32)
    at Function.Module._load (internal/modules/cjs/loader.js:620:12)
ERROR: "scenario1:feed2" exited with 1.

[mafintosh]

@serapath can you share a gist or something that illustrates the problem?

[ninabreznik]

@RangerMauve I made a little screencast https://www.loom.com/share/8e926897bc494bcbad033d41511bd1c5

@mafintosh Mauve has some additional info that was not shared here so please let me know if I can explain the problem better for you too. But if the screencast explains it, then please if you have any ideas on why we can't join the swarm, then please let me know :)

Thanks so much to both in advance

[ninabreznik]

Ooops, @RangerMauve I just saw your tip about getting some logs from this._storage.openKey(....

So, this is what I get. I get self, I get key and the state, but there's nothing for state key and sig.

[mafintosh]

You'll either see an error or a ready event fired, can you check which one is triggered?

[ninabreznik]

@mafintosh Where? In Hypercore module ?

[mafintosh]

Yep, feed.on('error', ...) or feed.on('ready, ...)

[ninabreznik]

Ready gets logged. The thing is that @RangerMauve showed me the trick to use reallyReady function to start fetching data from the hypercore only when reallyready

function reallyReady (origHypercore, cb) {
  if (origHypercore.peers.length) {
    origHypercore.update({ ifAvailable: true }, cb)
  } else {
    origHypercore.once('peer-add', () => {
      origHypercore.update({ ifAvailable: true }, cb)
    })
  }
}

So, in my example (when I don't use ram), I check on the receiver side if reallyReady and then read and encode the data. And it works fine. If I trigger fetching data after only .on('ready'...) it doesn't work.

But when I then switch to ram, the code in reallyReady doesn't execute.

[mafintosh]

The above will only trigger if you have a peer connected, is that true in your case? Other than that I should work, which call is hanging exactly in the above snippet?

[ninabreznik]

I did a little screencast again. Hope this explains it.

https://www.loom.com/share/9e2bae0fc1c64d849d95f69a02b7de5b

[ninabreznik]

Just fixed this problem I had with using ram for my hypercore.

Ha! The problem was I created a hypercore AND immediately started working with it. So after hours of banging my head against the wall I went back to the beginning and read the whole hypercore docs and removed all the code and checked out your example.js when I discovered a magic word flush I think it would be super great to add flush to the docs

[RangerMauve]

I think I was bitten by not using flush() recently as well. 😁

[mafintosh]

Why do you need to call flush?

[ninabreznik]

Because I create a new hypercore and then in the same process also join the swarm and seed data to another peer and all this is running in the same process. Without flush() it created a hypercore but when I appended some data and then logged .length it was 0. After I ran flush, I get the length and all the other processes run ok after that.

On Mon, Feb 10, 2020 at 11:00 PM Mathias Buus notifications@github.com wrote:

Why do you need to call flush?

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/mafintosh/hypercore/issues/242?email_source=notifications&email_token=AAIGZS3LYVPROCSAVEX6YO3RCJEIXA5CNFSM4KIPHYS2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOELLMN5Q#issuecomment-584500982, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAIGZS5VUM4XEIVWBGJG2UTRCJEIXANCNFSM4KIPHYSQ .

-- Kind regards,

Nina Breznik, Vision Baker playproject.io wizardamigos.com

twitter.com/ninabreznik

Telegram: +1 510 747 84 39 gitter.im/ninabreznik

[RangerMauve]

Same thing seemed to be happening to me.

[mafintosh]

Did you wait for the append cb?

[ninabreznik]

Sweet. This does the job. I replaced the flush with append cb. DIdn't think of that :) Thanks

On Tue, Feb 11, 2020 at 3:30 PM Mathias Buus notifications@github.com wrote:

Did you wait for the append cb?

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/mafintosh/hypercore/issues/242?email_source=notifications&email_token=AAIGZS3WZP6JDK2TVJFYWMDRCMYJXA5CNFSM4KIPHYS2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOELOREAI#issuecomment-584913409, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAIGZS5XYRWLT7SLL2TGTTDRCMYJXANCNFSM4KIPHYSQ .

-- Kind regards,

Nina Breznik, Vision Baker playproject.io wizardamigos.com

twitter.com/ninabreznik

Telegram: +1 510 747 84 39 gitter.im/ninabreznik