Uncaught ERR_SOCKET_DGRAM_NOT_RUNNING when leaving network in joinNetwork callback

jaller94 commented 5 years ago

I am reporting:

[x] a bug or unexpected behaviour
Bug Report

Please give us details about your installation to assist you. Run dat -v to see the version of Dat you are using.

Operating system: Manjaro Linux
Node Version: v12.9.1
dat-node Version: 3.5.15

Expected behavior

All errors are catchable

Actual behavior

The following code sometimes crashes on me.

const folderPath='./archives/0';
const folderPath='./test-keys';
Dat(folderPath, {secretDir: path.join(rootFolder, 'secret_keys')}, function (err, dat) {
  if (err) reject(err);
  dat.importFiles();
  // It did not crash without this.    
  const network = dat.joinNetwork(() => {
    dat.leaveNetwork();
    console.log(`Leaving ${folderPath}`);
  });
});

Sometimes it just crashes. Note that I am opening 163 archives at the same time, waiting for them to finish their initial sync.

dgram.js:831
    throw new ERR_SOCKET_DGRAM_NOT_RUNNING();
    ^

Error [ERR_SOCKET_DGRAM_NOT_RUNNING]: Not running
    at healthCheck (dgram.js:831:11)
    at Socket.send (dgram.js:609:3)
    at RPC.send (/home/jaller94/Git/fdroid2dat/node_modules/k-rpc-socket/index.js:144:15)
    at RPC.query (/home/jaller94/Git/fdroid2dat/node_modules/k-rpc-socket/index.js:189:8)
    at RPC.query (/home/jaller94/Git/fdroid2dat/node_modules/k-rpc/index.js:146:17)
    at DHT._sendPing (/home/jaller94/Git/fdroid2dat/node_modules/bittorrent-dht/client.js:213:13)
    at test (/home/jaller94/Git/fdroid2dat/node_modules/bittorrent-dht/client.js:168:10)
    at DHT._checkNodes (/home/jaller94/Git/fdroid2dat/node_modules/bittorrent-dht/client.js:186:3)
    at DHT._checkAndRemoveNodes (/home/jaller94/Git/fdroid2dat/node_modules/bittorrent-dht/client.js:152:8)
    at KBucket.<anonymous> (/home/jaller94/Git/fdroid2dat/node_modules/bittorrent-dht/client.js:58:10)

The traceback doesn't look like dat-node is at fault, but I have to start somewhere on my journey to track down the issue. Maybe someone here knows more?

Debug Logs

Output is not any different when running it with DEBUG=dat.

RangerMauve commented 5 years ago

dat-node is being replaced by the Dat Software Development Kit, could you try using that instead and seeing if it works better?

jaller94 commented 5 years ago

@RangerMauve Ok, cool. Is there an equivalent of Dat(folderPath, {secretDir})?

There does not seem to be an example showing how to load an archive and then write to it. The only example that writes a file, creates a new archive.
I don't know how to tell which secret key belongs to which dat archive.

RangerMauve commented 5 years ago

Hmm, the SDK doesn't have anything for secret storage at the moment. It just stores the secret along with the rest of the data.

RangerMauve commented 5 years ago

Could you describe what you're working on so I can get a better picture of what needs to be done?

jaller94 commented 5 years ago

I'm downloading apps from the F-Droid store and create/update one dat archives per app. The dat archives can then be loaded by Dat Installer to install them on an Android phone.

Repo: https://gitlab.com/jaller94/fdroid2dat

RangerMauve commented 5 years ago

It might make sense to use dat-store for keeping stuff online. You should be able to cURL your fdroid data, and then add the folder to dat-store which will turn it into an archive and watch for file changes.

RangerMauve commented 5 years ago

dat-store should also be handling the networking layer in a more effective way (using the SDK). dat-node creates a separate swarm per archive which might get kinda heavy after a while, but dat-store shares a single swarm instance for all the archives.

jaller94 commented 5 years ago

Yes, I use dat-store on a remote server to keep the archives online. However, the creation of archives should be a oneshot job that can be run on a timer and should end when all updated repos are synced with at least one peer in the network (or after a timeout of 10 minutes or so).

The F-Droid repo has 2000+ apps and keeping all of them online on one machine (my laptop in this case) does not seem to be a good idea. That's why I want to listen for the initial sync and then move on to sync the next few hundreds of archives.

jaller94 commented 5 years ago

Ideally I would only want to connect to my list of peers, wait for them to reach 100% sync on the latest version and then disconnect, leaving it up to them to propagate it to other peers in the public network.

RangerMauve commented 5 years ago

Dang, the syncing with peers thing is closely related to this issue, I think. 😅

With the SDK, you can specify a path as the first argument to Hyperdrive and that'll be enough to turn it into an archive. To do a sync with a folder, you can use the diff-file-tree module.

jaller94 commented 5 years ago

My other option is to "blindly" keep them online for 5 minutes, then remove them from dat-store and add the next batch. Anyway, this may be getting a bit to off-topic. If things are moving to Dat SDK, I'll see what I can build with that one, instead of poking at issues of dat-node.

dat-ecosystem-archive / dat-node