search

dat-ecosystem / dat

:floppy_disk: peer-to-peer sharing & live syncronization of files via command line
https://dat.foundation
BSD 3-Clause "New" or "Revised" License
8.24k stars 449 forks source link

Clone and pull from npm.dathub.com fail with message "premature close". #408

Closed kemitchell closed 9 years ago

kemitchell commented 9 years ago

Perhaps this has something to do with:

  1. size of the dat?
  2. server settings on dathub.org?
kyle@dev:~$ dat --version
7.1.1
kyle@dev:~$ curl npm.dathub.org | json
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   280    0   280    0     0   1214      0 --:--:-- --:--:-- --:--:--  1217
{
  "transaction": false,
  "checkout": false,
  "heads": 1,
  "modified": "2015-07-28T15:40:58.378Z",
  "datasets": [
    "modules"
  ],
  "rows": 168306,
  "files": 1,
  "versions": 2,
  "size": 1763343976,
  "version": "ef655da237dd495d6233b56f97688c72e573923b7e617b02d09e94bec1271968",
  "dat": {
    "version": "7.1.1",
    "name": "npm"
  }
}
kyle@dev:~$ dat clone --verbose http://npm.dathub.org dat-npm
premature close
Error: premature close
    at onclose (/home/kyle/.local/lib/node_modules/dat/node_modules/end-of-stream/index.js:43:54)
    at emitNone (events.js:72:20)
    at emit (events.js:166:7)
    at Duplexify._destroy (/home/kyle/.local/lib/node_modules/dat/node_modules/transport-stream/node_modules/duplexify/index.js:191:8)
    at /home/kyle/.local/lib/node_modules/dat/node_modules/transport-stream/node_modules/duplexify/index.js:174:10
    at doNTCallback0 (node.js:407:9)
    at process._tickCallback (node.js:336:13)
mafintosh commented 9 years ago

Which version of node do you have installed?

kemitchell commented 9 years ago

@mafintosh, thanks!

$ node --version
v4.0.0

I'm about to try it with 0.12.

mafintosh commented 9 years ago

@kemitchell let me know if that works. i think 4.0.0 might have changed something in regards to http stream events as i've seen this bug elsewhere on 4.0.0

kemitchell commented 9 years ago

Reproduced on v0.12.7.

kyle@dev:~$ node --version
v0.12.7
kyle@dev:~$ dat --version
7.1.1
kyle@dev:~$ dat clone --verbose http://npm.dathub.org dat-npm
premature close
Error: premature close
    at onclose (/home/kyle/.local/lib/node_modules/dat/node_modules/end-of-stream/index.js:43:54)
    at emit (events.js:129:20)
    at Duplexify._destroy (/home/kyle/.local/lib/node_modules/dat/node_modules/transport-stream/node_modules/duplexify/index.js:191:8)
    at /home/kyle/.local/lib/node_modules/dat/node_modules/transport-stream/node_modules/duplexify/index.js:174:10
    at process._tickCallback (node.js:355:11)

For good measure:

kyle@dev:~$ uname -srvpio
Linux 3.2.0-4-amd64 #1 SMP Debian 3.2.65-1+deb7u1 unknown unknown GNU/Linux
kyle@dev:~$ cat /etc/debian_version
8.1
kemitchell commented 9 years ago

Doesn't look like a runtime thing:

https://travis-ci.org/kemitchell/dat-clone-npm.dathub.org/builds/81871051

That's for dat clone --verbose http://npm.dathub.com dat-npm with dat@7.1.1.

mafintosh commented 9 years ago

This was caused by a bug when using debug logging and serving over http. Should be fixed in 7.1.2 (I've updated dathub as well). Could you verify that it works now?

kemitchell commented 9 years ago

Thank you so much!

$ node --version
v4.1.1
$ dat --version
7.1.2
$ dat clone http://npm.dathub.org dat-npm
Cloning http://npm.dathub.org into dat-npm...
Progress:  [+168307, -0] and 1 file(s).
Clone from remote to dat-npm has completed.
jeromew commented 8 years ago

I don't know if I should open a new issue or if you want to re-open this one. I have the same problem with

$ dat --version
7.1.2

for
$ node --version
v0.12.7

and
$ node --version
v4.1.1

the cloning starts and progress is shown until I get

Cloning http://npm.dathub.org into dat-npm...
Progress:  [+3147, -0] and 1 file(s).
premature close

sometimes it gets to [+9494, -0] before the premature close.

is there something I can do to have more info on whether the issue is client side or server side ?

jeromew commented 8 years ago

For what it's worth, I tried

DEBUG=* dat clone --verbose http://npm.dathub.org dat-npm

and the process also stops with "premature close". The last debugs shown are

  transport-stream in +0ms { length: 4084,
  data: ' of little plugins, editor component is provided by\\n[codemirror plugin](https://github.com/Gozala/codemirror-ambiance-plugin),\\n(BTW there is no reason not to h           ave [ace][] plugin that could be installed\\nto replace it), that command line interface is provided by\\n[gcli plugin](https://github.com/Gozala/gcli-ambiance-plugin) and            even code\\nthat reads and writes files to disk is a\\n[plugin](https://github.com/Gozala/fs-ambiance-plugin). And plugins are just\\nan [npm][] packages, wanna create one            ? You got all the goods from npm wanna\\nshare just publish to npm!\\n\\n\\n## Screenshots\\n\\n![](http://f.cl.ly/items/291W3B0P2W0Q2a0j1h2U/Screen%20Shot%202012-10-18%20a           t%2022.46.58%20.png)\\n![](http://f.cl.ly/items/3P471q2T0k3E2Y1d061b/Screen%20Shot%202012-10-18%20at%2022.47.23%20.png)\\n\\n## Try\\n\\nThere is several ways you can try t           his thing out, but be aware it\'s very\\nunstable and far from being finished (any help is welcome BTW).\\n\\n#### Download\\n\\nIf you\'re a mac user you can just [downloa           d][downloads] `.dmg` file.\\n\\n#### Use node-webkit\\n\\nAlternatively you can use [node-webkit][], to do that you\'ll need to first\\n[download](https://github.com/rogerw           ang/node-webkit#downloads) a bulid.\\n\\nThen download ambiance:\\n```sh\\ngit clone https://github.com/Gozala/ambiance.git && cd ambiance\\nnpm install\\n```\\n\\nFinally            ether [run with node-webkit](https://github.com/rogerwang/node-webkit/wiki/How-to-run-apps)\\n\\n```sh\\nnw .\\n```\\n\\nOr [make a bulid](https://github.com/rogerwang/node           -webkit/wiki/How-to-package-and-distribute-your-apps)\\nof your own.\\n\\n## Credits\\n\\nAll the credits got to an amazing open source projects without which this would\\n           not be possible:\\n\\n[nodejs][], [node-webkit][], [npm][], [codemirror][], [gcli][]\\n\\nAnd to [Svengraph](http://svengraph.deviantart.com/) who made beautiful icon\\nwit           h [cc](http://creativecommons.org/licenses/by-nd/3.0/) license that editor\\nis using.\\n\\n\\n[npm]:https://npmjs.org/\\n[nodejs]:http://nodejs.org/\\n[vim]:http://www.vim           .org/\\n[codemirror]:http://codemirror.net/\\n[try wisp]:http://jeditoolkit.com/wisp/\\n[wisp]:https://github.com/Gozala/wisp\\n[prose]:http://jeditoolkit.com/prose/\\n[Lit           erate programming]:http://en.wikipedia.org/wiki/Literate_programming\\n[shebang]:http://en.wikipedia.org/wiki/Shebang_%28Unix%29 \\n[prose compiler]:https://github.com/goza           la/prose\\n[jsfiddle]:http://jsfiddle.net/\\n[jsbin]:http://jsbin.com/\\n[node-webkit]:https://github.com/rogerwang/node-webkit\\n[GCLI]:https://github.com/mozilla/gcli\\n[           ace]:http://ace.ajax.org/\\n[downloads]:https://github.com/Gozala/ambiance/downloads\\n","maintainers":[{"name":"gozala","email":"rfobic@gmail.com"}],"time":{"modified":"20           12-10-19T07:07:13.313Z","created":"2012-10-19T07:07:11.731Z","0.0.1":"2012-10-19T07:07:13.313Z"},"author":{"name":"Irakli Gozalishvili","email":"rfobic@gmail.com","url":"ht           tp://jeditoolkit.com"},"repository":{"type":"git","url":"https://github.com/Gozala/ambiance.git","web":"https://github.com/Gozala/ambiance"},"key":"ambiance","couchSeq":847           9}\u0011�\u0013\b\u0001\u0012\u0007modules\u001a\u0018ambiance-command-manager"�\u0012{"_rev":"3-d581b8c50c19adc7d7b1b3c78dff3dc9","name":"ambiance-command-manager","descri           ption":"Command manager for ambiance editor","dist-tags":{"latest":"0.0.1"},"versions":{"0.0.1":{"name":"ambiance-command-manager","id":"ambiance-command-manager","version"           :"0.0.1","description":"Command manager for ambiance editor","keywords":["manager","ambiance","commands","plugin"],"author":{"name":"Irakli Gozalishvili","email":"rfobic@gm           ail.com","url":"http://jeditoolkit.com"},"homepage":"https://github.com/Gozala/ambiance-command-manager","repository":{"type":"git","url":"https://github.com/Gozala/ambianc           e-command-manager.git","web":"https://github.com/Gozala/ambiance-command-manager"},"bugs":{"url":"http://github.com/Gozala/ambiance-command-manager/issues/"},"devDependenci           es":{"test":">=0.4.4","repl-utils":">=1.0.0"},"main":"./command-manager.js","scripts":{"test":"node test/test-all.js","repl":"node node_modules/repl-utils"},"licenses":[{"t           ype":"MIT","url":"https://github.com/Gozala/ambiance-command-manager/License.md"}]' }

  transport-stream in +1ms { length: 3384,
  data: ',"readme":"# ambiance-command-manager\\n\\n[![Build Status](https://secure.travis-ci.org/Gozala/ambiance-command-manager.png)](http://travis-ci.org/Gozala/ambiance           -command-manager)\\n\\nCommand manager for ambiance editor\\n\\n## Install\\n\\n    npm install ambiance-command-manager\\n","_id":"ambiance-command-manager@0.0.1","dist":{           "shasum":"58d5dd91038e3995472d61deb831635d64b87718","tarball":"http://registry.npmjs.org/ambiance-command-manager/-/ambiance-command-manager-0.0.1.tgz"},"_npmVersion":"1.1.           63","_npmUser":{"name":"gozala","email":"rfobic@gmail.com"},"maintainers":[{"name":"gozala","email":"rfobic@gmail.com"}]}},"readme":"# ambiance-command-manager\\n\\n[![Buil           d Status](https://secure.travis-ci.org/Gozala/ambiance-command-manager.png)](http://travis-ci.org/Gozala/ambiance-command-manager)\\n\\nCommand manager for ambiance editor\           \n\\n## Install\\n\\n    npm install ambiance-command-manager\\n","maintainers":[{"name":"gozala","email":"rfobic@gmail.com"}],"time":{"modified":"2012-10-19T04:32:23.333Z"           ,"created":"2012-10-19T04:32:21.965Z","0.0.1":"2012-10-19T04:32:23.333Z"},"author":{"name":"Irakli Gozalishvili","email":"rfobic@gmail.com","url":"http://jeditoolkit.com"},           "repository":{"type":"git","url":"https://github.com/Gozala/ambiance-command-manager.git","web":"https://github.com/Gozala/ambiance-command-manager"},"key":"ambiance-comman           d-manager","couchSeq":8480}\u0011�\u000f\b\u0001\u0012\u0007modules\u001a\u0017ambiance-plugin-manager"�\u000f{"_rev":"3-8996b0c2546fee0eb4ad4fa75389a5b1","name":"ambiance-           plugin-manager","description":"plugin manager for ambiance editor","dist-tags":{"latest":"0.0.1"},"versions":{"0.0.1":{"name":"ambiance-plugin-manager","id":"ambiance-plugi           n-manager","version":"0.0.1","description":"plugin manager for ambiance editor","keywords":["plugins","manager","plugin","ambiance"],"author":{"name":"Irakli Gozalishvili",           "email":"rfobic@gmail.com","url":"http://jeditoolkit.com"},"homepage":"https://github.com/Gozala/ambiance-plugin-manager","repository":{"type":"git","url":"https://github.c           om/Gozala/ambiance-plugin-manager.git","web":"https://github.com/Gozala/ambiance-plugin-manager"},"bugs":{"url":"http://github.com/Gozala/ambiance-plugin-manager/issues/"},           "main":"./plugin-manager.js","dependencies":{"npm":"1.1.63","micro-promise":"*"},"devDependencies":{"test":">=0.4.4"},"scripts":{"test":"node tests/test-all.js"},"licenses"           :[{"type":"MIT","url":"http://jeditoolkit.com/LICENSE"}],"readme":"# ambiance-plugin-manager\\n\\nPlugin manager component of ambiance editor\\n","_id":"ambiance-plugin-man           ager@0.0.1","dist":{"shasum":"397c742397a4846617070afa4bf6042a59f67908","tarball":"http://registry.npmjs.org/ambiance-plugin-manager/-/ambiance-plugin-manager-0.0.1.tgz"},"           _npmVersion":"1.1.63","_npmUser":{"name":"gozala","email":"rfobic@gmail.com"},"maintainers":[{"name":"gozala","email":"rfobic@gmail.com"}]}},"readme":"# ambiance-plugin-man           ager\\n\\nPlugin manager component of ambiance editor\\n","maintainers":[{"name":"gozala","email":"rfobic@gmail.com"}],"time":{"modified":"2012-10-19T04:35:21.990Z","create           d":"2012-10-19T04:35:20.664Z","0.0.1":"2012-10-19T04:35:21.990Z"},"author":{"name":"Irakli Gozalishvili","email":"rfobic@gmail.com","url":"http://jeditoolkit.com"},"reposit           ory":{"type":"git","url":"https://github.com/Gozala/ambiance-plugin-manager.git","web":"https://github.com/Gozala/ambiance-plugin-manager"},"key":"ambiance-plugin-manager",
Cloning http://npm.dathub.org into dat-npm...
Progress:  [+3016, -0] and 1 file(s).
premature close
jeromew commented 8 years ago

For information, I could get the cloning go through by modifying

var queue = parallel(16, function (node, cb) {

to

var queue = parallel(1, function (node, cb) {

in dat-core/lib/replicate.js

I am new to dat and I am not sure I understand how the parallelism work, but something may be wrong in the algorithm depending on network conditions.

In my case, it also works for parallel(4) but not parallel(8) or parallel(16).