search

componentjs / downloader.js

download components from a remote
11 stars 4 forks source link

bitbucket download failure #4

Closed jonathanong closed 10 years ago

jonathanong commented 10 years ago

https://github.com/component/downloader.js/commit/4f2a98108c8d1c1bb620220083a54384e788a5e5

  remotes:bitbucket GET "https://bitbucket.org/jongleberry/component-test-glob/raw/0.0.0/component.json" +0ms
  cogent redirecting from https://bitbucket.org/jongleberry/component-test-glob/raw/0.0.0/component.json to https://bitbucket.org/account/signin/?next=/jongleberry/component-test-glob/raw/0.0.0/component.json +841ms

no idea waht's going on. i'm authenticated and remotes works, but it keeps redirecting me to sign on. the URL works in incognito mode.

@netpoetica take a look?

netpoetica commented 10 years ago

It should be hitting the api.bitbucket.org or /api URL, taking a look now

jonathanong commented 10 years ago

i'm goign to add you to the c8 team. you've been helpful, and it'll be easier to use local branches :D

netpoetica commented 10 years ago

Awesome, thanks a ton. Quick question - any solution to tell the test command to ignore Github? I think I have hit my limits which is causing some strange test failures that are not failing too gracefully

jonathanong commented 10 years ago

comment the tests out as well as remoiving github from https://github.com/component/downloader.js/blob/master/lib/downloader.js#L35

maybe we should just separate the downloading tests per remote. might be easier to test that way.

jonathanong commented 10 years ago

how did you hit 5600 requests per hour!?

netpoetica commented 10 years ago

I highly doubt I did, although, I did run the test command a lot yesterday/this morning working on that Bitbucket remote :-p I will remove it in downlaoder - just wasn't sure if maybe there was an "--ignore-remote" type of thing

When I run test on downloader I get this series of errors:

> component-downloader@1.0.2 test /Users/KeithOMB/Desktop/projects/downloader.js
> NODE=gnode make test

  mocha:suite bail undefined +0ms
  mocha:suite timeout 30000 +5ms
  mocha:suite bail true +1ms
  mocha:suite timeout 30000 +513ms
  mocha:suite slow 75 +0ms
  mocha:suite bail true +0ms
  component-downloader no remote specified, defaulting to regular remote +0ms
  mocha:runnable timeout 30000 +0ms
  mocha:runnable timeout 75 +1ms
  mocha:suite timeout 30000 +2ms
  mocha:suite slow 75 +0ms
  mocha:suite bail true +0ms
  mocha:runnable timeout 30000 +0ms
  mocha:runnable timeout 75 +0ms
  mocha:runnable timeout 30000 +0ms
  mocha:runnable timeout 75 +0ms
  mocha:runnable timeout 30000 +1ms
  mocha:runnable timeout 75 +0ms
  mocha:runnable timeout 30000 +0ms
  mocha:runnable timeout 75 +0ms
  mocha:runnable timeout 30000 +0ms
  mocha:runnable timeout 75 +0ms
  mocha:suite timeout 30000 +1ms
  mocha:suite slow 75 +0ms
  mocha:suite bail true +0ms
  component-downloader no remote specified, defaulting to regular remote +2ms
  mocha:runnable timeout 30000 +0ms
  mocha:runnable timeout 75 +0ms
  mocha:suite timeout 30000 +0ms
  mocha:suite slow 75 +1ms
  mocha:suite bail true +0ms
  mocha:runnable timeout 30000 +1ms
  mocha:runnable timeout 75 +0ms
  mocha:runnable timeout 30000 +0ms
  mocha:runnable timeout 75 +0ms
  mocha:runnable timeout 30000 +0ms
  mocha:runnable timeout 75 +0ms
  mocha:runnable timeout 30000 +0ms
  mocha:runnable timeout 75 +0ms
  mocha:runnable timeout 30000 +0ms
  mocha:runnable timeout 75 +0ms
  mocha:runner grep /.*/ +0ms
  mocha:runner globals ["ArrayBuffer","Int8Array","Uint8Array","Uint8ClampedArray","Int16Array","Uint16Array","Int32Array","Uint32Array","Float32Array","Float64Array","DataView","DTRACE_NET_SERVER_CONNECTION","DTRACE_NET_STREAM_END","DTRACE_NET_SOCKET_READ","DTRACE_NET_SOCKET_WRITE","DTRACE_HTTP_SERVER_REQUEST","DTRACE_HTTP_SERVER_RESPONSE","DTRACE_HTTP_CLIENT_REQUEST","DTRACE_HTTP_CLIENT_RESPONSE","global","process","GLOBAL","root","Buffer","setTimeout","setInterval","clearTimeout","clearInterval","setImmediate","clearImmediate","console","wrapGenerator","Reporter","before","after","beforeEach","afterEach","context","describe","xcontext","xdescribe","specify","it","xspecify","xit","XMLHttpRequest","Date"] +1ms
  mocha:runner globals [] +0ms
  mocha:runner start +0ms

  mocha:runner run suite  +0ms

  mocha:runner run suite When downloading via fields +2ms
  When downloading via fields
  mocha:runner run suite When downloading via fields .download(repo, ref) +4ms
    .download(repo, ref)
  remotes:github GET "https://raw.githubusercontent.com/component/emitter/1.1.2/component.json" +0ms
  component-downloader resolved component/emitter@1.1.2 to github +671ms
  component-downloader downloading "https://raw.githubusercontent.com/component/emitter/1.1.2/index.js" to "/Users/KeithOMB/Desktop/projects/downloader.js/components/component/emitter/1.1.2/index.js" +18ms
  component-downloader installed component/emitter@1.1.2 in 972ms +294ms
      ✓ should download component/emitter@1.1.2 (975ms)
  remotes:github GET "https://raw.githubusercontent.com/component/domify/master/component.json" +973ms
  component-downloader resolved component/domify@master to github +298ms
  component-downloader downloading "https://raw.githubusercontent.com/component/domify/master/index.js" to "/Users/KeithOMB/Desktop/projects/downloader.js/components/component/domify/master/index.js" +2ms
  component-downloader installed component/domify@master in 594ms +296ms
      ✓ should download component/domify@master (596ms)
  remotes:github GET "https://raw.githubusercontent.com/component-test/glob/0.0.1/component.json" +595ms
  component-downloader resolved component-test/glob@0.0.1 to github +282ms
  remotes:github GET "https://api.github.com/repos/component-test/glob/git/trees/0.0.1?recursive=1" +282ms
      1) should unglob component-test/glob@0.0.1

  mocha:runner run suite When downloading archives +2s
  When downloading archives
  mocha:runner run suite When downloading archives .download(repo, ref) +0ms
    .download(repo, ref)

  mocha:runner finished running +0ms

  2 passing (2s)
  1 failing

  1) When downloading via fields .download(repo, ref) should unglob component-test/glob@0.0.1:
     Error: failed to get component-test/glob's git tree
      at GitHub.<anonymous> (/Users/KeithOMB/Desktop/projects/downloader.js/node_modules/remotes/build/remotes/github.js:614:13)
      at Generator.invoke (evalmachine.<anonymous>:141:31)
      at Generator.invoke (evalmachine.<anonymous>:79:50)
      at Generator.invoke (evalmachine.<anonymous>:79:50)
      at Generator.invoke (evalmachine.<anonymous>:79:50)
      at Generator.invoke (evalmachine.<anonymous>:79:50)
      at Context.next (/Users/KeithOMB/Desktop/projects/downloader.js/node_modules/co/index.js:77:21)
      at /Users/KeithOMB/Desktop/projects/downloader.js/node_modules/co/index.js:96:18
      at IncomingMessage.onEnd (/Users/KeithOMB/Desktop/projects/downloader.js/node_modules/remotes/node_modules/cogent/node_modules/raw-body/index.js:109:7)
      at IncomingMessage.g (events.js:175:14)
      at IncomingMessage.EventEmitter.emit (events.js:117:20)
      at _stream_readable.js:920:16
      at process._tickCallback (node.js:415:13)

which looks very similar to the error I just posted in regards to failing Travis build

netpoetica commented 10 years ago

From what I can tell from the error there, it looks like it's a problem with trying to bypass the API in that url. The login stuff shouldn't be a problem if we are hitting this endpoint:

https://api.bitbucket.org/1.0/repositories/jongleberry/component-test-glob/raw/0.0.0/component.json

Will keep looking into it, just giving you a heads up about likely cause. Additionally, since it's a public repo you shouldn't have to be authenticated at all

netpoetica commented 10 years ago

@jonathanong If you can, check your BITBUCKET_USERNAME and BITBUCKET_PASSWORD in exports. When mine is correct, everything works no problem, but as soon as I set my username to something invalid, I get the same redirect error as you

We may want to consider removing that non-api backup URL from the array in bitbucket.js in order to never get this redirect issue. You will still need valid bitbucket credentials in your exports, but the API endpoints should work 100% of the time - I only put that backup URL in there just in case, but I think it's not a good idea

jonathanong commented 10 years ago

yeah my bitbucket username and password are correct. not sure what's going on ~_~

jonathanong commented 10 years ago

doesn't work when i unset them as well. ugh.

travis is passing though: https://travis-ci.org/component/downloader.js/jobs/22407089

are you comfortable enough to just publish this in its current form?

netpoetica commented 10 years ago

Pretty strange -with invalid creds I get the same error as you, but if they are correct, i do not get the error when isolating only the bitbucket test and using only the bitbucket remote.

I think it may be worth removing this line, because the API will not give you the redirect. This way we can at least avoid the strange redirect which might be difficult to handle/properly detect and report about

Other than that, it seems to work with valid creds and I don't think it's dangerous to publish

jonathanong commented 10 years ago

okay publishing. going to leave this open.

netpoetica commented 10 years ago

Strangely, if I use "unset", it still works but if I use

export BITBUCKET_USERNAME=fakename
npm test

I get your error

netpoetica commented 10 years ago

Just out of curiosity, what OS + node version are you running?

jonathanong commented 10 years ago

0.11.12 and OS X 10.9.2

jonathanong commented 10 years ago

i added you as owner to component-downloader and remotes on npm, so if you find any bugs, feel free to push and release

netpoetica commented 10 years ago

Sounds good, I will keep my ears open too for any BitBucket related issues, and I'll keep you posted about any other remotes I can start popping in. Feel free to @ me any time if I don't catch a related issue

netpoetica commented 10 years ago

@jonathanong Can't figure out why for the life of me and I'm not sure if it's only happening to me due to some strange cacheing mechanism... but the issue is that for some reason the bitbucket remote, when it makes a request through Cogent, is setting the basic http auth header to

Authentication: Basic btoa(BITBUCKET_USERNAME:GITHUB_PASSWORD);

So for some reason, if you have a Github password in your env, its will use that. I have verified this 100% by looking at the headers and decoding base64 on the basic auth.

The error that I was getting before was the result of me not having any GITHUB creds in my env - which is why it worked for me, but didn't work for you.

If you

unset GITHUB_USERNAME
unset GITHUB_PASSWORD

and then try only the bitbucket remote & bitbucket test, you will probably get a pass. However, if you inspect the headers of the request to bitbucket with the Github creds in your env, you will see that somehow that github password is getting shoved where Bitbucket password should be.

My guess is there is probably a global somewhere, or potentially a piece of code somewhere that is hard-coded to github password? I am running grep -r 'GITHUB_PASSWORD' where it makes sense to see if something is sucking it up where it shouldn't, but haven't had any luck yet. Any ideas?

Steps to reproduce:

  1. Comment out all tests except for the bitbucket test (because when you remove github creds, tests will fail)
  2. unset both github creds but leave bitbucket creds
  3. run DEBUG=* npm test succesfully
  4. source your rc file to add github creds back
  5. run DEBUG=* npm test unsuccesfully. Check headers, look for Authentication: Basic base64encoded_uname:pass and decode it at http://www.base64decode.org/ and see that it is using your Github password
jonathanong commented 10 years ago

ah ok. i think i need to do some cogent fixing.

netpoetica commented 10 years ago

I sure hope so because I can't imagine what else it could be :) I did a couple of things including fixing a mistyped err vs res in remotes. I am curious - would you prefer that I push direct to master or create a branch and then make a pull request? I am hesitant to push to master but I also don't want to pollute branches. What is your preferred strategy on remotes.js?

jonathanong commented 10 years ago

depends on what the fix is. if i have an issue for it, it's obvious, or it's copy fixes, then feel free to push to master. if you're not sure, make a PR

jonathanong commented 10 years ago

got it. here you go! https://github.com/component/remotes.js/commit/85ab4782dc1e55feae2c3586b8b992414c3a5c03