component-update: fails to fetch tags from private BitBucket repositories

[stephenmathieson]

running component-update on a project with BitBucket deps fails:

error : failed to get some-bitbucket-user/some-bitbucket-component's tags. please check that this repository still exists!

[netpoetica]

@stephenmathieson is this a private component? If so, have you configured your globals?

Can you run with DEBUG=* and provide the output? Leaving a note for myself to verify that this isn't a problem with BitBucket's lack of support for Git trees, could be related to https://github.com/component/remotes.js/pull/14#discussion-diff-11324698R110

[stephenmathieson]

yup it's private. I've got my credentials setup, but still seeing this. I'll look more in depth in a bit

On Jul 3, 2014, at 2:26 PM, Keith Rosenberg notifications@github.com wrote:

@stephenmathieson is this a private component? If so, have you configured your globals?

Can you run with DEBUG=* and provide the output?

— Reply to this email directly or view it on GitHub.

[stephenmathieson]

here's output with all of the debug stuff: https://gist.github.com/stephenmathieson/b4c065fb3e32fe491f2d. no mention of bitbucket at all :/

[netpoetica]

@stephenmathieson yeah, it's pretty difficult for me to tell if this has anything to do with the bitbucket remote. It's worth noting that if the BitBucket repo you are trying to use doesn't conform to semver and use release tags, it's possible this could be causing the issue. It is very strange that the error message from debug != the error message from your original post, I'm not too sure what's going on with that.

Out of curiousity, can you confirm that the BitBucket repo (private) you are looking for is not being treated as local in your Gist? i.e. "some-bitbucket-user/some-bitbucket-component" whatever that was, is it in the Gist at all?

Additionally, try navigating to this URL directly and seeing if you get any JSON (in browser, logged in as a user who has access to that repo):

https://api.bitbucket.org/1.0/repositories/some-bitbucket-user/some-bitbucket-component/tags
// replace some-bitbucket-user/some-bitbucket-component with details

As an example, this is what it should look like in a public repo

https://api.bitbucket.org/1.0/repositories/fenics-project/instant/tags

The error that you are getting in your original post is only possible if you get a 404 from https://api.bitbucket.org/1.0/repositories/some-bitbucket-user/some-bitbucket-component/tags

Relevant code here for reference: https://github.com/component/remotes.js/blob/master/lib/remotes/bitbucket.js#L46

[stephenmathieson]

haha, yeah, some-bitbucket-user isn't the actual username :p

the actual error is displayed on lines 724-736

the API will 403 without credentials:

$ curl -v https://api.bitbucket.org/1.0/repositories/dmusser/project-x-highlighter/tags
[ . . . ]
HTTP/1.1 403 FORBIDDEN
[ . . . ]

but with credentials, we'll get back:

HTTP/1.1 200 OK
Content-Language:en
ETag:"a6bb68d2f5e363089010c410d9b9551e"
X-Render-Time:0.471938848495
Content-Length:1039
X-Served-By:app22
X-Version:69e6e47e5db0
X-Request-Count:284
Server:nginx/1.5.10
X-Content-Type-Options:nosniff
X-Frame-Options:SAMEORIGIN
Strict-Transport-Security:max-age=31536000
Date:Mon, 14 Jul 2014 17:52:41 GMT
Vary:Authorization, Accept-Language, Cookie
X-Static-Version:2de54e0b41c0
Content-Type:application/json; charset=utf-8

{
  "0.1.0": {
    "node": "9fbb859b6821",
    "files": [
      {
        "type": "modified",
        "file": "History.md"
      },
      {
        "type": "modified",
        "file": "component.json"
      },
      {
        "type": "modified",
        "file": "package.json"
      }
    ],
    "raw_author": "Stephen Mathieson <smath23@gmail.com>",
    "utctimestamp": "2014-06-02 14:01:32+00:00",
    "author": "stephenmathieson",
    "timestamp": "2014-06-02 16:01:32",
    "raw_node": "9fbb859b68217620b93c194021291595c134d49e",
    "parents": [
      "c80e7a26b472"
    ],
    "branch": "master",
    "message": "Release 0.1.0\n",
    "revision": null,
    "size": -1
  },
  "0.0.1": {
    "node": "bcb0050fc43b",
    "files": [
      {
        "type": "added",
        "file": "History.md"
      },
      {
        "type": "modified",
        "file": "component.json"
      },
      {
        "type": "modified",
        "file": "package.json"
      }
    ],
    "branches": [],
    "raw_author": "Stephen Mathieson <smath23@gmail.com>",
    "utctimestamp": "2014-05-29 14:21:09+00:00",
    "author": "stephenmathieson",
    "timestamp": "2014-05-29 16:21:09",
    "raw_node": "bcb0050fc43b4a9a395585c44465f1cc4a4acee5",
    "parents": [
      "c2bdd4597059"
    ],
    "branch": null,
    "message": "Release 0.0.1\n",
    "revision": null,
    "size": -1
  }
}

EDIT: typo

[stephenmathieson]

btw, it looks like we're never hitting the API at all: https://gist.github.com/stephenmathieson/b4c065fb3e32fe491f2d#file-gistfile1-txt-L471

[netpoetica]

So that line 471 should be using bitbucket but instead it's using Github?

[stephenmathieson]

yup

[netpoetica]

Interesting. The way it works right now is that it is cascading, that is, component checks locally first, then reaches out to github, and if it doesn't find it, it should then go to the next remote (only BitBucket). This is because we cannot explicitly tell component which repo belongs to which remote via component.json (the spec doesn't support this but it has been discussed, I can't find the thread though). See: https://github.com/component/remotes.js/blob/master/lib/remotes.js#L98

It looks like for some reason, in this case, it's throwing an err for Github and then never trying for BitBucket. I will definitely spend some more time with this and see what's up

[timaschew]

maybe change the order in the component.json:

"remotes": ["bitbucket", "github"] 

But I think then you can't use private gihub repos.

[netpoetica]

At some point ong and I discussed having the user specify remotes in their dependencies. It could also be a list of remotes in the component.json, or something similar.

"dependencies": {
  "bitbucket:netpoetica/my-component": "~1.2"
}

I like your previous example where remote order is specified via component.json though, that's a good idea. I would ideally combine the two ideas, so that people can still prefix their deps to optimize speed and accuracy of downloads, and avoid issues of failed lookups.

Because of how component/remotes.js is constructed, it's fairly simple to add any remote that has a decent API. I was working on CodePlex for a while but their API is godawful. Post 1.0 I would like to try to add all known remotes :-) that would keep Component relevant as well, "the package manager that works across all Git hosting platforms", etc