403 resolving private dependency

0x6e6562 commented 5 years ago

I've successfully published a private package to hex.pm that I am able to resolve using mix:

$ mix hex.search mq_series
Organization  Package    Description  Version  URL
relops        mq_series  MQ series    1.3.21   https://hex.pm/packages/relops/mq_series

However, I'm getting a 403 when I try to include this dependency in a rebar3 project:

$ DEBUG=1 rebar3 get-deps
===> Load global config file /Users/0x6e6562/.config/rebar3/rebar.config
===> Expanded command sequence to be run: [{default,app_discovery},
                                                  {default,install_deps},
                                                  {default,lock},
                                                  {default,'get-deps'}]
===> Provider: {default,app_discovery}
===> Provider: {default,install_deps}
===> Verifying dependencies...
===> Hex get_package request failed: {ok,
                                             {403,
                                              #{<<"accept-ranges">> =>
                                                 <<"bytes">>,
                                                <<"age">> => <<"0">>,
                                                <<"connection">> =>
                                                 <<"keep-alive">>,
                                                <<"content-length">> =>
                                                 <<"0">>,
                                                <<"date">> =>
                                                 <<"Sun, 17 Feb 2019 21:01:22 GMT">>,
                                                <<"retry-after">> => <<"0">>,
                                                <<"server">> => <<"Varnish">>,
                                                <<"via">> => <<"1.1 varnish">>,
                                                <<"x-cache">> =>
                                                 <<"MISS, MISS">>,
                                                <<"x-cache-hits">> =>
                                                 <<"0, 0">>,
                                                <<"x-served-by">> =>
                                                 <<"cache-iad2147-IAD, cache-bne7722-BNE">>,
                                                <<"x-timer">> =>
                                                 <<"S1550437282.231335,VS0,VE318">>},
                                              <<>>}}
===> Failed to update package from repo hexpm:relops
===> Package not found in any repo: mq_series 1.3.21

My ~/.config/rebar3/rebar.config looks like this:

{plugins, [rebar3_hex]}.
{hex, [{repos, [
  #{name => <<"hexpm:relops">>}
]}]}.

I can resolve this with a mix project if I define the dependency like

{:mq_series, "~> 1.3.21", organization: "relops"}

In the rebar.config it is defined as

{mq_series, "1.3.21"}

$ rebar3 --version
rebar 3.9.0 on Erlang/OTP 21 Erts 10.2.3

cat ~/.cache/rebar3/plugins/rebar3_hex/ebin/rebar3_hex.app
{application,rebar3_hex,
             [{registered,[]},
              {description,"Hex.pm plugin for rebar3"},
              {vsn,"6.4.0"},
              {applications,[kernel,stdlib]},
              {maintainers,["Tristan Sloughter"]},
              {licenses,["MIT"]},
              {links,[{"Github","https://github.com/tsloughter/rebar3_hex"}]},
              {modules,[rebar3_hex,rebar3_hex_cut,rebar3_hex_docs,
                        rebar3_hex_key,rebar3_hex_owner,rebar3_hex_publish,
                        rebar3_hex_repo,rebar3_hex_retire,rebar3_hex_revert,
                        rebar3_hex_search,rebar3_hex_user,rebar3_hex_utils]}]}.

Is there anything obvious I'm doing wrong? Is a 403 error linked to any missing permissions?

tsloughter commented 5 years ago

Yea, this is a known issue that @starbelly was investigating.

Did you get anywhere with that, @starbelly ? I can probably take a look tomorrow morning.

0x6e6562 commented 5 years ago

@tsloughter Initially I asked via the hex.pm support mail, but it seems to work fine with mix, so I'm not sure to what extent they can help. Also, I was able to publish the package to hex.pm using rebar3_hex. It is just that rebar3_hex doesn't seem to be able to access it for resolution.

starbelly commented 5 years ago

@tsloughter Yes, there is a PR open for this https://github.com/erlang/rebar3/pull/2020 in rebar3

tsloughter commented 5 years ago

Ok great! Assuming this fixes it for @0x6e6562 then I think I see what what a better fix would be, which works for also non-hexpm.

0x6e6562 commented 5 years ago

Is there an easy way to test the PR? Can you run rebar by doing a git checkout on the pull request commit?

starbelly commented 5 years ago

@0x6e6562 Yes

Clone my repo
Checkout the branch in the PR
./bootstrap
rebar3 local install (or just add pwd to your path)

0x6e6562 commented 5 years ago

Thanks for the heads up.

Unfortunately the behavior is the same with this local version of rebar3:

DEBUG=1 rebar3 --version
===> Load global config file /Users/0x6e6562/.config/rebar3/rebar.config
===> Expanded command sequence to be run: [{default,version}]
===> Provider: {default,version}
rebar 3.5.3+build.4305.ref2e31b833 on Erlang/OTP 21 Erts 10.2.3

This was built against 2e31b83348be01624ef7a32b717e33f5b867edd0.

Previously I was using released 3.9.0 version, but this branch seems to base off 3.5.3. Would this have any effect?

starbelly commented 5 years ago

@0x6e6562 The other possible case for a 403 is config related. We're going to be doing some patches to rebar3 are coming up in the following days to address this... can you show me your ~/.config/rebar3/hex.config with the keys redacted?

0x6e6562 commented 5 years ago

#{<<"hexpm">> =>
      #{read_key => <<"...b2a7">>,
        repos_key => <<"...375b">>,
        username => <<"...">>,
        write_key =>
            {<<...,236>>,
             {<<...,12,
                ...,58>>,
              <<...,2>>}}},
  <<"hexpm:relops">> => #{auth_key => <<"...94cb">>},
  <<"relops">> => #{auth_key => <<"...a55c">>}}.

0x6e6562 commented 5 years ago

Given that the mix is working locally, is there a mix config for hex.pm that we can use for comparison?

starbelly commented 5 years ago

Yeah, so all looks fine except the hexpm:relops entry, you need repo_key in there. This should be a read key for the org. You also don't need the relops repo, just the hexpm:relops repo. This is currently not documented anywhere I believe, but should be soon and as stated previously we'll be getting in some other things in shortly to validate and/or help with this. I saw your comment about the branch you were on based on 3.5.3 ... are you sure you are on the right branch? fix-repo-url-for-custom-repos. You will also at this point need to make a _checkouts dir and git clone my hex_core repo too 😄 So, let me revise steps.

git clone https://github.com/starbelly/rebar3.git
cd rebar3
git checkout fix-repo-url-for-custom-repos
mkdir _checkouts
cd _checkouts
git clone https://github.com/starbelly/hex_core
cd hex_core
git checkout fixup-tarball_url
cd ../../
./bootstrap
rebar3 local install (or just add pwd to the beginning of your $PATH)

Note, if successfully you will get a error/warning message still but the compile finishes, just ignore it, it's a lie ;) Something else that needs to be fixed up.

Eager to see if these fixes work for you.

0x6e6562 commented 5 years ago

OK, so the plot thickens slightly: by manually tweaking the hex.config to look like this:

#{<<"hexpm">> =>
      #{read_key => <<"...b2a7">>,
        repos_key => <<"...375b">>,
        username => <<"...">>,
        write_key =>
            {<<...,236>>,
             {<<...,12,
                ...,58>>,
              <<...,2>>}}},
  <<"hexpm:relops">> => #{read_key => <<"...5fba">>,
                          auth_key => <<"...94cb">>}}.

The resolution fails in a slightly different way:

$ DEBUG=1 rebar3 get-deps
===> Load global config file /Users/0x6e6562/.config/rebar3/rebar.config
===> Expanded command sequence to be run: [{default,app_discovery},
                                                  {default,install_deps},
                                                  {default,lock},
                                                  {default,'get-deps'}]
===> Provider: {default,app_discovery}
===> Provider: {default,install_deps}
===> Verifying dependencies...
===> Fetching mq_series ({pkg,<<"mq_series">>,<<"1.3.21">>})
===> Request for package mq_series-1.3.21 failed: status code 403
===> Failed to fetch and copy dep: mq_series-1.3.21

This also followed the steps you outlined such that

$ DEBUG=1 rebar3 --version
===> Load global config file /Users/0x6e6562/.config/rebar3/rebar.config
===> Expanded command sequence to be run: [{default,version}]
===> Provider: {default,version}
rebar 3.5.3+build.4306.refe0d86ac0 on Erlang/OTP 21 Erts 10.2.3

With starbelly/rebar3 at e0d86ac0449512b3987225a81b120b9cde350ff0 and starbelly/hex_core at d383e7f1d507c732bb767c000d905985ea801949.

Also, judging by the hex.pm dashboard, the read_key I created for the hexpm:relops repo has a non-null Last used value.

So rebar3_hex has ostensibly been able to detect the existence of the package I'm trying to resolve.

0x6e6562 commented 5 years ago

I also get the same behavior as above when I use the 3.9.0 release version of rebar3 - both versions of rebar3 produce the output

Request for package mq_series-1.3.21 failed: status code 403

After having added the read_key to hex.config.

So on face value there appears to be no difference in behavior between the e0d86ac0 and the 3.9.0 release for this issue.

0x6e6562 commented 5 years ago

@starbelly Would it help at all if I were to locally apply the patch from https://github.com/hexpm/hex_core/compare/emj/remove-organization to see if this resolves the 403?

starbelly commented 5 years ago

@0x6e6562 Yes, that will help for sure! I have tested it all on my end using an org I have on hex. Just checkout the branch in _checkouts/hex_core. Also, the config still may need to be adjusted. I found this is working for me:

#{<<"hexpm">> =>
      #{
        read_key => <<"read_key">>,
        repos_key => <<"repos_key">>,
        username => <<"username">>,
        write_key => {<<"this">>, {<<"is produced">>, <<"by rebar3_hex">>}},
  <<"hexpm:myorg">> =>
      #{
        read_key => <<"read_key">>,
        auth_key => <<"auth_key">>,
        repo_key => <<"read_key">>
      }
  }.

0x6e6562 commented 5 years ago

Cool - what's the best development process for working off a branch with the hex plugin?

AFAICT hex is located in .cache/rebar3/lib/hex_core, but that appears to be the binary version. Where would the source go?

starbelly commented 5 years ago

So pretty much the same...

git clone https://github.com/starbelly/rebar3.git
cd rebar3
git checkout fix-repo-url-for-custom-repos
mkdir _checkouts
cd _checkouts
git clone https://github.com/hexpm/hex_core
cd hex_core
git checkout emj/remove-organization
cd ../../
./bootstrap
rebar3 local install (or just add pwd to the beginning of your $PATH)

rebar3 will use compile in and/or use the version of hex_core that you placed in _checkouts. You could rm -rf .cache/rebar3/lib/hex_core for good measure.

0x6e6562 commented 5 years ago

In the stanza <<"hexpm:myorg">> there is a key called repo_key whilst in the <<"hexpm">> stanza there is a key called repos_key (i.e. with and without an s). Are they both correct, respectively?

starbelly commented 5 years ago

Yeah, there's some confusion about that one. rebar3_hex does use repos_key, so I'd leave it be. Gonna circle back to this and suss out why repos_key is used and how it differs from repo_key. That aside, for the test case here rebar3_hex isn't even needed. The affected areas are rebar3 and hex_core.

0x6e6562 commented 5 years ago

w00t!

rebar3 @ 841ddf746abf999ce84f9481aa09ddcf73f8217a with hex_core @ f379fe0a83a186f5f2406116007f865d1dc4aade appears to have fixed the issue.

I was able to rebar3 get-deps and resolve the private dependency from hex.pm.

Judging from the hex.pm dashboard, the key with the scope ORG:relops appears to have a last access timestamp, whereas previously it did not.

Is this patch likely to make the next rebar3 release?

starbelly commented 5 years ago

That's awesome. Yes, hopefully within the next few days. hex_core needs to merge in the change, then we'll vendor hex_core into rebar3 and do a release with this plus some other hex related fixes. And plenty more on the way after that 😄

/cc @ericmj

wojtekmach commented 5 years ago

We've merged https://github.com/hexpm/hex_core/pull/57 (remove organization from config in hex_core). Let us know if you run into any other issues. And thanks everyone for pushing this forward!

0x6e6562 commented 5 years ago

@starbelly Whilst I've managed to successfully base a local rebar3 off the fix-repo-url-for-custom-repos branch until the fix lands as a release, I've noticed that using this patched version rebar3 appears to break the hex publish -r hexpm:organization command in this way:

===> Uncaught error in rebar_core. Run with DEBUG=1 to see stacktrace or consult rebar3.crashdump
===> Uncaught error: undef
===> Stack trace to the error location:
[{hex_tarball,create,
              [#{<<"app">> => <<"mq_series">>,
                 <<"build_tools">> => [<<"rebar3">>],
                 <<"description">> => <<"MQ series">>,
                 <<"files">> =>
                     [<<"src/mq_series.app.src">>,<<"README.md">>,
                      <<"priv/mq_series_drv.so">>,<<"rebar.config">>,
                      <<"rebar.lock">>,<<"src/mq_series.erl">>,
                      <<"src/mq_series_app.erl">>,<<"src/mq_series_sup.erl">>],
                 <<"licenses">> => [<<"MIT">>],
                 <<"links">> => #{},<<"name">> => <<"mq_series">>,
                 <<"requirements">> => #{},<<"version">> => <<"1.3.22">>},
               [{"src/mq_series.app.src",
                 <<"{application,mq_series,\n             [{description,\"MQ series\"},\n              {vsn,\"1.3.22\"},\n              {registered,[]},\n              {mod,{mq_series_app,[]}},\n              {applications,[kernel,stdlib]},\n              {licenses,[\"MIT\"]},\n              {include_files,[\"priv/mq_series_drv.so\"]},\n              {exclude_files,[\"c_src\",\"README.txt\"]},\n              {env,[]}]}.\n">>},
                {"README.md","/mnt/hgfs/Workspace/mq_driver/README.md"},
                {"priv/mq_series_drv.so",
                 "/mnt/hgfs/Workspace/mq_driver/priv/mq_series_drv.so"},
                {"rebar.config","/mnt/hgfs/Workspace/mq_driver/rebar.config"},
                {"rebar.lock","/mnt/hgfs/Workspace/mq_driver/rebar.lock"},
                {"src/mq_series.erl",
                 "/mnt/hgfs/Workspace/mq_driver/src/mq_series.erl"},
                {"src/mq_series_app.erl",
                 "/mnt/hgfs/Workspace/mq_driver/src/mq_series_app.erl"},
                {"src/mq_series_sup.erl",
                 "/mnt/hgfs/Workspace/mq_driver/src/mq_series_sup.erl"}]],
              []},
 {rebar3_hex_publish,create_and_publish,3,
                     [{file,"/root/.cache/rebar3/plugins/rebar3_hex/src/rebar3_hex_publish.erl"},
                      {line,207}]},
 {rebar3_hex_publish,publish,8,
                     [{file,"/root/.cache/rebar3/plugins/rebar3_hex/src/rebar3_hex_publish.erl"},
                      {line,181}]},
 {lists,foldl,3,[{file,"lists.erl"},{line,1263}]},
 {rebar_core,do,2,
             [{file,"/mnt/hgfs/Workspace/mq_driver/rebar3/src/rebar_core.erl"},
              {line,154}]},
 {rebar_prv_do,do_task,5,
               [{file,"/mnt/hgfs/Workspace/mq_driver/rebar3/src/rebar_prv_do.erl"},
                {line,87}]},
 {rebar_core,do,2,
             [{file,"/mnt/hgfs/Workspace/mq_driver/rebar3/src/rebar_core.erl"},
              {line,154}]},
 {rebar3,run_aux,2,
         [{file,"/mnt/hgfs/Workspace/mq_driver/rebar3/src/rebar3.erl"},
          {line,178}]}]

The 3.9.0 release version of rebar running in the same environment does not seem to have this issue (it publishes just fine).

Have I somehow polluted my local rebar3 version or is this potentially some kind of regression?

starbelly commented 5 years ago

It's probably config changes that have happened as a result of fixing things. I will check it out, but now the config should look like:

#{<<"hexpm">> =>
      #{
        read_key => <<"read_key">>,
        repo_key => <<"repos_key">>,
        username => <<"username">>,
        write_key => {<<"this">>, {<<"is produced">>, <<"by rebar3_hex">>}},
  }.

The default in hex_core is to generate a repo access key for all repos you have or will have access too. This was not clear at the start of this. However, you may have a repo with an access key and that will look like:

#{<<"hexpm">> =>
      #{
        read_key => <<"read_key">>,
        repos_key => <<"repos_key">>,
        username => <<"username">>,
        write_key => {<<"this">>, {<<"is produced">>, <<"by rebar3_hex">>}},
  <<"hexpm:myorg">> =>
      #{
        repo_key => <<"read_key">>
      }
  }.

Also, here is a gist I put together for testing all this locally (i.e., not hitting the actual hex.pm):

https://gist.github.com/starbelly/e7c0604513e41d0a2af456aa7abcc8a2#the-script

I will try to reproduce the error you got in a bit.

starbelly commented 5 years ago

@0x6e6562 Yeah, can not replicate. Try fetching the latest changes.

starbelly commented 5 years ago

Also note, the publish command was not honoring the -r switch, so while it didn't fail, it didn't work either. I've added this to https://github.com/tsloughter/rebar3_hex/pull/105

starbelly commented 5 years ago

@0x6e6562 FYI everything that's everything for the moment is in rebar3 master and rebar3_hex master if you'd like to test that ;)

starbelly commented 5 years ago

@0x6e6562 bump... latest rebar3 and rebar3_hex include all these changes... if you can confirm on your end that all this works for you (has for everyone else so far) then I think this can be closed.

starbelly commented 5 years ago

@0x6e6562 bump

mpope9 commented 4 years ago

I seem to be running into the 403 issue on the latest version of rebar3 and latest version of rebar3_hex while trying to pull from a private hex repo. Using the following configs while being auth'd through the commandline and adding a read key or repo key through the commndline:

{hex, [
   {repos, [
      #{name => <<hexpm:repo_name>>}
   ]}
]}.

Same scenario, can get the package through a mix dep, but not through rebar3.

BUT, when using config listed above (starting on a fresh VM without being auth'd or adding a key through the commandline):

#{<<"hexpm">> =>
      #{
        name => <<"hexpm:reponame">>
        repo_key => <<"repos_key">>,
        username => <<"username">>
  }.

the dep is successfully pulled from the private hex repo.

Should the README be updated to include these configuration settings? Unless I missed a configuration step, these configuration settings seem non-obvious and finding this issue took a good amount of time.

Love the plugin, it is very much appreciated!

starbelly commented 4 years ago

It's a solid point! I think ultimately we want to get away from users ever having to muck around in this config file, that said, it should still be documented. I have a branch which I'll be turning into a PR over the weekend that will cover all of this and will also be updating the docs on hex.pm itself. ❤️

erlef / rebar3_hex

403 resolving private dependency #103