Force authentication with username & password?

[eehret]

Hi all, I'm sorry if this is an obvious one, but I am stumped at the moment.

In our Nexus repository, we have disallowed anonymous access for licensing reasons. Every time I try to grab something from the Rubygems proxy repository that I created in Nexus, it appears that gem is trying to just access Nexus anonymously. This will obviously not work. How do I force gem to authenticate when fetching?

Note: i am able to upload to a hosted repo just fine. If I use the 'gem nexus' command, I can upload to it using my stored credentials that I provided when prompted.

However with a proxy repo, I can't upload to it and it always seems to want to access anonymously...

Any ideas? Thanks

[mkristian]

not sure if I understand all of it. you can ONLY upload to hosted repo, the proxy can have a hosted repo as source or any other repo anywhere else.

[eehret]

Ok, I'm confused.

When you say you can ONLY upload to hosted repo, what do you mean? normally with Nexus I can have a hosted repo that I would upload things into, but I can also read from them just fine. My 3rd party repo is like this.

To have better control over the Gem repo, I'm leaning toward making it hosted instead of a proxy. So that's fine. But I will need to force authentication somehow. Right now gem keeps wanting to connect to Nexus anonymously.

i.e: When I do a 'gem sources -add my_repo_url' it fails with a 401 error because it's trying to connect anonymously, which our Nexus configuration does not allow.

[mkristian]

I guess your main point is that you have your repo password protected and can not use gem install or 'bundle install` -

you can set the username and password with the source url for your repo, something like

http://deployment:deployment123@localhost :8181/nexus/content/repositories/gems/

that is the only way I can think of.

the hosted vs. proxy I am sure you will figure it out - sorry for confusing you.

[eehret]

Hi,

Interesting. Using the syntax you suggested, I can add the source to my sources.

However, now when I do a 'gem list -r', the list is empty. No remote gems.

I know there's one in there, because I can see it when I browse the Nexus index after logging in via the browser.

Any ideas?

Thanks for your patience and your help.

[mkristian]

seeing an empty list is when I had not the permissions to read the repo.

my setup:

• user has permissions 'Repo: All Repositories (Read)'
• repository has configuration 'Publish URL' set to True

[eehret]

Thanks for getting back to me.

I checked to make sure I had the role you indicated above. I did not. I've added it. Publish URL was already set to true (and I can see the gems in my web browser too)

I turned on DEBUG logging, to try to see what's happening. I see a lot of lines involving latest_specs.4.8.gz. It's definitely trying to do something, but I'm not sure how to interpret what I'm seeing.

If I sent you a log snippet, would you be able to help me troubleshoot this? I would really appreciate it.

Thanks! Eric

[eehret]

Snippet is here: https://gist.github.com/eehret/5a70faa620a4f67c3126

[mkristian]

hi, those recursive attributes are scary and I do not see them locally :(

beside the attribute trail the log file looks OK.

a wrong user or wrong password would end the log with jvm 1 | 2013-09-05 15:49:59 DEBUG [nxevthost-1-thread-10] org.sonatype.nexus.feeds.record.NexusAuthenticationEventInspector - Unable to authenticate user [dieployment] from IP address 127.0.0.1

no permissions looks like jvm 1 | 2013-09-05 15:51:26 DEBUG [nxevthost-1-thread-12] org.sonatype.nexus.feeds.record.NexusAuthorizationEventInspector - Unable to authorize user [deployment] for read(HTTP method "HEAD") to /nexus/content/repositories/gemshosted/latest_specs.4.8.gz from IP Address 127.0.0.1, user agent:"n/a"

and authentication went OK looks like jvm 1 | 2013-09-05 15:52:51 DEBUG [qtp932236882-53] deployment org.sonatype.nexus.plugins.ruby.hosted.DefaultRubyHostedRepository - gemshosted retrieveItem() :: FOUND gemshosted:/latest_specs.4.8.gz

you can also gem list -r nexus --verbose to see the client side status.

currently I am using nexus-2.6.0_05 and the latest snapshot of the nexus-ruby-plugin with java version "1.7.0_25" (fedora19 OS)

the attribute thingy is a bad sign and could be the reason why things do not get delivered to the client. could you setup a fresh host repo and see if the attributes thingy remains. in if so please give some info of your setup (versions) so I can try to reproduce it.

[eehret]

I agree about the recursion...

It looks like authentication was fine. So now I'm stuck on the recursion stuff... What I was wondering is if somehow the rubygem metadata got messed up. Is there a way to tell Nexus to rebuild it?

[mkristian]

rebuild no - maybe a good idea to add.

but you could at least delete all those attributes

sonatype-work/nexus/storage/gemshosted/.nexus/attributes

that directory sits usually beside your NEXUS_HOME and my hosted repo is called 'gemshosted'

can you download the latest_specs.4.8.gz via curl or wget using the url with authentication.

[eehret]

What is contained in those latest_specs files? Is it like some kind of manifest?

With a hosted repository that contains only 1 gem, which I have uploaded by hand, how would this file get updated (assuming it is indeed a manifest) ?

To answer your question, I don't have curl or wget (non-*nix environment here...) but I can grab it just by pointing my web browser to the published URL and downloading the file. It appears to be empty or corrupt. Only 4 bytes long. https://gist.github.com/eehret/f3d5e6056c7ba952899e

[mkristian]

the latest_specs file is basically a hashmap with gemname to latest version. whenever you upload a gem to hosted the latest_specs specs will updated respectively.

but I agree that a single gem should not corrupt any of those files.

so what OS, java version nexus version are you using - assuming you use the latest nexus-ruby-plugin.

[eehret]

Wow...

Looking at the contents of those subfolders (and I have no idea how many there are... they go very deep) it looks like Nexus may have been updating the wrong copy of the latest_specs file. Some of the ones deeper in the tree actually have contents.

I'm running Nexus on Windows 7, under Java 7 (patch 21). This is Nexus 2.6.0-05. I was planning to update Nexus soon.

If I update nexus to latest, will I need to recompile the ruby support?

[eehret]

It looks like some changes were done in the indexer in 2.6.1. Perhaps that will help.

[mkristian]

On Thu, Sep 5, 2013 at 6:58 PM, Eric Ehret notifications@github.com wrote:

If I update nexus to latest, will I need to recompile the ruby support?

not sure - in the past that was a problem between those version

[eehret]

Ok. I'm going to update my Nexus to the latest version, rebuild the ruby support module, and give it another try. I'll get back to you.

Thanks again :)

[eehret]

Well, I updated to Nexus 2.6.2-01, with nexus-ruby-plugin version 1.0.1-SNAPSHOT.

I'm not sure whether this version is stable. There are no tags to build from, so I grabbed the latest from master branch.

Nexus started up fine, and I can create a rubygems repo, but now the artifact upload tab is missing on the hosted rubygems repo!

edit: may be a problem with Nexus as opposed to the plugin, because my artifact upload tab is also missing from my Snapshots hosted repo. But my 3rd party repo is fine. I may need to open a Sonatype support ticket for that one.

[mkristian]

I think you have already the nexus gem installed. then you can upload a gem with

$ gem nexus my-gem-1.0.gem

[eehret]

Good point. That worked to upload the gem.

Now we're back to my original problem: although I can see the gem when I browse the public URL (after I authenticate in the browser), I get an empty gem list when I do 'gem list -r'

[eehret]

It is doing that crazy recursive .nexus attributes thing again.

I think the Nexus indexer is failing badly here.... let me see if I can exclude that repo from my indexing task.

update: nevermind. it isn't the scheduled task that's causing the problem. I can see that task hasn't run. It seems that as soon as I create the repo, before I even do anything with it, Nexus goes and create those massive numbers of recursive folders under .nexus

[eehret]

After multiple repeated tests, I have so far determined that this recursion problem does not happen with proxy repository. Only with hosted repository.

I hope that helps or gives some clues...

[mkristian]

what does gem list -r --verbose produce ? 401 Unauthorized ? that output also shows the complete request the gem command uses.

yes - I will see if I can find something on hosted regarding those attributes.

but you say that the browser works - which basically indicates that the server is doing it right but the client does not handle those urls (With credentials) correct.

[eehret]

D:>gem list -r --verbose

* REMOTE GEMS *

HEAD http://@10.159.2.112:8081/nexus/content/repositories/ruby-gems/latest_specs.4.8.gz 304 Not Modified

I omitted the authentication token.

fyi - I have also opened a support ticket with Sonatype to see if someone can help identify why we're getting that recursion problem happening with the hosted repo.

[eehret]

In the browser, this is what I see:

[mkristian]

you got 304 NOT MODIFIED which basically means that the local data has the same timestamp then the remote one.

first you can delete cached data with

$ gem sources --clear-all

then I think it is best to remove those permissions and try to get it work without username/password thingy.

[mkristian]

got a PR from tamas about the path recursion - looks like we are a step closer.

[eehret]

Should I try it again?

[mkristian]

yes - please, I just pushed the changes before I read your last message.

[mkristian]

does this work now ?

[eehret]

Yes.. working beautifully now!

Thank you!