Cannot do credentials for url based rosinstalls

[stonier]

Having a problem getting wstool to merge a http url to a rosinstall file from a private bitbucket account.

e.g.

> wstool init src /opt/ros/groovy
> wstool merge https://bitbucket.org/yujinrobot/cslam/raw/groovy-devel/cslam.rosinstall
ERROR in config: Is not a local file, nor able to download as a URL
[https://bitbucket.org/yujinrobot/cslam/raw/groovy-devel/cslam.rosinstall]: HTTP Error 401: 
UNAUTHORIZED

Note that this works for me (hands off) with wget, though I've no idea how. So I'm assuming it's possible to get this to work with wstool as well, but I'm not familiar enough with the urlib2 authentication stuff.

This isn't high priority, but would be a nice fix to have. If you need more information or testing, let me know.

[stonier]

Did some testing - wget works because it reads $HOME/.netrc and therein is my credentials for bitbucket and github.

[stonier]

Probably wouldn't take much work to add that functionality, maybe aroud the urlopen call in src/rosinstall/config_yaml.py. Apparently the usual process is to try, get a fail, and then try with credentials.

[tkruse]

Thanks for already looking into solutions. The process you describe is also presented here: http://www.voidspace.org.uk/python/articles/authentication.shtml Looking at how other projects solve the problem: https://github.com/vikraman/firewall-auth/commit/19552b93de91f6c2e01ad8b2f714b9f09bc4eddd It seems we should be able to get something clean using the standard python urllib2 and netrc.

There is also the python-requests project doing similar things, but I am not sure yet whether we'd be better off using that or relying on standard modules only. In any case if we solve this, we should also look at other python code in the ROS ecosystem for similar usages or urlopen.

[tkruse]

Hi Daniel,

so I tried to put something together. Can you confirm this would work for you: https://github.com/vcstools/rosinstall/pull/70 It worked for me with a .netrc file. On 401 response, this tries to use username and password from netrc. Without a netrc file, rowsw and wstool behave like before.

[tkruse]

What I am not sure of with this solution is whether we should call install_openener() and thus modify the behavior for this whole python process, or call Opener,open instead each time. In particular when fetching n tarballs from an authentication-requiring source, we can gain time if we do not wait for the 401 each time. On the other hand, it is cleaner if our api does not modify the default openener. So I think I'll rather go for the clean way that takes some more time.

[stonier]

I downloaded and created a deb from your fork with the netrc branch and used it with a deb created from vcstools/wstool's master branch. The following works for me:

wstool init src https://bitbucket.org/yujinrobot/cslam/raw/groovy-devel/cslam.rosinstall

:+1:

In your last comment, you are talking about caching the result for a machine so you can skip the 401 after the first result? Sounds intelligent, but greatly optimised rosinstall speed is not an essential requirement (for us at least). I guess it depends if you have other use cases where that's more important than simple and clean api, which I do think is important for development on a relatively unloved tool.

PS Just out of curiosity, is wstool init /opt/ros/groovy no longer slated to work? It failed for me with the master wstool branch.

[tkruse]

Hi Daniel, sounds good. Yes, the 401 could be skipped (and are skipped in the version I gave you,which won't be released), but like you I don't see great value and it would devalue vcstools as a generic library. I don't even know why Python offers such a hack in the standard lib. In any case clients of vcstools can do this to save 401 returns, if they want.

"wstool init . /opt/ros/groovy" has no purpose, as wstool is for catkin workspaces, where no environment files are generated by wstool. So it is supposed to not work, to tell users they are doing something wrong. Users should use

wstool init [<target>]

for catkin workspaces, or

rosws init [<target> [/opt/ros/groovy]]

for rosbuild workspaces.

I know it is confusing, I knew it beforehand, I discussed it, nobody had a better idea. Maybe the error message could be improved, even if that would mean the error message of wstool would have to know about rosws usage.

[stonier]

Got it...just made sense to me today when I wanted parallel build directories for native and cross compiles that were pinned to different install directories (e.g. /opt/yujin/groovy/native and /opt/yujin/groovy/arm-none-linux-gnu. Pinning the source workspace doesn't make sense as it would have to choose between native or compile locations. So it has to be the responsibility of the parallel build directory to know that.

What would be nice is a catkin_init_build command to make it easier to remember/set all of the cmake variables that we might use when setting up a parallel build directory.

[tkruse]

Hi Daniel,

yes, that's maybe a better way to put it. Since catkin was made for cross-compilation, it does not make sense to tie the sourcespace to a specific build environment.

Regarding your other suggestion, you can take that up with the catkin designers. So far I could not persuade anyone there that this would justify the additional required design and maintenance effort of any such tool. Note though that you should be able to write a simple shell or python script for that purpose yourself.

cheers

On Mon, Mar 4, 2013 at 8:23 AM, Daniel Stonier notifications@github.comwrote:

Got it...just made sense to me today when I wanted parallel build directories for native and cross compiles that were pinned to different install directories (e.g. /opt/yujin/groovy/native and /opt/yujin/groovy/arm-none-linux-gnu. Pinning the source workspace doesn't make sense as it would have to choose between native or compile locations. So it has to be the responsibility of the parallel build directory to know that.

What would be nice is a catkin_init_build command to make it easier to remember/set all of the cmake variables that we might use when setting up a parallel build directory.

— Reply to this email directly or view it on GitHubhttps://github.com/vcstools/wstool/issues/8#issuecomment-14366985 .

[stonier]

This is working now, so I'll close this.