./update --copy fails: File exists in make_launcher

[ahmadia]

When trying to follow @cekees's instructions for building the stack, I notice that the command

./update --copy ${PROTEUS_PREFIX}

fails if I have already run it before:

[python/pahz ERROR] [Errno 17] File exists in make_launcher('/Users/aron/sandbox/acoe/proteus/hashstack/opt/python/pahzt/bin/2to3', u'/Users/aron/sandbox/acoe/proteus/darwin/bin/2to3')
[python/pahz ERROR] hit command failed

[aterrel]

It should be:

rm ${PROTEUS_PREFIX}
./update --copy ${PROTEUS_PREFIX}

[ahmadia]

I suppose you mean rm -rf ${PROTEUS_PREFIX}

[aterrel]

Well only for n00b$ that don't use

alias rm='rm -rf'

=P

[aterrel]

On a more serious note, we had a long conversation about this.

• @dagss wants to get rid of --copy because it makes it so the user can futz with the installed profile.
• @certik likes to futz with it since that is often what it takes to "just make it work" (C)
• @aterrel pointed out if we keep copying and want to keep things working we should have something like brew doctor

The compromise was: Worry about it later...

[dagss]

First off, the failure if you already built is the correct behaviour in this case -- what if you did lots of futzing around that you don't want to accidentally remove?

Second: The new spec system is going to replace ./update, so we can't wait too long.

What I remember as our conclusion was that we keep having this around as a flag (--copy), for debug purposes, but don't make it the default (like it is made the default in the Makefile). Also, encourage users to install things into their own prefix independent of the Hashdist prefix.

@certik, my point in this discussion is that the different is rather slight between

export PATH=/home/dagss/local/bin:/home/dagss/myhashdistprofile/bin:$PATH

and

export PATH=/home/dagss/myhashdistprofile/bin:$PATH

and the only advantage of the --copy flag that I can see is that it allows the latter.

Proposal: Allow a user to specify additional prefixes that are put in front when doing hit use myhashdistprofile. I.e. hit use myhashdistprofile could lead to the first export above rather than the second one.

[certik]

@dagss --- so without --copy how do you allow users to install things by hand into their profile? I don't understand your proposal, so far --copy seems to me like the only way to do it.

[dagss]

Well, my answer above was that you just don't...

Consider a typical Linux distro: The user must stay away from /usr, but gets to play with /usr/local. That way the user doesn't trample on the foots of the install tool. They are both in PATH and are transparently used together. We can just encourage the same, for the same reasons.

I know of exactly 2 reasons for why it would be convenient to install things into a Hashdist-generated profile, rather than just some ~/my-own-precious-prefix-tree (hereafter the "manual overlay profile"):

1) One extra entry in PATH rather than two.

2) How Python looks up packages.

Any more reasons?

The disadvantages to --copy (as the default!) are pretty huge, because it removes the 1:1 correspondance between a profile description and a profile/prefix-tree. This affects the ability to do hit checkout HEAD^, or hit checkout someotherbranch. Basically it means you can't jump around between branches quickly any longer, but must only do it carefully, to avoid breaking manually installed software.

My idea above was simply that when you do hit use myprofile, we insert some user-defined manually managed prefix (the manual overlay profile) in front of the Hashdist profile bin-dir.

It does need more sketching out for how to work cleanly with Python, but we get 90% there simply by letting hit use also modify PYTHONPATH to include the manual overlay profile, and then demand that users type python setup.py install --prefix=$PREFIX (where hit use could even set $PREFIX).

The 100% solution would be something that didn't touch PYTHONPATH and allows a simple python setup.py install. But I think the simple thing above is already sufficient to drop the --copy flag.

Concretely, I propose that in a profile YAML file, there's a new clause,

user_overlay: /home/dagss/my-own-previous-prefix-tree

and then when I do hit use someprofile, and someprofile/default.yaml contains this clause, PATH, PYTHONPATH and PREFIX are set as discussed above.

[dagss]

@certik in case you didn't get an email for the above

[certik]

I got the message. There is more advantages:

• having one prefix for everything, thus allowing to easily investigate by hand, change things, and at the end just "rm -rf" the directory and be done with it
• lot of people are used to virtualenv. Doesn't anaconda btw does the same thing too? So it'd be nice to provide similar workflow for people that want it.

I agree --copy should not be the default.

[dagss]

These are both "workflow people are used to", and Hashdist is somewhat about experimenting with a different worflow... put another way, I'm not against people using Anaconda or virtualenv per se if their needs are filled by them :-) In fact, I think some users may at some point be well served by Hashdist producing Anaconda binaries of its packages, if one wants that style of user interface, rather than Hashdist cloning it (I think conda doctor makes more sense than hit doctor, and it's to some degree already there with their packaging-up-user-changes-support).

But we agree here!, I didn't argue for removing it, just that it shouldn't be default. So we're good.

[dagss]

BTW my words were to strong, Hashdist as a project isn't this or that (and I don't define it), I should have said that "my own interest in Hashdist lies primarily with experimenting with a different workflow".

But if we want to push the reproducibility angle (which is where I think we have a real competitive advantage), 1:1 correspondance between git repo of configuration and the profile is not something to give up lightly, so I weigh that heavily myself

[aterrel]

For now I think we can table the issue. The --copy flag (which I would call --copy_to_venv) is one way to use hashstack, but not the only.

[certik]

+1

Sent from my mobile phone. On Sep 20, 2013 4:11 PM, "Andy R. Terrel" notifications@github.com wrote:

For now I think we can table the issue. The --copy flag (which I would call --copy_to_venv) is one way to use hashstack, but not the only.

— Reply to this email directly or view it on GitHubhttps://github.com/hashdist/hashstack/issues/107#issuecomment-24844535 .