Weird result when moving from subversion to git

[mnlevy1981]

Summary of Issue:

I have a branch of POP where we pointed to an SVN tag of MARBL; we have decided to point to a git tag instead

Expected behavior and actual behavior:

I expected to be able to change protocol from svn to git in my externals file, but instead I get the error included below. I ended up running rm -rf and then checkout_externals but it seems like tool should be able to handle this case.

Steps to reproduce the problem (should include model description file(s) or link to public repository):

1. Mirror a git repository in subversion (or vice versa)
2. Generate an external pointing to subversion and run checkout_externals
3. Replace subversion protocol with git and try to rerun checkout_externals

What is the changeset ID of the code, and the machine you are using:

This is on hobart, using manic-v0.8.0

have you modified the code? If so, it must be committed and available for testing:

Nope, this is clean code.

Screen output or log file showing the error message and context:

$ ./externals/manage_externals/checkout_externals
Processing externals description file : Externals.cfg
Processing externals description file : Externals_POP.cfg
Checking status of externals: pop_externals, cvmix, marbl,
    ./externals/CVMix
?   ./externals/MARBL
----------------------------------------------------------------------
The external repositories labeled with 'M' above are not in a clean state.

The following are two options for how to proceed:

(1) Go into each external that is not in a clean state and issue either
    an 'svn status' or a 'git status' command. Either revert or commit
    your changes so that all externals are in a clean state. (Note,
    though, that it is okay to have untracked files in your working
    directory.) Then rerun checkout_externals.

(2) Alternatively, you do not have to rely on checkout_externals. Instead, you
    can manually update out-of-sync externals (labeled with 's' above)
    as described in the configuration file Externals.cfg.

----------------------------------------------------------------------

[billsacks]

I sympathize with this issue, because I ran into the same thing recently. However, given that manage_externals is stateless (and so doesn't know that you changed something that used to be in svn to git) and fairly conservative (so it wants to avoid overwriting files in directories that it's not managing), it wasn't clear to me how it could feasibly handle this situation. Do you have any ideas?

[mnlevy1981]

Instead of assuming the existing protocol is whatever is listed in the configuration file, maybe loop through all acceptable protocols in a reasonable order (e.g. start with the listed protocol)? If the listed protocol is git but the current checkout is svn, manage_externals could still detect that and run svn status to make sure it's not clobbering any uncommitted changes.

[billsacks]

Okay, I can see how that could work. I'll defer to @bandre-ucar on how hard that would be.

[mnlevy1981]

After talking to @billsacks, I think my preferred behavior would be to generate an error message specific to the case of the current external not being under version control of the expected protocol. Given subversion's centralized nature, I don't see an issue with letting checkout_externals remove a subversion-based external and then check out the same external from git but going the other way is not as clear cut. What if a component moves from git to subversion but a user has local branches in his or her git checkout that have not been pushed to a remote?

So an error message along the lines of

$ ./externals/manage_externals/checkout_externals
Processing externals description file : Externals.cfg
Processing externals description file : Externals_POP.cfg
Checking status of externals: pop_externals, cvmix, marbl,
    ./externals/CVMix
?   ./externals/MARBL
----------------------------------------------------------------------
The external repositories labeled with '?' above are not under version
control using the expected protocol. If you are sure you want to switch
protocols then run "rm -rf [directory]" before re-running the
checkout_externals tool.

----------------------------------------------------------------------

(exact text TBD).

[billsacks]

I spent a few minutes looking into this. I can't see an easy way to answer the questions, "Do any repos have a status of M? Do any repos have a status of ?". So I'd suggest the simple approach of just extending this message: https://github.com/NCAR/manage_externals/blob/1a7c59d3a13d204c78ce7e010fe3ce32705a61b6/manic/checkout.py#L333-L346

with your suggested text. Users will see some irrelevant information in the message, but in my mind that isn't a big deal. How does that sound to you, @mnlevy1981 ?

[billsacks]

@mnlevy1981 This needs to be done by Friday if we want this in the release (which may or may not be important). How do you feel about my last comment - i.e., simply adding a bit of extra text to the output message?

[mnlevy1981]

@billsacks Sorry about ignoring your previous message. I'm okay with updating the message you highlighted rather than introducing a new message for the CESM 2 release... but can we leave this ticket open in hopes of a better solution after the release(s) are behind us? I'm happy to spend some time looking at the code in more depth this summer but definitely don't have the spare cycles at the moment.

[billsacks]

I inserted the suggested message in #102 . Assigning this to @mnlevy1981 for a more robust fix long-term if he wants to put one in place (though, for the record, I personally don't feel it's critical).