pack install git ref error in st2packs docker build

[lukepatrick]

SUMMARY

Using the st2 pack install from st2common/util/pack_management.py in the st2packs docker builder, an error condition on getting branch name can occur.

ISSUE TYPE

Pick one below and delete the rest:

• Bug Report

STACKSTORM VERSION

st2packs base https://github.com/StackStorm/st2packs-dockerfiles

OS / ENVIRONMENT / INSTALL METHOD

Docker - st2packs docker builder https://github.com/StackStorm/st2packs-dockerfiles

STEPS TO REPRODUCE

To reproduce, we use a CI job to build packs for changes on master, tags, and certain branches, e.g. release.

The CI will checkout the release branch, and then the Docker file will run copying the pack files into the container and attempting to install:

FROM myregistry/st2/base:v1.0.0 AS builder

COPY . /opt/stackstorm/packs/my-pack

RUN /opt/stackstorm/st2/bin/st2-pack-install /opt/stackstorm/packs/my-pack

Now we did realize now that we should be using st2-pack-install with a file prefix, e.g. st2-pack-install file:///opt/my-pack

However, the above Dockerfile process has worked for master and tags, just not this release branch

EXPECTED RESULTS

pack to install with git ref of a branch or for this to error out on all git ref's

ACTUAL RESULTS

errors

 File "/opt/stackstorm/st2/lib/python2.7/site-packages/st2common/util/pack_management.py", line 119, in download_pack
    ref=pack_version)
  File "/opt/stackstorm/st2/lib/python2.7/site-packages/st2common/util/pack_management.py", line 163, in clone_repo
    raise e
TypeError: HEAD is a detached symbolic reference as it points to 'dd4047d9816aa31f5d65df56b1cd420c2dff3add'
The command '/bin/sh -c /opt/stackstorm/st2/bin/st2-pack-install /opt/stackstorm/packs/foo' returned a non-zero code: 1

[LindsayHill]

I wonder if this is at least in part related to your copying the content to /opt/stackstorm/packs/mypack, and trying to run st2-pack-install file:///opt/stackstorm/packs/mypack.

The issue there is that the pack install is probably trying to copy the content from the specified location across to the packs dir - that is, to the same place.

Do you see the same errors if you put your pack in some other temporary location first?

[lukepatrick]

this errors (only when I'm on a branch, not master/tag): RUN /opt/stackstorm/st2/bin/st2-pack-install /opt/stackstorm/packs/my-pack

this works as intended RUN /opt/stackstorm/st2/bin/st2-pack-install file:///opt/stackstorm/packs/my-pack

I assume because file:// will cause a by-pass of the TypeError here https://github.com/StackStorm/st2/blob/master/st2common/st2common/util/pack_management.py#L158

[LindsayHill]

Right, but what if you use a different path? Right now you're putting your content in place, and then running st2-pack-install which will copy it to the same destination.

[lukepatrick]

you mean like follow the examples/instructions? that's crazy talk.. :wink: Let me run some tests and get back to you

[LindsayHill]

There could still be an underlying issue. This is just a hunch about what might be happening. Haven’t looked at code at all.

I do recall a related issue a while back with st2 pack install. Unclear if this script might be affected by the same thing

[Kami]

Yeah, what @LindsayHill said - if you copy your pack directory directly to /opt/stackstorm/packs/ this will likely cause issues and won't work correctly since st2-pack-install also copies files there.

One approach would be to copy it to some temporary directory and then run st2-pack-install file:///....

Or even better (and preferred) since you already have pack content on disk in the correct place - you really just want to setup pack virtualenv environment - that's all that st2-pack-install script does (in addition to downloading a pack from git repo and moving it to the correct directory).

To do that you you can use st2-pack-setup-virtualenv command - st2-pack-setup-virtualenv <pack name>.

[Kami]

^ @warrenvw @armab Perhaps an action item for HA docs - in such scenario, there is really no need to use st2-pack-install so we should probably document that scenario and point user directly to st2-pack-setup-virtualenv...

[lukepatrick]

Ok, I tested different iterations of the path - copying the files to a /tmp dir or leaving as is. This has no affect, I could always reproduce the error TypeError: HEAD is a detached symbolic reference as it points to 'de3ad4a simply by being on a commit in my release/1.0 branch

[Kami]

Then it could indeed be a bug in our code or the git library we use.

Which version of StackStorm are you using? I'm asking because the line numbers from your traceback don't match line numbers from the latest version which could indicate you are using an older version...

[lukepatrick]

2.10

[arm4b]

The initial message (and https://github.com/StackStorm/st2/issues/4629#issuecomment-481495438) shows that user is trying to install pack in the following way:

(by using direct path)

/opt/stackstorm/st2/bin/st2-pack-install /opt/stackstorm/packs/my-pack

while this works OK (with file:///):

/opt/stackstorm/st2/bin/st2-pack-install file:///opt/stackstorm/packs/my-pack

The issue is an absense of file:// prefix. We documented local dir pack install via file:// in both https://github.com/StackStorm/st2packs-dockerfiles and https://docs.stackstorm.com/packs.html#installing-a-pack so I'm not sure where it comes from.

However would be better to catch that case & throw some better error message to avoid such misunderstanding in future.

[arm4b]

To be clear,

this (documented with file://) works OK:

$ /opt/stackstorm/st2/bin/st2-pack-install file:///tmp/ansible
2019-04-10 12:38:36,698 INFO [-] Installing pack "file:///tmp/ansible"
2019-04-10 12:38:36,755 INFO [-] Successfully installed pack "ansible"
2019-04-10 12:38:36,755 INFO [-] Setting up virtualenv for pack "ansible"
2019-04-10 12:38:57,303 INFO [-] Successfully set up virtualenv for pack "ansible"

this (never documented way) fails & I can reproduce the same error message:

$ /opt/stackstorm/st2/bin/st2-pack-install /tmp/ansible
2019-04-10 12:51:49,326 INFO [-] Installing pack "/tmp/ansible"
Traceback (most recent call last):
  File "/opt/stackstorm/st2/bin/st2-pack-install", line 42, in <module>
    sys.exit(install_pack.main(sys.argv[1:]))
  File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/st2common/cmd/install_pack.py", line 66, in main
    proxy_config=proxy_config, force_permissions=True)
  File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/st2common/util/pack_management.py", line 119, in download_pack
    ref=pack_version)
  File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/st2common/util/pack_management.py", line 163, in clone_repo
    raise e
TypeError: HEAD is a detached symbolic reference as it points to 'eba7616bacbaed1f2f10eee2c2d1483502ab1744'

@lukepatrick To confirm, does it work OK with file:/// for you?

[Kami]

If it's indeed just an issue with missing file:// prefix then I agree that we should update the code to throw a more user-friendly error message.

[lukepatrick]

correct, it's not an issue with file://

The issue I have noticed is the inconsistencies based on the git ref, whether on master or a branch. As others have noted, improved error messaging and investigating the git libraries used would be likely fixes.