charm build fails, and doesn't tell me why

[stub42]

Checklist

• [X] Confirmed this is an issue with charm-tools, not charmstore-client
• [x] Provide versions of tools used
• [ ] Described the feature or ways to replicate the issue

What version am I running?

I ran the following command: snap info charm and got the following ouput:

name:      charm
summary:   charm and charm-tools
publisher: charms
contact:   juju@lists.ubuntu.com
license:   unknown
description: |
  charmstore-client and charm-tools
commands:
  - charm
snap-id:      2Rryoc2ylScfbFl4eQtpntHD9iuZuMvt
tracking:     stable
refresh-date: 2018-04-19
channels:                                          
  stable:    2.2.3                     (114) 103MB -
  candidate: 2.2.3                     (114) 103MB -
  beta:      ↑                                     
  edge:      2.2.4-dev+git-17-g6ccd114 (148) 41MB  -
installed:   2.2.3                     (114) 103MB -

I am using: Ubuntu 18.04

Issue/Feature

'charm build' was working on Friday, now it is not.

I expect/expected the following

The charm to build.

What I got

'charm build' fails. It gives me no information why. It does spit out a few lines of an interface I'm including indirectly, with no context.

charmtools.build.fetchers: Checking layer index: https://juju.github.io/layer-index/interfaces/nrpe-external-master.json
charmtools.build.fetchers: Found repo: https://github.com/cmars/nrpe-external-master-interface
charmtools.fetchers: git clone https://github.com/cmars/nrpe-external-master-interface /home/stub/charms/repo/deps/interface/tmpzVATgo: Cloning into '/home/stub/charms/repo/deps/interface/tmpzVATgo'...
warning: templates not found /snap/charm/114/share/git-core/templates

build: Processing layer: layer:basic (from ../layers/basic)
build: Processing layer: layer:apt (from ../layers/apt)
build: Processing layer: layer:nagios (from ../layers/nagios)
build: Processing layer: plinth (from .)
build: Processing interface: nrpe-external-master
build: Processing interface: pgsql (from ../interfaces/pgsql)
../../deps/interface/nrpe-external-master/provides.py --> /home/stub/charms/repo/builds/plinth/../../deps/interface/nrpe-external-master/provides.py:
11   |                   servicegroups=None, unit=None):
12   |         nagios_files = self.get_local('nagios.check.files', [])

        if not unit:
            unit = hookenv.local_unit()

[stub42]

The cited lines where part of the changes landed in interface:nrpe-external-master several hours ago, and that would be the trigger. The issue here is that 'charm build' is not telling me what the problem is, so I can't raise the issue upstream since I don't know what it is.

[stub42]

More confusingly, if I clone the nrpe-external-master repository locally, 'charm build' works, but 'charm build --no-local-layers' continues to fail.

[johnsca]

The output is coming from utils.delta_python_dump which is being called from InterfaceCopy.lint.

I'm not at all clear on the purpose of that bit of code. It's detecting some sort of change to the interface layer, but it's not a local modification, as that's checked elsewhere using the manifest. Also odd is that both the source and the target seem to be the deps directory, though the relative paths mean they might be different deps directories even though they seem like they should be the same.

I don't know if that code is detecting a valid issue or not. Perhaps it's guarding against two layers somehow fighting over the same interface layer name, or it might just be buggy code that we haven't hit before and should be removed. It could even be some odd environment issue with the source and target path meant to be the same but somehow being different due to the snap confinement or some combination of the environment vars (i.e., JUJU_REPOSITORY, LAYER_PATH, and INTERFACE_PATH). Certainly, whatever it was meant to detect is not being explained in the output, so it's super unhelpful.

[johnsca]

Apparently, I touched the lint method a couple of years ago but didn't meaningfully change that bit, and it's otherwise from the initial implementation.

[johnsca]

The delta_python code isn't used anywhere else, and I'm struggling to come up with a reason why it is included there. The only even somewhat plausible thing I can think of would be to prevent charm authors from overriding Python code from interface layers but I don't know why we would want to prevent that.

My inclination would be to strip all of that code out, but I would like to know more about why it's failing in this case. Perhaps you can dive more into the differences between those two files and where the difference is coming from?

[stub42]

The problem is triggered by my tree layout. The paths are not absolute, so end up pointing to different locations depending on what the current working directory is if symlinks are involved. I had ended up with two deps directories, in ~/charms/deps and ~/charms/builds/deps, and one of them was not being updated and contained an older clone of the nrpe-external-master git repo.

I think the fix is for charm-build to only pass absolute paths to subprocesses (or just use absolute paths everywhere), so that subprocesses that change the working directory or resolve the paths themselves get consistent destinations.

[stub42]

Interestingly, after cleaning out all deps directories, only one gets recreated when I 'charm build'. Maybe the newest version of 'charm build' was broken by deps caches created by older versions of the tool?