Improved package support/modularization

[AdamIsrael]

The way the layer currently handles packages is inconsistent. Packages to be installed via apt are placed into layer.yaml, but python packages are in wheelhouse.txt, and there's no support for other packaging formats, such as snaps or npm.

A more modular approach would allow developers to write support for a particular packaging format, with a consistent way to install them. Two possible formats of layer.yaml to support this idea:

options:
  basic:
    packages:
      - apt:traceroute
      - pip:paramiko==2.0.1
      - npm:grunt
      - npm:grunt-cli
      - snap:hello-world

options:
basic:
  packages:
    apt:
        - traceroute
    pip:
        - paramiko==2.0.1
    npm:
        - grunt
        - grunt-cli
    snap:
        - hello-world

[marcoceppi]

totally +1 to this.

On Thu, Jun 30, 2016 at 3:24 AM Adam Israel notifications@github.com wrote:

The way the layer currently handles packages is inconsistent. Packages to be installed via apt are placed into layers.yaml, but python packages are in wheelhouse.txt, and there's no support for other packaging formats, such as snaps or npm.

A more modular approach would allow developers to write support for a particular packaging format, with a consistent way to install them. Two possible formats of layers.yaml to support this idea:

options: basic: packages:

• apt:traceroute
• pip:paramiko==2.0.1
• npm:grunt
• npm:grunt-cli
• snap:hello-world

options:basic: packages: apt:

• traceroute pip:
• paramiko==2.0.1 npm:
• grunt
• grunt-cli snap:
• hello-world

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/juju-solutions/layer-basic/issues/74, or mute the thread https://github.com/notifications/unsubscribe/AAET1XyaUHL6KGHrMmDJm_D4UVY7-uohks5qQ28_gaJpZM4JB37d .

[stub42]

Do we want to be able to specify if packages are:

• embedded as an architecture dependent binary?
• embedded in source form?
• downloaded at install/update time from 'Net?
• downloaded at install/update time as resources?

If so, pick a syntax that can support this.

[marcoceppi]

Thinking about this more, npm is a very specific type of package management, same with snap. You'd pretty much need an apt, npm layer, etc to install these. Where python/pip are already always available on all basic layer installs.

So I'm a plus one on:

options:
  basic:
    packages:
      apt:
        - pkg
      pip:
        - pkg==99.9

Then snap layers, or npm, or whatever, can build their own tactics for managing this in their layers.

[stub42]

@marcoceppi Or this issue is targeted incorrectly, and charm-build responsible. It is already handling pip packages by embedding them, and the base layer handling apt. Which is probably how the split occurred in the first place.

[marcoceppi]

Very true, but the basic layer could instead include the wheelhouse tactic to say "when pip packages, do this"

[johnsca]

The handling of wheelhouse.txt is actually currently split, with the code for building the wheelhouse currently living in charm-tools and the code for installing the wheelhouse living in layer:basic. It should be possible, however, to move the build-time processing into layer:basic via a custom tactic, which would also provide a pattern for other layers, like npm, that might want to bundle dependencies at build-time.

We could also break handling of the packages option out from layer:basic and into layer:apt, or fold layer:apt into layer:basic depending on whether we want to try to make layer:basic support other OSes.

Also, the reason I went with wheelhouse.txt instead of putting them as options in layer.yaml was twofold:

• I don't think layer options were available at the time
• It follows the Python standard requirements.txt file format, which is useful with unit tests

[johnsca]

We might also want to consider, instead of embedding the built wheelhouse directly in the charm, using it to create a resource blob. That way, those dependencies could potentially be updated independently. (That, of course, would restrict it to Juju 2.0.)

[cmars]

Some of these installation sources require repositories to be setup ahead of time -- apt repositories & keys, or a snap store URL. Will there be a way to specify these?

[johnsca]

@cmars I do think we want to add support for PPAs and keys to the layer options, but you can always use the code-driven API in layer:apt as well. You'll have to use that if you want the user to be able to change the PPA, since layer options can't respond to deploy or run time changes.

[cmars]

@johnsca Would layer:apt settings be interpreted and applied before layer:basic tried to install packages though?

[johnsca]

@cmars No. If you use layer:apt, you can not specify the packages in the layer options for basic. We do still need to add support for PPAs and keys to basic.

I just remembered the reason we didn't want to break package handling out entirely from layer:basic, which is because some charms will need packages installed before the charm code runs, which would be too late for layer:apt to handle. Perhaps layer:apt could use the exec.d functionality, though?

[marcoceppi]

Layer packages are packages that need to exist before a hook runs, much like wheelhouse is needed before any reactive runs.

Marco

On Thu, Jun 30, 2016, 8:22 PM Cory Johns notifications@github.com wrote:

@cmars https://github.com/cmars No. If you use layer:apt, you can not specify the packages in the layer options for basic. We do still need to add support for PPAs and keys to basic.

I just remembered the reason we didn't want to break package handling out entirely from layer:basic, which is because some charms will need packages installed before the charm code runs, which would be too late for layer:apt to handle. Perhaps layer:apt could use the exec.d functionality, though?

— You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub https://github.com/juju-solutions/layer-basic/issues/74#issuecomment-229745542, or mute the thread https://github.com/notifications/unsubscribe/AAET1b6vsse-BEszJjDCf51lD_XrZkGOks5qRAlWgaJpZM4JB37d .