search

michaelsauter / crane

Crane - Lift containers with ease
MIT License
922 stars 61 forks source link

env vars, build args errors / plugins and multi-config management #342

Open roscopecoltran opened 6 years ago

roscopecoltran commented 6 years ago

Hi guys,

Hope you are all well !

1. Env errors

I just found an error while trying to create dynamic image names with crane as it works well with docker-compose.

.env:

###########################################################
################ Containers Customization #################
###########################################################

### CONSOLE ############################################
GOLANG_BASE_IMAGE_NAME=golang
GOLANG_BASE_IMAGE_TAG=1.11beta3-alpine3.8
GOLANG_BASE_IMAGE=${GOLANG_BASE_IMAGE}:${GOLANG_BASE_IMAGE_TAG}

crane.yml:

services:
  crane:
    image: ${GOLANG_BASE_IMAGE}
    # image: michaelsauter/golang:1.8
    rm: true
    interactive: true
    tty: true
    volume:
    - "$PWD:/go/src/github.com/michaelsauter/crane"
    # volume: ["$GOPATH:/go"]
    workdir: /go/src/github.com/michaelsauter/crane
    share-ssh-socket: true
    cmd: ["/bin/sh"]

commands:
  test: run crane make test
  build: run crane make build
  build-darwin: run crane make build-darwin
  build-darwin-pro: run crane make build-darwin-pro
  gofmt: run crane gofmt -w crane

accelerated-mounts:
  crane:
    uid: 1000
    gid: 1000

error:

crane lift crane
Pulling image  ...
invalid reference format
ERROR: exit status 1

Also, I noticed that if I want to create a multi-stage Dockefile and use dynamic base image names, crane is not passing them.

Please note that even if I use only the "image tag" as a variable and hard code the "image base name", it doesn't work. Even, if I pass build-args in the docker-compose file and not in the .env file.

####################################
## Builder - image arguments
####################################
ARG BUILDER_ALPINE_VERSION=${BUILDER_ALPINE_VERSION:-"3.7"}
ARG BUILDER_GOLANG_VERSION=${BUILDER_GOLANG_VERSION:-"1.10.3"}
ARG BUILDER_IMAGE_TAG=${BUILDER_IMAGE_TAG:-"${BUILDER_GOLANG_VERSION}-alpine${BUILDER_ALPINE_VERSION}"}
ARG BUILDER_IMAGE_NAME=${BUILDER_IMAGE_NAME:-"golang:${BUILDER_IMAGE_TAG}"}

####################################
## Builder
###################################
FROM ${BUILDER_IMAGE_NAME} AS builder

Initially, I spotted these errors while working with laradock (https://github.com/laradock/laradock), and especially with the MySQL container (https://github.com/laradock/laradock/blob/master/mysql/Dockerfile#L1)

Plugins / Composable commands

I started to extend custom commands as sometimes I have to group them by type and other dynamic stuff but I realized that the best approach would be to have a plugin registry for pluggable and composable macros in my custom commands.

The idea is to get more scriptable containers and make of crane a tool that simplifies the process of creating reproducible experiments from command-line executions, a frequently-used common denominator in machine learning or advanced local stacks.

Globally, I just want to import them as golang plugins so no feature creep for other users or myself with the open source/pro version of crane.

Refs:

Note: But I need to get the final struct loaded by crane after parsing the docker-compose and/or crane file in order to pass correct variables to my plugins.

So it would be awesome to have a plugin registry in crane, with shared contextual data, so we could extend it easily.

Go plugins examples:

3. Config files

Would be awesome if we could use https://github.com/jinzhu/configor with crane so we could maybe provide with a string slice some config files (yaml, json, toml or env variables) in order to import plugin config files without hassles too. I already added it in my private fork of crane so I could split a long docker-compose file like the laradock's one into sub-files per group of services.

Note: config files splitting could apply only for crane files.

I saw that json and yaml tags are already set, but with configor we could support YAML, JSON, TOML, Shell Environment. Implement, production/development crane configs (crane.production.yml,...), even more, be able to group my stacks for the same project per operating systems (crane.centos.yml, crane-rmdb.alpine.yml...) based config files... Please check the Advanced section usage at https://github.com/jinzhu/configor#advanced-usage

type config struct {
        RawPlugins  map[string]plugin                 `json:"plugins" yaml:"plugins" toml:"plugins"`
    RawPrefix            interface{}                  `env:"CRANE_PREFIX" json:"prefix" yaml:"prefix" toml:"prefix"`
    RawContainers        map[string]*container        `json:"services" yaml:"services"  toml:"services"`
    RawGroups            map[string][]string          `json:"groups" yaml:"groups"  toml:"groups"`
    RawHooks             map[string]hooks             `json:"hooks" yaml:"hooks"  toml:"hooks"`
    RawNetworks          map[string]*network          `json:"networks" yaml:"networks"  toml:"networks"`
    RawVolumes           map[string]*volume           `json:"volumes" yaml:"volumes" toml:"volumes"`
    RawCmds              map[string]interface{}       `json:"commands" yaml:"commands"  toml:"commands"`
    RawAcceleratedMounts map[string]*acceleratedMount `json:"accelerated-mounts" yaml:"accelerated-mounts"  toml:"accelerated-mounts"`
    RawMacSyncs          map[string]*acceleratedMount `json:"mac-syncs" yaml:"mac-syncs"  toml:"mac-syncs"`
    containerMap         ContainerMap
    networkMap           NetworkMap
    volumeMap            VolumeMap
    acceleratedMountMap  AcceleratedMountMap
    groups               map[string][]string
    cmds                 map[string][]string
    path                 string
    prefix               string
    tag                  string
    uniqueID             string
}

4. Accelerated Mounts for Mac

Mainly, I have already forked crane, and started to modify the code to implement all points above but I just realized that, as a mac user, the accelerated mount was a mandatory feature for me too; that's the only reason why I purchased a crane license... ^^ So either, I spent a couple of 1-2 weeks in order to create my own solution with Unison and solve all the problems above, or I keep using crane because I can extend it for more advanced use and keep merging your latest changes.

So my question(s) is/are double-sided, either there is a way to import accelerated mounts if you are a license owner and merge them with a fork and that's fine, or is it possible to create a registry for external golang plugins and config files in the open source/pro version...

The env variables errors, it would be nice to fix it in any of crane's version.

While hoping that I was clear enough, let me thank you, in advanced, for your time and replies.

Cheers, Rosco Pecoltran

michaelsauter commented 6 years ago

Hi,

that's a lot to digest :) First of, thanks for using Crane Pro, I hope you enjoy it!

To address your points:

1) Not sure where the issue is. Have you tried with ´--verbose´? It usually helps to debug problems as it displays all commands Crane is executing.

2 and 3) I want to keep Crane simple and not add anything big to it. So I'm very sceptical about your proposed changes ... maybe there is a way for you to use another tool as a wrapper to crane?

4) What I am selling in the PRO version is basically just the integration of Unison into Crane. I won't publish this code or give it out, sorry. I see why you would want this if you fork it, but there is no way for me to give it out and keep it proprietary at the same time. Sorry!

So ... I would say run Crane in verbose mode and paste the output to solve issue no. 1, then we can fix any errors if there are any with Crane and release a new version. The other points are too broad for me right now, maybe you can narrow the scope / suggestions and then we can talk over more specific items?

Cheers!

roscopecoltran commented 6 years ago

Just publish/provide an accelerated_mount_pro.so shared library instead the whole executable for the pro version, so I can still benefit of the open source core and customize it...

That's what I meant, in the conclusion, that for you, it has no sense get my changes, but having this pro features loaded as golang plugin would be the solution for everybody.

Does it make more sense ?

package main

import (
    "fmt"
    "os"
    "plugin"
)

type Greeter interface {
    Greet()
}

func main() {
    // determine module to load
    lang := "english"
    if len(os.Args) == 2 {
        lang = os.Args[1]
    }
    var mod string
    switch lang {
    case "english":
        mod = "./eng/eng.so"
    case "chinese":
        mod = "./chi/chi.so"
    default:
        fmt.Println("don't speak that language")
        os.Exit(1)
    }

    // load module
    // 1. open the so file to load the symbols
    plug, err := plugin.Open(mod)
    if err != nil {
        fmt.Println(err)
        os.Exit(1)
    }

    // 2. look up a symbol (an exported function or variable)
    // in this case, variable Greeter
    symGreeter, err := plug.Lookup("Greeter")
    if err != nil {
        fmt.Println(err)
        os.Exit(1)
    }

    // 3. Assert that loaded symbol is of a desired type
    // in this case interface type Greeter (defined above)
    var greeter Greeter
    greeter, ok := symGreeter.(Greeter)
    if !ok {
        fmt.Println("unexpected type from module symbol")
        os.Exit(1)
    }

    // 4. use the module
    greeter.Greet()

}