fix: clean and debug the provider

QuentinBrosse commented 4 years ago

Work:

Introduce go modules
Upgrade dependencies to fix issues (mainly related to volume sizes: #100)
Fix the sudo command not found issue (firstly discovered here).

Manual tests

Type	Image local ID	Image label	Result
`DEV1-S`	`f974feac-abae-4365-b988-8ec7d1cec10d`	`ubuntu_bionic`	✅
`DEV1-L`	`b5a754d1-8262-47d2-acb2-22739295bb68`	`ubuntu_xenial`	✅
`DEV1-S`	`9a53e825-19f9-4e1f-b841-dd921e4e1b39`	`debian_stretch`	✅

Addressed issues

Volume sizes

Fix #100

Each instance type has a minimum and maximum volume total size to respect, you can find them in instance server endpoint in the volumes_constraint key. For all new instance ranges (eg. DEV, GP ) volumes_constraint.min == volumes_constraint.max. For better stability, we recommend using these new instance types.

With this fix:

By default (if you don't use --scaleway-volumes) the maximum root volume size will be used (keep in mind that volumes_constraint.min == volumes_constraint.max for all new instance types)
If you use the --scaleway-volumes flag, the root volume will be guessed and will have a maximum size of 50G

Examples

```bash # DEV1-S with ubuntu-bionic # For DEV1-S, image volumes constraints are: min: 20G, max 20G. # This command will create 1 volume of 20G. docker-machine create -d scaleway-dev --scaleway-commercial-type DEV1-S --scaleway-image "f974feac-abae-4365-b988-8ec7d1cec10d" ubuntu-bionic # GP1-S with debian-buster # For GP1-S, image volumes constraints are: min: 300G, max 300G. # This command will create 1 volume of 300G. docker-machine create -d scaleway-dev --scaleway-commercial-type GP1-S --scaleway-image "b3042271-d2b1-4f87-b407-aedd3bbd1663" debian-buster # GP1-S with debian-buster and additional volumes # For GP1-S, image volumes constraints are: min: 300G, max 300G. # I want two volumes of 100G each and one of 50G for a total of 300G (do not forget the guessed root volume of 50G). # # This command will throw an error: "The total size of local-volume(s) of instances GP1-S must be equal to 300G" # Currently: # 1 root volume: 50G ("guessed and will have a maximum size of 50G") # + 2 100G additionnal volumes: 200G # + 1 50G additionnal volumes: 50G # --------------------------------- # = 300G => ⚠️ 300G is required for a GP1-S ! docker-machine create -d scaleway-dev --scaleway-commercial-type GP1-S --scaleway-image "b3042271-d2b1-4f87-b407-aedd3bbd1663" --scaleway-volumes="200G 100G 100G" debian-buster # GP1-S with debian-buster and additional volumes # For GP1-S, image volumes constraints are: min: 300G, max 300G. # I want two volumes of 100G each. # # This command will throw an error: "The total size of local-volume(s) of instances GP1-S must be equal to 300G" # Currently: # 1 root volume: 50G ("guessed and will have a maximum size of 50G") # + 2 100G additionnal volumes: 200G # --------------------------------- # = 250G => ⚠️ 300G is required for a GP1-S ! docker-machine create -d scaleway-dev --scaleway-commercial-type GP1-S --scaleway-image "b3042271-d2b1-4f87-b407-aedd3bbd1663" --scaleway-volumes="100G 100G" debian-buster # GP1-S with debian-buster and additional volumes # For GP1-S, image volumes constraints are: min: 300G, max 300G. # I want three volumes of 100G each. # # This command will throw an error: "The total size of local-volume(s) of instances GP1-S must be equal to 300G" # Currently: # 1 root volume: 50G ("guessed and will have a maximum size of 50G") # + 3 100G additionnal volumes: 300G # --------------------------------- # = 350G => ⚠️ 300G is required for a GP1-S ! docker-machine create -d scaleway-dev --scaleway-commercial-type GP1-S --scaleway-image "b3042271-d2b1-4f87-b407-aedd3bbd1663" --scaleway-volumes="200G 100G 100G" debian-buster ```

You can find more details here.

Command sudo not found during provisioning

Fix #84

This issue was described here. docker-machine execute commands into instances via the sudo command (and the root user). Some Linux distributions do not have sudo installed by default anymore. This fix adds a cloud-init configuration to install sudo and add root to sudoers if needed.

tmoreira2020 commented 4 years ago

@QuentinBrosse thanks for your work on this during this period of the year 🎄. Let me know if I can help you on testing.

Thanks

QuentinBrosse commented 4 years ago

Hi @tmoreira2020, You're welcome! :)

Can you test this branch and tell me if everything is ok for you?

Here is the process (the only dependency is Golang):

# Clone the repo and switch to this PR
$ git clone git@github.com:scaleway/docker-machine-driver-scaleway.git
$ git fetch upstream refs/pull/107/head:debug
$ git checkout debug

# Build the driver under the name `scaleway-dev`
$ go build -i -o $GOPATH/docker-machine-driver-scaleway-dev .

# Use this dev driver.
$ docker-machine create -d scaleway-dev ...

Thanks for your contribution! 🎄

tmoreira2020 commented 4 years ago

Hey @QuentinBrosse, thanks for the fix, unfortunately it didn't work for me. I got the following exception Error creating machine: Error in driver during machine creation: StatusCode: 400, Type: invalid_request_error, APIMessage: Volume size must be bigger than or equal to its base

The list of parameters are docker-machine create -d scaleway-dev --scaleway-token=BLABLA --scaleway-organization=BLABLA --scaleway-name="www-blabla-com" --scaleway-commercial-type "DEV1-S" www-blabla-com

Did I miss something?

tmoreira2020 commented 4 years ago

Hey @QuentinBrosse, did I miss something in the test? Can I run the test with a debug flag?

QuentinBrosse commented 4 years ago

Hi @tmoreira2020 and sorry for the delayed answer.

By default, the driver was using a relatively old image: 265b32a3. I changed it for ubuntu-bionic. Now, this test works for me:

$ docker-machine -D create -d scaleway-dev --scaleway-commercial-type="DEV1-S" --scaleway-name="dm-test-1" dm-test-1

Note that you can use --scaleway-image to choose a specific image, by label or UUID. Can you make your tests on your side?

gregfr commented 4 years ago

My first steps with this branch.

docker-machine -D create -d scaleway-dev --scaleway-commercial-type="DEV1-S" --scaleway-ip="ipref" --scaleway-name="test01" test01

Docker Machine Version:  0.16.2, build bd45ab1
Found binary path at /path/go/docker-machine-driver-scaleway-dev
Launching plugin server for driver scaleway-dev
Plugin server listening at address 127.0.0.1:41101
...
(test01) Creating server...
(test01) DBG |   IMAGEID           FROM                          NAME                    ZONE                ARCH
(test01) DBG | - 6bd566a1          image:ubuntu-bionic-ml-9-2    Ubuntu Bionic ML 9.2    par1                x86_64
(test01) DBG | - f974feac          image:ubuntu-bionic           Ubuntu Bionic           par1                x86_64
(test01) DBG | - 5f96d290          image:ubuntu-bionic-ml-10-1   Ubuntu Bionic ML 10.1   par1                x86_64
Error creating machine: Error in driver during machine creation: Too many candidates for ubuntu_bionic (3)
notifying bugsnag: [Error creating machine: Error in driver during machine creation: Too many candidates for ubuntu_bionic (3)]

Same command with --scaleway-image="5f96d290" :

(test01) Creating server...
Error creating machine: Error in driver during machine creation: StatusCode: 400, Type: invalid_request_error, APIMessage: Volume size must be bigger than or equal to its base
notifying bugsnag: [Error creating machine: Error in driver during machine creation: StatusCode: 400, Type: invalid_request_error, APIMessage: Volume size must be bigger than or equal to its base]

gregfr commented 4 years ago

I changed defaultImage = "ubuntu_bionic" to defaultImage = "ubuntu-bionic" and the "too many candidates error" doesn't show anymore; the instance is up and running with the default 20G disk

QuentinBrosse commented 4 years ago

@gregfr yep, I need to revert this commit.

tmoreira2020 commented 4 years ago

@QuentinBrosse I was able to create new hosts with the fix provided by you. Thanks! Can you release it as a fix?

scaleway / docker-machine-driver-scaleway