CloudAPI should allow provisioning machines with delegated datasets

[coyhile]

Unless I'm missing something, the ability to create an instance with a delegated dataset is missing when creating that instance via CloudAPI. In AdminUI, one sees a checkbox "delegate dataset". Certain technologies require datasets to function (manatee comes to mind first), and it is a less than ideal user experience that one cannot use CloudAPI to create such instances.

Similar to #13, this allows end-user tools or separation of duties. One doesn't need to be a Triton administrator to be able to manage instances with delegated dataset.

[timfoster]

We did have this working at one point, to support the tooling needed to build Triton/Manta images on systems in the public cloud.

The problem at the time, was that tooling needed to be able to zfs recv send streams into a delegated dataset, and that means that malicious or accidentally corrupt streams can cause a compute node to panic. There is the ability to create "safe" delegated datasets, where it's impossible to zfs recv data, which might be enough here, but we shelved the overall CloudAPI work because (at the time) we needed to be able to zfs recv for our use-case.

There's a bit more at https://smartos.org/bugview/TRITON-853

[headw01]

I was looking for a way to do this via Terraform the other day. Presumably, this feature would have to be added to CloudAPI before it could be added to the Terraform Provider. https://github.com/terraform-providers/terraform-provider-triton

[timfoster]

For what it's worth, the patches mentioned at the bug report were

node-triton.git:

From ebaab9add3005b80b9f43e076d6ac5da701a250b Mon Sep 17 00:00:00 2001
From: Tim Foster <tim.foster@joyent.com>
Date: Fri, 19 Oct 2018 14:54:03 +0100
Subject: [PATCH] TRITON-853 cloudapi should allow creation of instances with
 delegated datasets

---
 lib/do_instance/do_create.js | 27 +++++++++++++++++++++++++++
 1 file changed, 27 insertions(+)

diff --git a/lib/do_instance/do_create.js b/lib/do_instance/do_create.js
index cbedb35..11a8896 100644
--- a/lib/do_instance/do_create.js
+++ b/lib/do_instance/do_create.js
@@ -299,6 +299,15 @@ function do_create(subcmd, opts, args, cb) {
                     createOpts.firewall_enabled = opt.value;
                 } else if (opt.key === 'deletion_protection') {
                     createOpts.deletion_protection = opt.value;
+                } else if (opt.key === 'delegate_dataset') {
+                    var allowed_delegate_vals = ['on', 'off', 'safe'];
+                    if (allowed_delegate_vals.indexOf(
+                        opt.value.toLowerCase()) === -1) {
+                        throw new errors.UsageError(format(
+                            '--delegate-dataset value must be one of: %s',
+                            allowed_delegate_vals.join(', ')));
+                    }
+                    createOpts.delegate_dataset = opt.value;
                 }
             }

@@ -460,6 +469,24 @@ do_create.options = [
             'cannot be deleted until the protection is disabled. See ' +
             '<https://apidocs.joyent.com/cloudapi/#deletion-protection>'
     },
+    {
+        names: ['delegate-dataset'],
+        type: 'string',
+        help: 'Enable Delegated Datasets on this instance. If TYPE is set to ' +
+              '"on", no additional restrictions are imposed on the way ' +
+              'the instance can use the delegated dataset. If TYPE is set to ' +
+              '"safe", delegated datasets are enabled, but the instance is ' +
+              'prevented from receiving ZFS datasets. If set to "off" ' +
+              ' (the default), Delegated Datasets are not enabled on this ' +
+              'instance. Note that Triton CloudAPI instances must have the ' +
+              'SAPI config value ' +
+              '"experimental_cloudapi_delegate_dataset=true" ' +
+              'for create requests using this option to be allowed.',
+        helpArg: 'TYPE',
+        // mark this option hidden for now, see TRITON-853, at least until we
+        // get a full implementation of RFD 044.
+        hidden: true
+    },
     {
         names: ['volume', 'v'],
         type: 'arrayOfString',
-- 
2.21.0

sdc-cloud-api.git:

From cba939bdd68c88c80018c83b2ea5d2af2f8d5ef0 Mon Sep 17 00:00:00 2001
From: Tim Foster <tim.foster@joyent.com>
Date: Fri, 19 Oct 2018 14:54:30 +0100
Subject: [PATCH] TRITON-853 cloudapi should allow creation of instances with
 delegated datasets

---
 lib/machines.js                  | 27 +++++++++++++++++++++++++++
 sapi_manifests/cloudapi/template |  6 ++++++
 2 files changed, 33 insertions(+)

diff --git a/lib/machines.js b/lib/machines.js
index 153afd1..0754cc9 100644
--- a/lib/machines.js
+++ b/lib/machines.js
@@ -57,6 +57,8 @@ var DEFAULT_CONTAINER_BRAND = 'joyent';
 var DEFAULT_HVM_BRAND = 'kvm';
 var VALID_BRANDS = ['bhyve', 'joyent', 'joyent-minimal', 'kvm', 'lx'];

+var DELEGATE_DATASET_VALUES = ['on', 'off', 'safe'];
+
 var sprintf = util.format;

@@ -150,6 +152,7 @@ function translate(machine, req)  {
         memory: Number(machine.ram),
         disk: (Number(machine.quota) * 1024) || 0,
         deletion_protection: !!machine.indestructible_zoneroot,
+        delegate_dataset: machine.delegate_dataset || 'off',
         metadata: machine.customer_metadata || {},
         tags: machine.tags,
         credentials: credentials,
@@ -456,6 +459,30 @@ function getCreateOptions(req) {
         opts.indestructible_zoneroot = true;
     }

+    var delegateDataset = params.delegate_dataset;
+    if (delegateDataset !== undefined) {
+        if (!req.config.experimental_cloudapi_delegate_dataset) {
+            throw new InvalidArgumentError('Support for delegated ' +
+                'datasets is not enabled ' +
+                '(experimental_cloudapi_delegate_dataset in SAPI)');
+        }
+
+        if (DELEGATE_DATASET_VALUES.indexOf(delegateDataset) === -1) {
+            throw new InvalidArgumentError(sprintf(
+                'Unsupported value "%s" for "delegate_dataset". ' +
+                'Supported values are: %s',
+                delegateDataset, DELEGATE_DATASET_VALUES.join(', ')));
+        }
+        if (delegateDataset === 'on') {
+            opts.delegate_dataset = true;
+        } else if (delegateDataset === 'safe') {
+            opts.delegate_dataset = true;
+            // when vmapi supports it, we may want additional safety here,
+            // perhaps setting limit_snapshots or limit_datasets.
+            opts.limit_priv = 'default,-sys_fs_import';
+        }
+    }
+
     // Starting in version 7.3, CloudAPI supports what we call interface-
     // centric provisioning. Traditionally, CloudAPI accepted provisioning
     // here in the form of:
diff --git a/sapi_manifests/cloudapi/template b/sapi_manifests/cloudapi/template
index fc09b6d..171544d 100644
--- a/sapi_manifests/cloudapi/template
+++ b/sapi_manifests/cloudapi/template
@@ -198,6 +198,12 @@
     {{#experimental_cloudapi_automount_nfs_shared_volumes}}
     "experimental_cloudapi_automount_nfs_shared_volumes": {{{experimental_cloudapi_automount_nfs_shared_volumes}}},
     {{/experimental_cloudapi_automount_nfs_shared_volumes}}
+    {{^experimental_cloudapi_delegate_dataset}}
+    "experimental_cloudapi_delegate_dataset": false,
+    {{/experimental_cloudapi_delegate_dataset}}
+    {{#experimental_cloudapi_delegate_dataset}}
+    "experimental_cloudapi_delegate_dataset": {{{experimental_cloudapi_delegate_dataset}}},
+    {{/experimental_cloudapi_delegate_dataset}}
     {{^CLOUDAPI_TEST_MODE}}
     "test": false,
     {{/CLOUDAPI_TEST_MODE}}
-- 
2.21.0

[coyhile]

It would seem that my use case (my own manatee for user database workloads) would require the "unsafe" version, which I guess should be behind "thou shalt be admin" walls for reason of being able to panic the CN if the dataset were bad.

My other workloads are just data persistence (DNS and my ISC dhcpd instance for the OOB and other non-triton-managed pools.)

[twhiteman]

I'm not sure people would require a delegated dataset for data persistence - instead one could use a Triton volume for that task.

The beauty of a delegated dataset is in being able to use zfs commands (snapshot/send/recv) on the filesystem, and being able to reprovision (update container image) without wiping out data.

[headw01]

I was interested in the delegated dataset to control the recordsize settings for a Percona/MySQL server for the data and log.

[coyhile]

I’m unfamiliar with the specifics of what you mean there, Todd. Please feel free to respond offline if we needn’t clutter the discussion here. Short of fun with databases, my use case was making application data survive a reprovision.

For the database workload, yeah, being able to use zfs commands natively is a hard requirement. Or for a NAS.

On May 12, 2020, at 3:12 PM, Todd Whiteman notifications@github.com wrote:

I'm not sure people would require a delegated dataset (most cases) for data persistence - instead one could use a Triton volume for that task.

The beauty of a delegated dataset is in being able to use zfs commands (snapshot/send/recv) on the filesystem.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or unsubscribe.

-- Coy Hile coy.hile@coyhile.com

[pannon]

I was looking for a way to do this via Terraform the other day. Presumably, this feature would have to be added to CloudAPI before it could be added to the Terraform Provider. https://github.com/terraform-providers/terraform-provider-triton

Same for our use case too. We have cases where we would want to provision instances with delegated datasets via Terraform. Also would be good to have the reprovision feature, which is already present in the adminUI.

[jasonbking]

We're unlikely to add reprovision support to CloudAPI (at least in its current form). The problem is that if it fails, it often does so in a manner that requires a fair amount of manual operator intervention on the CN to clean up.

[twhiteman]

A Triton volume (NFS) allows reprovision of instances (well, you delete the instance and create a new one, mounted back to the same volume).

Here are some sdcadm commands that will enable volapi and enabling volume access from containers:

sdcadm post-setup volapi --help
sdcadm experimental nfs-volumes --help
triton volume --help

Initially volumes were created for Docker, c.f. https://apidocs.joyent.com/docker/features/volumes but once you have a volume created, you can also just manually mount it via NFS (for SmartOS, Bhyve, ...).

CloudAPI also supports passing in volumes as a part of CreateMachine (not for HVM though): https://apidocs.joyent.com/cloudapi/#CreateMachine and this (instance/volume creation) will be flowing into Terraform soon: https://github.com/terraform-providers/terraform-provider-triton/issues/131

Also, here's the RFD for the broad overview of volapi: https://github.com/joyent/rfd/blob/master/rfd/0026/README.md

[pannon]

volapi is great for files which are infrequently written or accessed.

The delegated datasets topic/need mostly comes up where databases are deployed, both single instance and distributed ones.

The feature sought after for these use cases is ZFS snapshots with send/recv and custom ZFS record size.

[headw01]

It looks like this PR will solve this (for cloudapi): https://github.com/joyent/sdc-cloudapi/pull/64

Terraform would need some updates https://github.com/terraform-providers/terraform-provider-triton

[twhiteman]

The delegate_dataset was added to CloudAPI in #64 - closing this as a duplicate of that work.