"Traits" potential bug discovered

[jbarfield]

Example "traits" from our "triton datacenter" deployment:

CN 1 Traits:

{
  "hvm": "bhyve"
}

CN 2 Traits:

{
  "hvm": "kvm",
  "ssd": "true"
}

VM IMAGE & PACKAGE

The base64 Image should be allowed to provision on any hardware assuming that package traits are also properly aligned. Hence the reason for the JSON array below.

Base64 Image Traits:

{
  "hvm": [ "kvm","bhyve"]
}

The package traits mandate that this base64 image should be provisioned on a CN with the SSD trait set

Package Traits:

{
  "ssd": true
}

Actual Behavior

Attempting to provision a base64 VM with the aforementioned IMAGE and PACKAGE traits assigned causes provisioning to fail. The provisioning system chooses CN1 immediately and since the "ssd" trait is not set simply produces "failed to provision, no compute resources available".

Workaround

Within ops portal or CNAPI: Update CN1 and set {"ssd": false}, the provisioning system will now skip CN1 and provision onto CN2 successfully.

Expected behavior:

A compute node that has a nil trait should automatically be skipped during provisioning preventing every compute node in the datacenter that does not match a specific trait from requiring a manual update to its "traits" property.

[bahamat]

In the cnapi zone there's a script, /opt/smartdc/cnapi/bin/alloc-reasons.sh. Can you give me the output for that script from the failed provision attempt?

If you want to send it in private, hit me up on IRC.

[jbarfield]

I emailed it to you directly. Hope that is okay

[jbarfield]

Im closing this bug report after many years of using Triton and debugging compute resource provisioning failures.

I have found The trait failures, in my experience, were mostly caused by using non-boolean "key:values" or simply complicating node selection by assigning too many traits.

ie: problem between chair and keyboard.

Traits appear to be implemented loosely in the allocation algorithm library and appear to work best with a single boolean to simplify pattern matching as determined by operators.

Im also noting that I discovered a caveat in the source; being that boolean values are not "true booleans" in this instance and so regex matching appears to be used to determine if the trait is a boolean or not.

I discovered this during one failure or another and I am simply documenting this for visibility to others in the triton community.

Allocating resources while matching multiple traits in an array, does not really appear to work very well at all. IMO simple single trait pattern matching should be documented.

I use multiple traits purposely on CNs where operators would manually assign workloads such as on secondary headnodes for HA just to be sure customer workloads are never provisioned on HA headnodes.

Ive ultimately found that using plugins works very well for the purpose of dedicated customer hardware provisiong (adding server uuids to account UUIDs in sapi/services/cnapi/metadata) and over the years have solved other compute node allocation requirements like so:

1. Most of our compute nodes are all NVME SSD now (fastdisk) with the price of NVME being almost comparable to platter disks at this point. So I removed the "ssd: true" trait and made packages with "nossd": true instead, for the rare instance where SSD storage is undesirable. (ie: sdc-volume nodes or anywhere an instance needs a "bigdisk" package internally.

2. "shrimp": true, is the exception to #1 because we want shrimp nodes dedicated to manta nodes alone internally.

3. HVM node type traits lost relevance once smartOS supported running KVM and BHYVE instances on the same CN. So this aspect of the original bug report is entirely moot now. Also I do not think that we have a single KVM instance running any longer at this point anyway.

4. "Provisioning packages" in our cloud deployment is now much more like AWS with package-prefix matching CPU family and architecture rather than workload optimization. We chose AWS packages with dedicated NVME storage to more closely align with Triton/SmartOS local storage requirement omitting any package which was EBS optimized.

For example;

m4.* packages match broadwell/haswell CPUs and provision on CNs which match "m4": true.

The RAM/Compute/Storage optimizations simply convolute package management and maintenance. So we now group by "Native" vs "HVM" optimized workloads so that "joyent", "lx", and "bhyve" brand zones co-exist nicely on CN's, align well with customer expectations in terms of capacity planning, pricing, or SP migrations, while also eliminating end user confusion when choosing instance type and package during provisioning.

Hope this helps other Triton users in the future.