Open ggilmore opened 4 years ago
Octal values are an official part of the YAML spec
That's a reference for YAML 1.1, but yaml-to-dhall
's YAML parser (primarily?) tries to comply with YAML 1.2, where octal notation apparently requires a 0o
prefix, not just 0
.
Good to know, but apparently kubectl
(or the Kubernetes API server) supports the YAML 1.1 syntax. The official K8s documentation also uses the 0
prefix octal notation: https://kubernetes.io/docs/concepts/configuration/secret/#secret-files-permissions
The nature of these silent transformations also makes it difficult to detect / workaround
Based on the discussion so far it sounds like this can be split into two separate questions:
Should Dhall have language support for octal literals?
This is more of a question for the dhall-lang
repository, but we can still briefly share our thoughts here
Should yaml-to-dhall
support YAML 1.1 octal literal syntax
This question is appropriate for this repository
For the former question, I would be fine with standardizing support for octal literals for the language
For the latter question, I think it depends on two things:
Does kubectl
support the newer 0o…
octal syntax? (It sounds like it doesn't, but I just want to confirm)
Does kubectl
support non-octal numeric literals for permissions? (Based on a quick Google search it sounds like it does)
In general, adding support for YAML 1.1 octal literals might be problematic for us since the upstream HsYAML
package that yaml-to-dhall
is based appears to only supports YAML 1.2, which is why I'm exploring workarounds.
Does kubectl support the newer 0o… octal syntax? (It sounds like it doesn't, but I just want to confirm)
I think it does (at least from my brief test with the following versions):
> kubectl version
Client Version: version.Info{Major:"1", Minor:"16+", GitVersion:"v1.16.6-beta.0", GitCommit:"e7f962ba86f4ce7033828210ca3556393c377bcc", GitTreeState:"clean", BuildDate:"2020-01-15T08:26:26Z", GoVersion:"go1.13.5", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"14+", GitVersion:"v1.14.10-gke.42", GitCommit:"42bef28c2031a74fc68840fce56834ff7ea08518", GitTreeState:"clean", BuildDate:"2020-06-02T16:07:00Z", GoVersion:"go1.12.12b4", Compiler:"gc", Platform:"linux/amd64"}
I don't know if support for this was always there or if this was a recent change (or even if it varies between cloud providers).
Does kubectl support non-octal numeric literals for permissions? (Based on a quick Google search it sounds like it does)
Yes it does (you're able to specify defaultMode: 511
directly and it works as you expect).
I'd still like to encourage support for YAML 1.1 octal literals because that's the only style that I've seen "in the wild" for Kubernetes (probably because that's what's used in the official docs). I'm going to guess that a fair amount of people (maybe most?) use the this style when writing their manifests, so they're likely to be bitten by this when they want to use yaml-to-dhall
.
As a data point from the https://github.com/helm/charts repository:
I found 60+
examples of manifests using the YAML 1.1 octal notation style (0...
): https://sourcegraph.com/search?q=repo:%5Egithub%5C.com/helm/charts%24++%28mode%7CdefaultMode%29:+0%5Cd%2B+lang:yaml+++&patternType=regexp
I couldn't find any examples of manifests using the YAML 1.2 octal notation style (0o...
): https://sourcegraph.com/search?q=repo:%5Egithub%5C.com/helm/charts%24++%28mode%7CdefaultMode%29:+0o%5Cd%2B+lang:yaml+++&patternType=regexp
- Should Dhall have language support for octal literals?
This question seems only tangentially related to this issue to me?! I wouldn't mind if support for octal literals would be added though.
- Should
yaml-to-dhall
support YAML 1.1 octal literal syntax
This seems tricky, since IIUC these literals have different meaning in YAML 1.1 vs. 1.2. For example 010
is 8
with YAML 1.1, but 10
with 1.2.
It would be nice if yaml-to-dhall
would have a --yaml-1.1
option or something like that, I guess.
Or maybe there's a YAML-1.1 to YAML-1.2 converter that we could recommend for use with yaml-to-dhall
?!
@sjakobi: I like the idea of a YAML 1.1 to YAML 1.2 converter
The reason I suggested Dhall support for octal literal syntax was to ensure that the octal literal would be preserved when converting back from Dhall to YAML
Or maybe there's a YAML-1.1 to YAML-1.2 converter that we could recommend for use with
yaml-to-dhall
?!
I have created a SO question about this: https://stackoverflow.com/questions/63472830/how-do-i-convert-yaml-1-1-to-yaml-1-2
While doing so, I remembered that YAML 1.2 is a superset of JSON.
So, to prepare a YAML 1.1 document for consumption by yaml-to-dhall
, you can convert it to JSON, for example with this yml2json
utility (which, mind you, I have only tested once, not properly vetted). Instead of yaml-to-dhall
, you can also use json-to-dhall
then.
I have opened https://github.com/dhall-lang/dhall-lang/issues/1058 to discuss language support for octal numerals.
yaml-to-dhall
seems to simply strip leading0
's from octal values when converting them to dhall. This leads to problems when we need to render the dhall back to yaml again.Consider the following Kubernetes Deployment yaml
Note the
defaultMode
of the configMap,0777
(octal)Converting it to dhall results in the following output
Note that
defaultMode
of the configMap is now777
(natural) in the dhall outputConverting that dhall back into kubernetes yaml:
Note that the configMap's defaultMode has changed from
0777
(octal) in the original yaml to777
(decimal) in the new yaml.And indeed, trying to apply this yaml into a cluster results in an error
Octal values are an official part of the YAML spec, and it'd be nice if Dhall supported this too (or at least had some sort of visible warning). I am not sure what the fix should be (an official octal number type,
yaml-to-dhall
converting octal values to the equivalent decimal values, or something else).