RFE: Allow every field to reference a remote resource

eriksjolund commented 4 years ago

I noticed at https://github.com/coreos/fcct/blob/master/docs/configuration-v1_1.md that systemd -> contents does not have variant for including the contents from a local file. For me it would have been convenient to have that feature.

Why not generalize the data input, so that strings and bytearrays always can be included from URLs, local files or inline? Something like two new "datatypes": general_text_data and general_binary_data

Just a sketch:

_general_textdata (object): the config that will replace the current.
- source (string): the URL of the config. Supported schemes are http, https, s3, tftp, and [data][rfc2397]. Note: When using http, it is advisable to use the verification option to ensure the contents haven't been modified. Mutually exclusive with inline and local.
- inline (string): the contents of the config. Mutually exclusive with source and local.
- local (string): a local path to the contents of the config, relative to the directory specified by the --files-dir command-line argument. Mutually exclusive with source and inline.
- _httpheaders (list of objects): a list of HTTP headers to be added to the request. Available for http and https source schemes only.
  - name (string): the header name.
  - value (string): the header contents.
- verification (object): options related to the verification of the config.
  - hash (string): the hash of the config, in the form <type>-<value> where type is either sha512 or sha256.

If it would be possible to reference the string contents of one of

passwd -> users -> ssh_authorized_keys

with a general_text_data, you could start referencing your public ssh key from Github, as they are downloadable like this

wget https://github.com/eriksjolund.keys

LorbusChris commented 4 years ago

I think this is reasonable. Something similar was already done for the Ignition types: https://github.com/coreos/ignition/pull/953

bgilbert commented 4 years ago

I suspect that allowing arbitrary fields to come from URLs or local files would introduce more complexity to the spec than the feature is worth.

For your original use case, see https://github.com/coreos/ignition/pull/986.

eriksjolund commented 4 years ago

At first I thought the URL:s were downloaded by fcct (at "transpile-time" so to say) but now I see the URL:s are given to the Ignition file. So this Github issue probably belongs here https://github.com/coreos/ignition instead. (Should I close this one and open a Github issue there instead?)

Regarding introducing more complexity: The change would be like changing a datatype, i.e. from string to general_text_data. In the source code, shouldn't that just be a small change?

I understand that this RFE would introduce more challenges in how to easily present the file format standard in the documentation for the users.

If many of the string would be replaced in the file https://github.com/coreos/fcct/blob/master/docs/configuration-v1_1.md with a _general_textdata sub tree, that file would become very big.

One way to solve it would be to define _general_textdata seperately outside the normal tree structure. Another way to solve it could be to generate out some HTML documentation where the file format specification could be browsed by a web browser. Then it would be easier to represent a "sub tree" data type.

Another idea: The fcct could be given a new command-line argument: --download-content-and-embed-into-ignition-file so that such _general_textdata -> source (with an URL) could be converted to a _general_textdata -> inline (where the string content is embedded into the Ignition file)

That could be useful when a Fedora CoreOS installation is performed without internet access.

bgilbert commented 4 years ago

Okay, I've moved the issue over to Ignition.

We already have a reusable data type for this, Resource, which you can see in action e.g. here. Expanding every field to use Resource would be a major breaking change and would make configs much more verbose. If you have a set of use cases where this would be really helpful, though, please do share them here.

bgilbert commented 4 years ago

You can also use the files section to write services and SSH keys:

variant: fcos
version: 1.1.0
storage:
  directories:
    - path: /home/core/.ssh
      user:
        name: core
      group:
        name: core
      mode: 0700
    - path: /home/core/.ssh/authorized_keys.d
      user:
        name: core
      group:
        name: core
      mode: 0700
  files:
    - path: /home/core/.ssh/authorized_keys.d/github
      user:
        name: core
      group:
        name: core
      mode: 0600
      contents:
        source: https://github.com/bgilbert.keys
    - path: /etc/systemd/system/example.service
      contents:
        source: https://example.com/example.service
systemd:
  units:
    - name: example.service
      enabled: true

eriksjolund commented 4 years ago

List of use cases

Use case 1. Including a systemd unit file from the file system

It seems to be addressed in https://github.com/coreos/ignition/pull/986 that @bgilbert mentioned in https://github.com/coreos/ignition/issues/1097#issuecomment-688981237

This use case can also be worked around by using storage -> files -> contents -> local

I did that in an experiment of mine https://github.com/Clinical-Genomics/scout/blob/systemd_services_podman/systemd/scout.fcc

storage:
  files:
    - path: /home/core/.config/systemd/user/scout-pod.service
      user:
        name: core
      group:
        name: core
      mode: 0644
      contents:
        local: scout-pod.service

(although that example uses systemd user units)

Use case 2. Being able to specify an SSH public key as an URL

Being able to specify an URL, it would be easier to identify the person being granted access just by looking at an Ignition/FCC file. For instance URL:s could be used in the form of https://github.com/<githubusername>.keys

@bgilbert Yes, your suggestion in https://github.com/coreos/ignition/issues/1097#issuecomment-700164654 to use storage -> files -> contents -> source would solve that.

Some more thoughts

My first thought by creating this RFE was that over time there will always pop up RFE:s requesting conversion of a string datatype into the Resource datatype in the spec. For every such change the spec would break. To avoid that situation, why not right away change all occurrences of the string datatype into the general Resource datatype?

I've been thinking a bit more about this RFE. As there are workarounds, I don't have such a big need for it. If you want you could close this issue, or just keep it open to gather user feedback.

coreos / ignition