dlech commented 7 years ago

By one-off, I mean changing some small thing about the image, like the hostname or a static ip address. One use case is for classrooms that may be using embedded devices and want to set them up without having to boot and log into each device (comes from here).

I'm guessing to make this work, the image file would have to have a FAT partition since that is pretty much the only partition type that works on all OSes without additional software. A simple solution might be to have the image file could have extended meta data mount this partition and open a configuration file in the OS default text editor when the image burn is finished instead of ejecting the drive. Or even better, a simple text editor built into Etcher so that Etcher can eject the drive when editing is finished.

For example, in our ev3dev OS for Raspberry Pi, users always need to edit config.txt to select which add-on board they are using since there are multiple add-on boards and they cannot be automatically detected.

jviotti commented 7 years ago

Hi @dlech ,

Coincidentally, this is a feature we've been internally discussing (in Gitter, so not really internally after all) as part of our "Extended Archives" feature (see the WIP manifest here: https://github.com/resin-io/etcher/pull/707).

We've implemented configuration support in our Resin CLI (https://github.com/resin-io/resin-cli) already, but we're looking to extract this functionality and make it generic in Etcher.

This is what we currently have

resin-image-fs: A very handy and reliable module to perform file operations in image's FAT partitions (like move, copy, replace, etc). It currently only supports FAT, but nothing prevents us from adding support for other file systems as long as its feasible to do so in a cross-platform fashion. We should also remove the "Resin" piece of the module, as it suggests the module is Resin.io-only.
A way to define "questions" to ask the user, for example:

options: [
  message: 'Processor'
  name: 'processorType'
  type: 'list'
  choices: [ 'Z7010', 'Z7020' ]
,
  message: 'Coprocessor cores'
  name: 'coprocessorCore'
  type: 'list'
  choices: [ '16', '64' ]
]

The answers from the options presented to the user are passed to resin-device-operations, a module used by the Resin CLI which accepts a set of configuration "commands" and executes them on an image. Check the following commands as an example:

[
    command: 'copy'
    from:
        partition:
            primary: 1
        path: '/bitstreams/parallella_e16_headless_gpiose_7010.bit.bin'
    to:
        partition:
            primary: 1
        path: '/parallella.bit.bin'
    when:
        coprocessorCore: '16'
        processorType: 'Z7010'
,
    command: 'copy'
    from:
        partition:
            primary: 1
        path: '/bistreams/parallella_e16_headless_gpiose_7020.bit.bin'
    to:
        partition:
            primary: 1
        path: '/parallella.bit.bin'
    when:
        coprocessorCore: '16'
        processorType: 'Z7020'
]

The actual commands make use of resin-image-fs.

This is what we're missing

Polishing the way options and commands are declared. We want to ditch the concept of "commands" and use a more "high-level" way of declaring what a file goes somewhere, etc; since the current approach is very imperative.
Add support for common "schema-like" languages, like JSON, YAML, INI, etc; and allow users to define that a certain value goes in a certain "key" of a certain file. We manage the rest.
Maybe templating support?

In any case, this is all taking shape and we don't have anything clear at the moment, but if you're interested, we're discussing a lot of this stuff in Gitter, so I suggest you to keep an eye there.

Given that you opened this issue, lets make this the home of more organized discussion about this topic.

ensonic commented 7 years ago

+1 Another usecase for such a feature would be to setup wifi. On many devices entering the wifi settings is hard. I don't want to complicate the feature request, but I wonder if it should be considered to be able to re-run the customization steps on a preflashed image?

alexandrosm commented 7 years ago

@ensonic you'll be glad to hear that @jviotti is going to be spending the next two weeks focusing on this precise problem. Wifi config is certainly a big use case.

lurch commented 7 years ago

I dunno how suitable it is for Etcher, but you might want to have a look at http://www.pibakery.org/ ( https://github.com/davidferguson/pibakery - also based around NodeJS and Electron I believe) which was created to solve issues very similar to those being discussed in this thread :-) https://www.raspberrypi.org/blog/pibakery/ ping @davidferguson

jviotti commented 7 years ago

Hi @lurch ,

We've indeed looked at PiBakery, which is in fact based on Etcher (it uses Etcher's backend modules, and our elevation system, among other stuff).

While PiBakery is very Raspberry Pi specific, we're creating a completely generic system that not only allows users to configure, but to re-configure as well (read config settings back from the device).

We are currently working on this system, which is the reason why I'm not very active anymore (I'll get back to normal speed once the foundations are done), and hopefully everything will make more sense afterwards (we'll publish specs, etc).

lurch commented 7 years ago

We've indeed looked at PiBakery, which is in fact based on Etcher (it uses Etcher's backend modules, and our elevation system, among other stuff).

Oooh, that's interesting! In that case I wonder why @davidferguson uses his own CommandLineDiskImager program (forked from Win32DiskImager), rather than using the image-writing code from Etcher?

we're creating a completely generic system that not only allows users to configure, but to re-configure as well (read config settings back from the device).

Sounds cool. If you've not already done so, you might want to have a look at Webmin and LuCI for ideas / inspiration / sample-code :-)

davidferguson commented 7 years ago

In that case I wonder why @davidferguson uses his own CommandLineDiskImager program (forked from Win32DiskImager), rather than using the image-writing code from Etcher?

I use CommandLineDiskImager for Windows because I was getting some really weird errors when I tried to use resin-image-write on Windows (can't remember the exact error now, but it appeared in resin-image-write, etcher-cli and also the main Etcher app, and on multiple Windows computers) so I decided to use Etcher on Mac (and Linux) and a version of Win32DiskImager on Windows.

I'll try resin-image-write again on Windows again at some point, but the error was tricky to reproduce and seemed to appear at random times, so for the moment I'll probably stick with CommandLineDiskImager.

alexandrosm commented 7 years ago

@davidferguson you're probably referring to the issue narrated in this post -- https://resin.io/blog/the-perils-of-writing-disk-images-on-windows/ , which have now been resolved in the latest version of etcher-image-write. The resin-image-write module is deprecated and we generally recommend people use etcher-image-write instead, which should be a drop-in replacement.

davidferguson commented 7 years ago

@alexandrosm That's the error - thanks for linking that! Great news that the issue is fixed - I'll work on using etcher-image-write in the next release of PiBakery.

alexandrosm commented 7 years ago

In the spirit of opening up the development of Etcher further, I'll paste here a high-level document of the discussions we've had with @jviotti on the configuration pipeline. It's super abstract and ambitious, but we think we've actually cracked it. Juanchi has been on this for the last week or so, and by the end of this one he should have a version zero for another resin.io project, which should be easy to transfer to Etcher shortly thereafter.

Device (re)configuration for Etcher

Introduction

Users that write OS images with Etcher, often want to configure these images with Etcher as well, before they boot the devices that these images are intended for. In addition, they want to be able to reconfigure these images later, either because they may have made a mistake, or because the settings are no longer correct and the device does not offer a way to change its configuration, very common with headless IoT devices.

Addressing this in a generic fashion is an extremely challenging problem, as the configuration information is often stored on the device in an unpredictable way, split between several files, each of which is of different format. To make things worse, these configuration files often do not express the user’s intent directly, but as a composite of various settings, which in aggregate express the user’s initial high-level input. Solving the configuration problem requires Etcher to have the ability to take user-level configuration and convert it into machine-level configuration, which is complex but possible. However solving the re-configuration problem optimally requires being able to reverse one’s steps, and convert machine-level configuration into user-level configuration, which is considerably harder.

If one is able to dictate the format of the configuration files on the device, things are fairly simple, as we can push the complexity to the device runtime. That’s how we do it on resin-cli for resin-os devices. However solving the problem in a general fashion requires being able to express as many different configuration approaches as possible.

This document describes an approach that excludes a fairly small amount of potential approaches to configuration, while at the same time being very economical with the metadata required to encode how to mediate between the user and the machine.

The four stages

The cornerstone of the approach is the definition of four distinct stages that the information goes through.

The first stage is the “user interface” stage, which is the stage in which the user interacts with the information. This may take the form of a web form, or a command line wizard, but the key is that it optimises for communicating with the end user and empowering them to express their intent, while avoiding errors, whether those are by misunderstanding or accident.

The second stage is the “dry json” stage. In this stage, the information is expressed as a JSON structure that is optimised for avoidance of repetition and internal consistency. This stage very closely resembles the visual stage, but is stripped of user interface considerations and reduced to a format that is easy for machines to process and transmit.

The next stage is the “wet json” stage. In this stage, the data is still encoded as a JSON structure, but instead of optimising for clarity and consistency, it optimises for matching the structure with which it is going to be stored on the device.

The final stage is the “file” stage, during which the information is stored on the device itself, potentially in different files, and potentially with those files having different serialisation formats between them, and even several formats co-existing nested within the same file. This stage is optimised for consumption by a running operating system, and while users are sometimes told to edit these files directly, the process is often quite technical and error prone.

In order for our approach to be fully described, we must examine each pair of stages, and how information is converted from the one to the other.

Transformation pipeline

In order to define a fully two-way transformation, we need to define two-way transitions between each pair of stages. Our pipeline should look something like this:

user interface <--(transition)--> dry json <--(transition)--> wet json <--(transition)--> files

Stage transitions

From user interface to dry json

The first transition is pretty straightforward, if only because this problem has been solved well by various existing systems. One system that is a good example, regardless of whether we decide to use it ultimately, is http://schemaform.io/. It can generate HTML forms from JSON Schema files. Those forms can be used to extract information from a user, which they convert to JSON. In reverse, given the data in JSON form, they can use it to populate a form, therefore enabling full two-way transitions. We may not use this transition as-is as we want more datatypes than just what is available in json schema, but the main approach will not diverge much.

From dry to wet json

This transition is the hardest one, and the one that is the least obvious. A single-field entry in the dry json form can become several fields in the wet json form. Further, those several fields can, in fact, operate as templates on which other fields from the dry json structure must be mapped.

While it’s not hard to think how to go from dry to wet json with the use of a templating system such as handlebars, it’s much harder to think how to reverse the process without having explicit code with which to do the transformation. The extremely interesting finding here is a library called jsonexp, which can take patterns as inputs, and not only match them on an arbitrary JSON file but in fact, extract undefined subsegments of those patterns and return them to the caller. The exciting piece of information is that the “patterns” that jsonexp would need to convert wet json to dry json are the same that a templating engine such as handlebars would need to convert dry json to wet json, if one ignores small syntactic differences which are easily overcome. As such, these patterns can be used to concisely express the relationship between dry and wet json and be used by different tools to transition data in either direction.

Upon further examination, we’ve decided not to use jsonexp directly, as we would want to stay within the confines of json tooling, but we will be following a fairly similar approach.

From wet json to files

The final transition is that between wet json and configuration files. Since wet json stores information in the same way as the files themselves, all that is needed is to express which files are to act as stores of data, and how the data is to be serialised (selected from a set of understood configuration file formats). The information intended for this stage should be expressed in a way that allows for the transition to be done in the reverse direction as well.

Additional notes

A note on manual file alterations

For this entire system to work, we must account for the case where users alter the configuration files on a device, and the resulting files are outside the range of the patterns. In that case, the system must partially break the abstraction and fall back to allowing the user to manually edit the unmatchable segment. Ideally, the user will be able to restore the segment to a default value, and also unaffected segments of the configuration file (or other configuration files) will still surface in structured inputs.

As a corollary functionality, we must consider the case when the user wants to manually edit the target files, perhaps to create a configuration which is not available by manipulating the high-level configuration. This too must be made possible by including a “custom value” alternative on the appropriate input.

A note on reconfiguration

For Etcher to be able to reconfigure a device, it must have access to the transformation description files. As such, those should be written to a partition of the flash drive as defined by the extended archive, so that they can be re-discovered by Etcher when it discovers the drive in the future.

A note on raw files

There exist situations in which we want to transfer a raw file from the UI all the way to the disk. For instance, if a picture of a logo, or some binary executable such as a kernel module or .dtb is needed. These assets can either be kept as they are from the visual to the file stage, or converted to ASCII data which is encoded in the transitional dry and wet json structures.

A note on “filesets”

When a variable number of files have to be created, which correspond to the instances of a multi-member entity (think network connections a device should try), the concept of mapping to a single file may break down. It is possible however to think of a single “fileset” entity, each member of which is serialised in a given way (e.g. yaml). We can then maintain the mechanisms that write to a given entity, which in this case is a fileset, but this is reflected on disk as a collection of files, achieving our goal. Given the right selection rule (e.g. all the files in a directory, or all filenames matching a certain pattern), the process can be reversed and the fileset generated based on the raw files.

konmouz commented 7 years ago

After a conversation with @alexandrosm regarding how (and if) image configuration can fit to v01, I have some exploration. The idea is to fit image configuration within 'Image Info modal'. This window can appear by user's request (when clicking the image name or the 'i' icon on the main UI window), or automatically when a new image is selected (only when necessary).

Pinging @jviotti & @taahirisaacs, to initiate this discussion. *Note: The modal is scrollable and current content is a rough estimation (would change according to the selected image).

Current 'Image Information Modal':

Potential Update:

jviotti commented 7 years ago

I like where this is going. I think we should definitely show this by default, and even have some sane way UX-wise to actually enforce it, since there might be images that make no sense to leave un-configured.

Re-using the image details for this is a clever idea, however I wonder how obvious it is to users that clicking an informative bubble on the first step will trigger a configuration dialog. Maybe we can have another bubble with say a "cog" icon to open a different configuration modal?

Another idea: could we rename the "Flash!" button label to something like "Configure & Flash!" if we detect the image can be configured? In that case, "Looks Good" in the configuration modal would become "Flash".

I'd also help to define the different type of controls that could take place in the modal. By seeing the current sketch, I have several questions from a technical point of view:

The "Persistent Logs" feature contains a danger icon. How do we present that in the JSON schema?
Several things are presented in a grid during the "Ethernet" configuration phase. How do we determine if we should show in a grid, one below the other, etc? Do we use certain heuristics?

jviotti commented 7 years ago

The possible controls here are (let me know if I'm missing any):

Number input
Text input
Password input
Text enumeration
Number enumeration
Boolean input (a checkbox)

WasabiFan commented 7 years ago

Re-using the image details for this is a clever idea, however I wonder how obvious it is to users that clicking an informative bubble on the first step will trigger a configuration dialog. Maybe we can have another bubble with say a "cog" icon to open a different configuration modal?

This is my main worry. Putting all of this configuration in a "details" pane just feels wrong... it would be much more natural to have an obvious way of accessing this configuration. My initial thinking is along the lines of a cog icon as you said, but that feels too subtle. Maybe it should pop up automatically when an image is selected.

It would also be nice if there were a way to see a quick summary of the chosen options; maybe everything that is non-default would be displayed in the image details along with the path.

alexandrosm commented 7 years ago

I also think it should pop up automatically for suitable images, and i think @konmouz kinda said that too, but not too clearly.

I agree with others that overloading the info is probably not awesome, and the resulting modal looks packed. A separate modal is probably the way to go.

--

Alexandros Marinos

Founder & CEO, Resin.io

+1 206-637-5498

@alexandrosm

On Mon, Dec 12, 2016 at 8:40 PM, Wasabi Fan notifications@github.com wrote:

Re-using the image details for this is a clever idea, however I wonder how obvious it is to users that clicking an informative bubble on the first step will trigger a configuration dialog. Maybe we can have another bubble with say a "cog" icon to open a different configuration modal?

This is my main worry. Putting all of this configuration in a "details" pane just feels wrong... it would be much more natural to have an obvious way of accessing this configuration. My initial thinking is along the lines of a cog icon as you said, but that feels too subtle. Maybe it should just pop up automatically when an image is selected.

It would also be nice if there were a way to see a quick summary of the chosen options; maybe everything that is non-default would be displayed in the image details along with the path.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/resin-io/etcher/issues/718#issuecomment-266546139, or mute the thread https://github.com/notifications/unsubscribe-auth/ABLUCNqRbMARSikMjOvu_LO-6rdrc8jCks5rHbEmgaJpZM4KFQHD .

konmouz commented 7 years ago

I am trying to reduce the amount of total modals and group information when possible. I'll explore it a bit more and I'll come back with a new UI.

lurch commented 7 years ago

I think we should definitely show this by default

How on earth would that work? IMHO this should only be available if the image-metadata explicitly asks for it, because it doesn't make sense to try and "guess" what settings an arbitrary image might need, what format they should be stored in, which files / partition they need to be saved on, etc.

Going back to my NOOBS experience: In NOOBS there's a mini "configuration step" while the OS is getting installed, where the OS gets an opportunity to configure itself based on what partitions it's actually getting installed onto. Since NOOBS can't know how different OSes need configuring, it looks for a partition_setup.sh script (see e.g. this one ), and if found it simply passes it a list of partitions as command-line arguments. It's then up to the script to mount whichever partitions it needs to write to, do whatever configuration is necessary, and unmount the partitions again.

When I saw @konmouz 's design pictures I assumed the image-metadata would simply specify whether it wants the "ethernet config" and "wifi config" configuration tabs visible (and we could define other standard configuration tabs). But from what @jviotti is saying with his discussion of widgets and layouts, it sounds like he's imagining each extended archive will be able to specify completely customisable configuration UIs? (which is obviously another whole level of complexity) If we are allowing custom-interfaces, then perhaps things like an "IP address input" control would also be useful? (or maybe just having a validation-regex option on the "Text input" (which the publisher could set to ^\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}$, or even the regex from http://www.regextester.com/22 ), would be sufficient?) Are the tickboxes next to the textboxes in @konmouz 's pictures so that we can tell the difference between NULL (i.e. don't store this setting) and "" (i.e. store this setting, but set it to an empty string) ?

I assume we'll be using reconfix-configurations stored in the image-metadata to convert the configuration captured by Etcher into the OS-specific config files? And then using LKL (as discussed in Athens) to actually write those config files to the SD card? Or for this first version are we just going to be writing a single JSON config file to a nominated FAT partition on the card, and leave it up to the OS to make use of that config file however it chooses? I guess (as an advanced option) we could even have a button to save the configuration as a 'default config' back into the image metadata itself (but just in the metadata, leaving the raw-image untouched), so that if the user is repeatedly writing images with the same config values (e.g. wifi login details) to multiple cards, they don't have to keep re-entering them each time they start Etcher.

Just having re-read through @alexandrosm 's "Device (re)configuration for Etcher" comment, one part left out of that which has just occurred to me is that if we're writing configuration files directly, we probably also need to be careful about the ownerships and permissions that new files get created with. (should reconfix just be given numeric uids and gids, or should reconfix be given the "more friendly" usernames and groupnames and attempt to read /etc/passwd and /etc/group/ from the image to lookup the appropriate uids and gids?)

Something else I've just thought of - obviously to ensure that we can verify the image got flashed correctly (by comparing checksums), we'll only want to "save" the config onto the SD card after we've flashed and verified it. But as https://resin.io/blog/the-perils-of-writing-disk-images-on-windows/ explains, on Windows we actually write the MBR-chunk of the image to the SD card last, because apparently Windows doesn't allow raw-access to 'partitioned' space. So in order to be able to write the config to the card, does that mean we'd actually have to read back the MBR-chunk from the card, erase the MBR-chunk, write the settings, and then write back the MBR chunk? :-S (and I wonder if that might lead to a catch-22 situation, with LKL needing to be able to read the MBR?)

WasabiFan commented 7 years ago

How on earth would that work? IMHO this should only be available if the image-metadata explicitly asks for it, because it doesn't make sense to try and "guess" what settings an arbitrary image might need, what format they should be stored in, which files / partition they need to be saved on, etc.

Of course. This would only apply if the image has Etcher-readable metadata and specifies things to configure.

When I saw @konmouz 's design pictures I assumed the image-metadata would simply specify whether it wants the "ethernet config" and "wifi config" configuration tabs visible (and we could define other standard configuration tabs). But from what @jviotti is saying with his discussion of widgets and layouts, it sounds like he's imagining each extended archive will be able to specify completely customisable configuration UIs? (which is obviously another whole level of complexity)

I think this feature is pretty much useless if you can't configure the controls that are shown. That will definitely be difficult and design-intensive, as with the rest of this idea.

jviotti commented 7 years ago

But from what @jviotti is saying with his discussion of widgets and layouts, it sounds like he's imagining each extended archive will be able to specify completely customisable configuration UIs? (which is obviously another whole level of complexity)

Exactly, what is shown is completely generated from the Reconfix schemas, and its indeed a big design challenge.

If we are allowing custom-interfaces, then perhaps things like an "IP address input" control would also be useful? (or maybe just having a validation-regex option on the "Text input" (which the publisher could set to ^\d{1,3}.\d{1,3}.\d{1,3}.\d{1,3}$, or even the regex from http://www.regextester.com/22 ), would be sufficient?)

That is a great idea: to have a collection of more complex domain-specific inputs like IP addresses. I'll create an issue in Reconfix to track it.

I assume we'll be using reconfix-configurations stored in the image-metadata to convert the configuration captured by Etcher into the OS-specific config files? And then using LKL (as discussed in Athens) to actually write those config files to the SD card? Or for this first version are we just going to be writing a single JSON config file to a nominated FAT partition on the card, and leave it up to the OS to make use of that config file however it chooses?

Reconfix will do the whole thing. LKL is only needed when writing to partitions other than FAT32, but I think we're fine with that, and therefore we'll delay the introduction of LKL here.

Just having re-read through @alexandrosm 's "Device (re)configuration for Etcher" comment, one part left out of that which has just occurred to me is that if we're writing configuration files directly, we probably also need to be careful about the ownerships and permissions that new files get created with.

This is a very good point. I'll create an issue on Reconfix for it.

Something else I've just thought of - obviously to ensure that we can verify the image got flashed correctly (by comparing checksums), we'll only want to "save" the config onto the SD card after we've flashed and verified it.

Exactly, configuration will happen after the flash/validation process.

But as https://resin.io/blog/the-perils-of-writing-disk-images-on-windows/ explains, on Windows we actually write the MBR-chunk of the image to the SD card last, because apparently Windows doesn't allow raw-access to 'partitioned' space. So in order to be able to write the config to the card, does that mean we'd actually have to read back the MBR-chunk from the card, erase the MBR-chunk, write the settings, and then write back the MBR chunk? :-S

The problem described in the blog post only applies when freely writing bites using raw I/O on the drive. Reconfix, on the other side, will mount the filesystems and interact with them instead.

lurch commented 7 years ago

need to be careful about the ownerships and permissions that new files get created with

But that's obviously not an issue if we're just writing to a FAT partition ;)

Reconfix, on the other side, will mount the filesystems and interact with them instead.

Ah, you mentioned https://github.com/resin-io/resin-image-fs in an earlier comment, so I'd assumed you were going to be using that, rather than mounting the FAT partition in the OS directly.

Several things are presented in a grid during the "Ethernet" configuration phase. How do we determine if we should show in a grid, one below the other, etc?

For the design images that @konmouz has posted, perhaps we could group controls together, into "meta-controls"? So there's a Priority meta-control (textbox and tickbox), a Static IP meta-control (textbox and tickbox), a Hostname meta-control (just a textbox) and a Persistent Logs meta-control (label and tickbox). And then the Ethernet tab is made up of a default section (no header) and an advanced section (with an Advanced Settings header). The default section is composed of the Priority and Static IP meta-controls, and a single label control (saying that Ethernet is the default). The advanced section is composed of the Hostname and Persistent Logs meta-controls. And then maybe the meta-controls in this example all specify that they use a half-width, and the label control specifies that it needs a full width (or maybe these sizes get calculated automatically), and the controls get "flowed" into each section, so that in the default section the Priority and Static IP meta-controls get flowed next to each other, and the label control gets flowed underneath.

So, in a crude (incomplete) JSON form, perhaps something like:

{
    "config": {
        "label": "Network Setup Configuration",
        "tabs": [
            {
                "label": "Ethernet",
                "sections": [
                    {
                        "controls": [
                            {
                                "label": "Priority",
                                "type": "meta-control",
                                "width": 0.5,
                                "controls": [
                                    {
                                        "type": "textbox",
                                        "default": "0"
                                    },
                                    {
                                        "type": "tickbox",
                                        "default": false
                                    }
                                ]
                            },
                            {
                                "label": "Static IP",
                                "type": "meta-control",
                                "width": 0.5,
                                "controls": [
                                    {
                                        "type": "textbox",
                                        "default": "192.168.1.111"
                                    },
                                    {
                                        "type": "tickbox",
                                        "default": true
                                    }
                                ]
                            },
                            {
                                "type": "label",
                                "width": 1.0,
                                "label-text": "*By default all devices are configured to use Ethernet"
                            }
                        ]
                    },
                    {
                        "label": "Advanced Settings",
                        "controls": [
                            {
                                "label": "Hostname",
                                "type": "meta-control",
                                "width": 0.5,
                                "controls": [
                                    {
                                        "type": "textbox",
                                        "default": "resin"
                                    }
                                ]
                            },
                            {
                                "label": "Persistent Logs",
                                "type": "meta-control",
                                "width": 0.5,
                                "controls": [
                                    {
                                        "type": "label",
                                        "label-icon": "warning",
                                        "label-text": "This can wear out Flash Media"
                                    },
                                    {
                                        "type": "tickbox",
                                        "default": false
                                    }
                                ]
                            }
                        ]
                    }
                ]
            },
            {
                "label": "WiFi"
            },
            {
                "label": "Cellular"
            }
        ]
    }
}

jviotti commented 7 years ago

But that's obviously not an issue if we're just writing to a FAT partition ;)

Ah, good catch!

Ah, you mentioned https://github.com/resin-io/resin-image-fs in an earlier comment, so I'd assumed you were going to be using that, rather than mounting the FAT partition in the OS directly.

Yeah, we have resin-image-fs for things like FAT, but we'll probably be switching to performing an actual mount as recommended by the LKL talk we had in Athens.

For the design images that @konmouz has posted, perhaps we could group controls together, into "meta-controls"?

Very nice feedback. What you're describing is basically the high-level "visuals" stage of Reconfix, for which we don't have much yet. Maybe we can create a couple of real-world examples based on your ideas to test the design against?

konmouz commented 7 years ago

@jviotti @lurch, for my screens I used a configuration sketch from @shaunmulligan. The more examples (print screens, text, anything) you can provide me with, the better. Then I can try to figure out a way to group objects into UI elements and fields. Also I need to understand how long and complicated this configuration can be, which can seriously affect the UI. @lurch's example/suggestion is already very helpful!

lurch commented 7 years ago

I need to understand how long and complicated this configuration can be

If we're letting the publishers define their own customisable configuration screens inside the image-metadata, then I guess there's no limit to how long and complicated they can be! Maybe having the tabs (Ethernet, WiFi, Cellular, etc.) across the top scrollable left<->right (of which there can be an "unlimited" number), with a configuration pane on each tab that's scrollable up<->down (which can have "unlimited" length), will be enough to contain "any" configuration UI?

konmouz commented 7 years ago

top scrollable left<->right

That's tricky since horizontal scrolling is not very effective. It might be better to have a LHS vertical navigation (scrollable if need be to host Ethernet, WiFi, Cellular, etc.) and the rest would contain sub-options of each option.

jviotti commented 7 years ago

Some examples I know of from other modules:

Resin Device Toolkit basic ResinOS configuration:

{
  message: 'Network SSID'
  type: 'input'
  name: 'networkSsid'
  default: data.networkSsid
}
{
  message: 'Network Key'
  type: 'input'
  name: 'networkKey'
  default: data.networkKey
}
{
  message: 'Do you want to set advanced settings?'
  type: 'confirm'
  name: 'advancedSettings'
  default: false
}
{
  message: 'Device Hostname'
  type: 'input'
  name: 'hostname'
  default: data.hostname,
  when: (answers) ->
    answers.advancedSettings
}
{
  message: 'Do you want to enable persistent logging?'
  type: 'confirm'
  name: 'persistentLogging'
  default: data.persistentLogging
  when: (answers) ->
    answers.advancedSettings
}

Parallella Processor Configuration

{
  message: 'Processor'
  name: 'processorType'
  type: 'list'
  choices: [ 'Z7010', 'Z7020' ]
},
{
  message: 'Coprocessor cores'
  name: 'coprocessorCore'
  type: 'list'
  choices: [ '16', '64' ]
}

ResinOS v1 network configuration

{
  message: 'Network Type'
  name: 'network'
  type: 'list'
  choices: [ 'ethernet', 'wifi' ]
}
{
  message: 'Wifi Ssid'
  name: 'wifiSsid'
  type: 'input'
  when:
      network: 'wifi'
}
{
  message: 'Wifi Key'
  name: 'wifiKey'
  type: 'input'
  when:
      network: 'wifi'
}

lurch commented 7 years ago

...and this is what the current resin.io dashboard offers you when downloading an image:

resinos_download_ethernet resinos_download_ethernet_advanced resinos_download_wifi resinos_download_wifi_advanced

jviotti commented 7 years ago

Its worth noting that the screens @lurch posted are being automatically generated already :)

konmouz commented 7 years ago

thanks guys, this is a good starting point.

konmouz commented 7 years ago

Design Update: I prepared two quick flows with adobe XD (be careful no all the UI buttons work). I am trying to reduce/ group the CTAs around the img to keep the UI light and clean (Info, change, configure). Give it a go and let me know your thoughts (ignore image configuration modal window content for now - I am still working on it).

Flow 1: https://xd.adobe.com/view/bbdfb1e8-f1b8-4de2-a467-fc17cf3cb5ae/ Flow 2: https://xd.adobe.com/view/4006fa10-79dd-48c2-aee8-f9541b17e041/

lurch commented 7 years ago

Look nice, but for "Flow 1" I don't like the fact that the "Change Image" button is only 'hidden inside' the image-info modal. And for "Flow 2" I'm not sure it's very obvious that clicking on the small "X" will open the file-selection dialog? My gut feeling is that "Change Image", "Image Info" and "Configure Image" are all buttons 'important enough' to warrant being on the main dialog (even if they're just icon-buttons).

Possibly too radical of a departure from the current UI, but if we detect that the chosen image is configurable (via its metadata), what if we changed the Etcher GUI to a four-step process, with a new "CONFIGURE IMAGE" stage appearing between the current "SELECT IMAGE" and "SELECT DRIVE" stages? (and I guess the image meta-data could specify whether configuration is optional (in which case we could immediately enable the "Select drive" or "Flash!" buttons), or whether configuration is compulsory, (in which case we could keep the "Select drive" and "Flash!" buttons disabled until a 'valid' configuration has been entered))

And I guess this "emphasising" of the Configuration step would also make it more obvious that Etcher (along with the extended image metadata) offers more functionality than just your common-or-garden disk-image-writing software :-)

If the image has no metadata, or if its metadata doesn't contain any configuration info, then we could either have the GUI automatically adapt to its current three-stage layout, or we could simply leave the "CONFIGURE IMAGE" stage greyed-out (i.e. how the "FLASH IMAGE" stage appears when you first start Etcher).

konmouz commented 7 years ago

My gut feeling is that "Change Image", "Image Info" and "Configure Image" are all buttons 'important enough' to warrant being on the main dialog (even if they're just icon-buttons).

I think these are too many options, even with icons. Anyway the image name itself is a strong CTA that would be nice to include one of the functions. It is expected to click the image name to either get details or change it.

Possibly too radical of a departure from the current UI, but if we detect that the chosen image is configurable (via its metadata), what if we changed the Etcher GUI to a four-step process, with a new "CONFIGURE IMAGE" stage appearing between the current "SELECT IMAGE" and "SELECT DRIVE" stages?

This is an interesting suggestion to explore, but since anyway we are planning a heavy UI redesign to accommodate new functionality (not for v1), I would try to keep v1 as simple as possible.

jviotti commented 7 years ago

Very cool stuff!

Can we rename "Image Configuration" to something else, in a more imperative way as we do for other CTAs, like: "Configure Image"?
I like displaying the image size next to the filename. How will this work for extremely long filenames though?
If an image allows configuration, then the call to action for the select image step should clearly be "Image Configuration", however what do we do when the image doesn't allow configuration? We can show something else (like "Change"), but switching the label depending on the image would be a bit strange/confusing to me (for example: why did image X allow me to configure but not image Y? Did I do something wrong? How do I enable configuration?). Maybe we should show no bottom label at all if the image can't be configured?

Flow 1:

I personally like the concept of the "cross" icon to remove the current image, but I worry that we're putting too much stuff horizontally on the same line (e.g: the filename, the size, the un-select icon, etc) given that we have no control over how long the filename could be. I guess its fine the way it is if we have a clear way forward for protecting ourselves agains long filenames.

Flow 2:

I don't like how the "Image Configuration" label changes to "Change" once the user clicks the informative icon. I think it should stay the same.

Overall I believe we're almost there. The flow is very cool, so just a matter of polishing it a bit more :)

konmouz commented 7 years ago

Can we rename "Image Configuration" to something else, in a more imperative way as we do for other CTAs, like: "Configure Image"?

Agreed

I like displaying the image size next to the filename. How will this work for extremely long filenames though?

We can set the classic solution of setting a limit and then use the triple-dot punctuation 'resinim...' or 'resinim... .img'

If an image allows configuration, then the call to action for the select image step should clearly be "Image Configuration", however what do we do when the image doesn't allow configuration? We can show something else (like "Change"), but switching the label depending on the image would be a bit strange/confusing to me (for example: why did image X allow me to configure but not image Y? Did I do something wrong? How do I enable configuration?). Maybe we should show no bottom label at all if the image can't be configured?

TBH I don't like showing and hiding buttons since it is highly confusing for the user. I think we can just grey out the 'configure image' and the tooltip can display 'this image is not configurable'.

I don't like how the "Image Configuration" label changes to "Change" once the user clicks the informative icon. I think it should stay the same.

Sorry my bad, I copied the old UI. I will update the image.

konmouz commented 7 years ago

Flow update: https://xd.adobe.com/view/4006fa10-79dd-48c2-aee8-f9541b17e041/

jviotti commented 7 years ago

This is looking great :+1: So just to make it clear: can we add the flow where an image is not configurable, just to see how that would look?

konmouz commented 7 years ago

@jviotti I've updated the link.

As discussed, we can remove the 'configure image' completely when an image is not configurable. The benefit with current solution is that we can let user know why the CTA is inactive (using a tooltip). If the CTA is not there, user has to make the assumption that this feature is not available with the selected image.

shaunmulligan commented 7 years ago

@konmouz this is looking awesome. One thing I worry about is the scrollable part of the configuration, but don't have solid ideas on alternatives.

konmouz commented 7 years ago

@shaunmulligan I am with you on this. Current solution is more like a placeholder and I am trying to come up with a better and modular solution to fit the needs of individual images.

jviotti commented 7 years ago

@konmouz

As discussed, we can remove the 'configure image' completely when an image is not configurable. The benefit with current solution is that we can let user know why the CTA is inactive (using a tooltip). If the CTA is not there, user has to make the assumption that this feature is not available with the selected image.

I agree, it makes a lot of sense. I love the tooltip and the current flow. I wonder if the label should be completely greyed out if the image is not configurable -- currently, it turns to another type of blue, which might still give the impression that is clickable.

WasabiFan commented 7 years ago

As a user, I feel a bit trapped when the only two options are "revert" and "looks good". I generally expect the secondary action to be passive (e.g. "cancel"), not active ("revert").

alexandrosm commented 7 years ago

It may be nice to have the "configure image" give an explanation of the sort "The current image cannot be configured due to absence of configuration metadata. Only extended archive images are configurable. (learn more)." That can act as a way for extended archives to be discoverable while not being super intrusive. thoughts?

--

Alexandros Marinos

Founder & CEO, Resin.io

+1 206-637-5498

@alexandrosm

On Tue, Dec 20, 2016 at 6:02 PM, Wasabi Fan notifications@github.com wrote:

As a user, I feel a bit trapped when the only two options are "revert" and "looks good". I generally expect the secondary action to be passive (e.g. "cancel"), not active ("revert").

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/resin-io/etcher/issues/718#issuecomment-268313178, or mute the thread https://github.com/notifications/unsubscribe-auth/ABLUCLsQLAp6ziTcx7sr337C9HnzxL5zks5rKBhSgaJpZM4KFQHD .

lurch commented 7 years ago

@konmouz Slight 'bug' in your current Flow in that when you click on the drive to display the drive-details modal, you can't click on the 'X' to close it. Also, when you select the non-configurable image, and click the small 'x' to deselect the drive, you can't then click on the small 'x' to deselect the image. I like the way you've got the image-details modal and drive-details modal looking so similar :+1: (but FYI the image-details modal may end up displaying more information when we add more info to the extended-archive metadata) In the current Flow you need to 'close' an image before you can 'open' a new one (and the same for drives), whereas with the current GUI it's only a single button-click to open the file-dialog. OTOH, clicking on an 'x' to signify "open new image" is probably confusing from a UX point of view. What if you instead showed a folder-icon next to the image-name to indicate "open different image"; and a small disk icon next to the drive-name to indicate "select different disk"? (but then that also means that the "Select Image" and "Select Drive" buttons would only be visible when you first started the GUI). Sorry, I'm waffling...

@alexandrosm that's obviously far too long for a tool-tip, and displaying a pop-up window just to explain that a feature isn't available does feel kinda awkward IMHO.

I agree with @WasabiFan that the "revert" button is potentially confusing - is it like "cancel" where all changes since the dialog were opened are discarded (but still keeping changes from previous dialog usages), or is it like "reset to defaults" where all changes are discarded and the image is reset to its virgin state? Also, maybe it's because I'm old and non-hipster, but I'd prefer "OK" to "Looks Good".

konmouz commented 7 years ago

@jviotti, the button gets a transparency when inactive, grey should be ok as well. @alexandrosm, I agree that this is a bit too long. what about the current text + learn more? @lurch, thanks for the bugs, I'll have a look. Regarding the icons, in the initial flow the 'x' button redirects to the selection window directly (drive or image) which can be a bit confusing since it acts more like a 'replace' button. This is why I added this extra step, but obviously it is a bit longer. The icons you suggested or an icon for 'replace' can have multiple meanings for the user, while the 'x' one is much more clear and this is why I decided to stick to it.

alexandrosm commented 7 years ago

what's the current text?

--

Alexandros Marinos

Founder & CEO, Resin.io

+1 206-637-5498

@alexandrosm

On Wed, Dec 21, 2016 at 9:26 AM, Konstantinos Mouzakis < notifications@github.com> wrote:

@jviotti https://github.com/jviotti, the button gets a transparency when inactive, grey should be ok as well. @alexandrosm https://github.com/alexandrosm, I agree that this is a bit too long. what about the current text + learn more? @lurch https://github.com/lurch, thanks for the bugs, I'll have a look. Regarding the icons, in the initial flow the 'x' button redirects to the selection window directly (drive or image) which can be a bit confusing since it acts more like a 'replace' button. This is why I added this extra step, but obviously it is a bit longer. The icons you suggested or an icon for 'replace' can have multiple meanings for the user, while the 'x' one is much more clear and this is why I decided to stick to it.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/resin-io/etcher/issues/718#issuecomment-268476017, or mute the thread https://github.com/notifications/unsubscribe-auth/ABLUCAUP17ezhRbS1_Ab8koKS5mLCqZqks5rKPDLgaJpZM4KFQHD .

konmouz commented 7 years ago

@alexandrosm, 'This image is not conﬁgurable'

alexandrosm commented 7 years ago

but how would a learn more link work with a tooltip?

--

Alexandros Marinos

Founder & CEO, Resin.io

+1 206-637-5498

@alexandrosm

On Wed, Dec 21, 2016 at 10:03 AM, Konstantinos Mouzakis < notifications@github.com> wrote:

@alexandrosm https://github.com/alexandrosm, 'This image is not conﬁgurable'

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/resin-io/etcher/issues/718#issuecomment-268483656, or mute the thread https://github.com/notifications/unsubscribe-auth/ABLUCEpWHYpEzULIlx8niT_NhubeT35eks5rKPmMgaJpZM4KFQHD .

konmouz commented 7 years ago

I guess @Shou may have the answer. But do we really need the 'learn more'? Where this will redirect?

alexandrosm commented 7 years ago

The idea is to inform people about the existence of the format. If it can't be discovered, it won't be used.

--

Alexandros Marinos

Founder & CEO, Resin.io

+1 206-637-5498

@alexandrosm

On Wed, Dec 21, 2016 at 10:26 AM, Konstantinos Mouzakis < notifications@github.com> wrote:

I guess @Shou https://github.com/Shou may have the answer. But do we really need the 'learn more'? Where this will redirect?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/resin-io/etcher/issues/718#issuecomment-268488695, or mute the thread https://github.com/notifications/unsubscribe-auth/ABLUCBv0cNXfTfl6t0u5gMAmIa1r5bkJks5rKP7ggaJpZM4KFQHD .

konmouz commented 7 years ago

@jviotti @lurch, regarding configuration options, can we put a limit on UI elements/tags (e.g. use up to 3 tabs)?

Shou commented 7 years ago

We could potentially keep displaying the tooltip on hover, then the link is clickable. Though if we want discoverability, I think it may make more sense to slap the message/link directly below, or somewhere else overt. Or perhaps we can trigger the tooltip to show up by default if it's non-configurable, but instead downwards so it doesn't overlap anything; it could either be dismissed by hovering over it when the cursor leaves (the normal tooltip behaviour), or a timeout.

balena-io / etcher

Image configuration support #718

This is what we currently have

This is what we're missing

Device (re)configuration for Etcher

Introduction

The four stages

Transformation pipeline

Stage transitions

From user interface to dry json

From dry to wet json

From wet json to files

Additional notes

A note on manual file alterations

A note on reconfiguration

A note on raw files

A note on “filesets”