Product configuration wizard

is your feature request related to a problem? Please describe. The product has so many configuration parameters with a lot of cross-dependencies between them already and new options keep coming with new releases. It would be not so easy task for a new user to go through all of them to configure a new product instance correctly according to the environment and business needs. Later, if the admin doesn't read each new release note, it would be difficult to stay on the same page and keep the product config in an optimal state.

Describe the idea / solution you'd like I propose to implement some stand-alone script, which would represent an installation and migration wizard with the next flow:

in case of fresh deployment it should guide a user through all the configuration options subsequently (respecting dependencies) with a detailed and clear description of each option, its impact, use cases, and examples. There might be a lot of steps, so some progress indication, intermediate save, and ability to stop and later get back to the same point is needed.
in case of an upgrade from a previous version (in-place), it should get the previous config and guide a user through the delta between the versions: pretty the same, but with much fewer steps, and if something previously available was changed, it should be clearly indicated.
in case of a migration to another system (side-by-side), I'm not sure what is better: from 1 side asking a user to go through all the steps, as for fresh installation while a significant part of options from the source deployment might be the same looks like wasting time, but from another side, we couldn't be sure which options should be changed and which - not: answer to this question is from user's business area, not from technical one. However, I'm not an expert in the Copyparty, so I might not see the whole picture. After the wizard completion, there should be a config for Copyparty with all user answers reflected and a step-by-step guide on how to deploy the product and apply the config (all supported setups should be taken into account as part of the wizard)

Describe any alternatives you've considered Keep it "as is" and force a user to manually configure the product putting all the responsibility of misconfiguration on the user's side

Additional context This request came from my personal experience of analyzing a long list of config params with not-so-clear (as to me) descriptions, and I believe that my setup is not optimal (it is running in a local network behind a firewall, so it isn't so critical, but still...) Feel free to ask some questions in comments in case you need to clarify something)

Yep, this is definitely becoming an issue -- and you're probably not surprised to hear that (something like) this has come up in discussions before :)

Part of the problem is that it's really hard to cover all the possible scenarios and usecases with an automated solution. I'll start by writing down all the stuff I can think of, using this post as a scratchpad and continue to edit over time. This section is just a list of things to keep in mind when designing this feature, to make sure we don't miss anything important.

How is copyparty deployed? The python sfx? The windows exe? An OS package (nixos/arch)? A docker container?
- The way you launch copyparty and give it configuration files is different, and the opportunities for copyparty to generate its own config can be tricky given some of these (well, primarily nixos)
Is copyparty facing the internet directly? Through a reverse-proxy? In a docker container?
- Need to configure X-Forwarded-For somehow; maybe it can show all the IP-addresses it can detect through common means and suggest different settings based on which IP looks to be correct
- The correct setting for --rp-loc can possibly be autodetected if copyparty were to serve a setup GUI over HTTPS
Are we running on Windows or Linux/macos?
- Could affect how to visualize the filesystem for volume configuration

So far, I have been playing with the idea of copyparty offering a server setup GUI over HTTPS:

By launching copyparty with --setup, it will autogenerate a passphrase for a URL which can be accessed to configure everything visually.
This URL would be extra powerful, since it will expose the entire filesystem of the server for the purpose of configuring volumes. It would primarily list folders only, and not files, however it will also allow the user to apply and activate the new config, meaning it could be used to download any file which is visible to the copyparty process. Thus the passphrase must be strong, and the setup API could possibly self-disable after setup has completed.
- Maybe there should be additional safeguards, but that could be non-trivial (for example, when running behind a reverse-proxy or within a docker-container, there is no guaranteed way to do IP allowlisting, for example).
Copyparty would generate config files and write them to disk automatically; and those config files would be loaded back into the GUI the next time you visit the setup UI.
- The config would also automatically apply when launching copyparty, no arguments necessary to do so, except for --profile if necessary/preferable (see below)
Since this means that copyparty would be generating its own config files, it would need to be fully in charge of the generated configs, since the GUI would probably be incapable of comprehending every conceivable scenario.
The generated config would need to have a predetermined structure; perhaps a folder of config files according to the following:
- global.conf for server-global settings
- vol-music-upload.conf for each volume; this example would be for /music/upload
- accounts.conf for accounts and their passwords
Many servers run multiple instances of copyparty, all with different settings, so there must be a "profiles" concept where you can specify which profile to use for this instance
There must be a way for users to provide their own handwritten config files in addition to the GUI-managed settings, to specify settings which are not available through the GUI.
- This will probably have to be specified as a command-line argument; having this defined in a config-file would be too much of a catch-22 to handle reliably
The GUI must do an excellent job at visualizing whether a setting is either applied globally or per-volume, maybe with columns of radiobuttons, and a separate column indicating the global configuration that is being overridden by a per-volume setting

Later, if the admin doesn't read each new release note, it would be difficult to stay on the same page and keep the product config in an optimal state.

This is a good point! I think copyparty should remember the version which was previously running on the server, and then show a "news" printout on startup, explaining what happened since that version, with suggestions on what might be necessary to change in the settings. The settings GUI should also have a separate page with this information, where you can specify any version you would like to compare against.

given the ideas so far, the following would become the new options to enable the setup GUI:

--setup enables a setup GUI available at /?setup=jfah51fenc46haeb2cua872ds (autogenerated password)
--setup-pwd specifies a non-autogenerated password to use instead
--setup-fs (if provided) limits the filesystem paths which are permitted to access by copyparty (can be repeated)
- in docker containers: automatically set to /w which is the container's intended mountpoint
--profile path to folder of config-files which is managed by the setup GUI; defaults to ~/.config/copyparty/1
- either provide an absolute path, or a word to use ~/.config/copyparty/WORD

and in --help-setup, there would be a note explaning how to mix autogenerated config with handwritten, more specialized configs:

note: to specify any settings which are not available through the GUI,
  add your settings to one or more files in ~/.config/copyparty/PROFILE/
  with filenames preferably starting with the letter x to ensure they
  will overwrite any settings generated by the GUI, or alternatively
  provide the paths to your custom config files using the -c option

Happy to help with the logic, providing comments and my point of view (which might trigger some useful thoughts):

hard to cover all the possible scenarios and usecases with an automated solution

sure. it definitely is, taking into account the # of options and flexibility of the product. However, it's better to start from some basic and most typical cases, iteratively adding more complex ones to expand the coverage. The wizard shouldn't cover each and every case from the beginning: there could be a disclaimer at the beginning, that this wizard is provided for your convenience and covers the next setup scenarios. if your scenario is not 100% aligned, you could go through the wizard choose the closest one, and then edit the generated config manually aligning it to your needs, or configure Copyparty manually from scratch.

How is copyparty deployed?

Is copyparty facing the internet directly?

Are we running on Windows or Linux/macos?

If I understood you correctly, these are the 1st steps of your wizard. the 3rd one doesn't even need to be asked: the answer could be easily found by the server itself.

By launching copyparty with --setup, it will autogenerate a passphrase for a URL which can be accessed to configure everything visually.

I'd not recommend making the wizard to be part of the main app, as it is a significant security risk. Instead, I'd make it as a separate server, running on a different port: in this case, the admin has all the power of routes and firewall configuration, making sure the admin UI is accessible to only the required subnet, as well as it doesn't depend on the main app, so the app and the wizard services could be started, stopped, and restarted independently at any time:

the generated config could be tested right after generation in the next browser tab and updated in the wizard again if needed with no additional movements
once configuration is done, the wizard's web server can be easily turned off without any additional logic for that in the main app, so even if a hacker would get somehow access to some API, there will be no even theoretical possibility to spin-up the config UI and reconfigure the server.

Since this means that copyparty would be generating its own config files, it would need to be fully in charge of the generated configs

Only in the scope of the covered scenarios, as it should clearly notify which are currently covered. So, in the case of additional configuration required, user has to do it manually taking all the responsibility of those changes.

The generated config would need to have a predetermined structure

Great idea: having a structure of configs, where each is for particular functionality, looks much better than having 1 long and complex file.

here must be a "profiles" concept where you can specify which profile to use for this instance

The wizard could ask for multi-instance usage, and if more than 1 profile is configured, it could show the server execution command for each profile configured, as a final step together with the path to generated configs.

There must be a way for users to provide their own handwritten config files in addition to the GUI-managed settings

Yes, as I mentioned before: just instruct a user to manually modify generated configs or server execution command if needed

The GUI must do an excellent job at visualizing whether a setting is either applied globally or per-volume

How about configuring all global settings at 1st, and then clearly notify a user that now we're going to configure volume level settings, after that, on each option's step could be a table with all volumes configured in lines and the default globally-defined value for the potion pre-selected in front of each volume with the ability to change it?

copyparty should remember the version which was previously running on the server, and then show a "news" printout on startup

I still think that the versioning of the config with parsing the version by the config GUI is much better:

the News page is functionally the same as the release description on Github with only the difference to be closer to the users, but if the admin doesn't read and follow the changes on Github, the News might be ignored as well.
who is going to see that News page? I could assume that the admin could be not so active user of the Copyparty instance, logging in only when some incident reported or maintenance needed. In that case, the admin usually is focused on getting the task done and most probably will ignore all noisy stuff. On the other side, if we'll have the config UI with the config versioning and parsing, the admin would be automatically notified about the new changes in the product, once the old config is loaded with clear instructions for change, natively integrated into the config modification wizard.

Again thanks for your input :>

First off, one thing I forgot to mention in https://github.com/9001/copyparty/issues/57#issuecomment-1782731760 is that I try my best to not break any existing installations or configs by changing the behavior of existing functionality. If such a change is made, it will be noticed in the breaking changes section in the readme, assuming I do not fail to realize this myself. Of course, there can be new features which could be a good idea to enable for an existing installation, in regards to performance or security, so currently the best way to become aware of those is to read the github release-notes as you mentioned. New security enhancements are always default-enabled assuming they do not break expected functionality, cause unexpected behavior, or cause major user inconveniences.

And sorry for the confusion regarding the initial numbered list -- the intention was to keep track of all the complicating factors to keep in mind when designing this. Like you said, going for a separate configuration utility would make most of these concerns easier to handle, so they are mainly an issue when considering a built-in configuration GUI.

If I understood you correctly, these are the 1st steps of your wizard. the 3rd one doesn't even need to be asked: the answer could be easily found by the server itself.

This part is interesting, because it brings up another open question. Assuming we decide to go for a separate configuration utility, running on a dedicated port, then that is another piece of software which must be downloaded and executed on the server, possibly requiring firewall configuration and such, so a user with a linux server might decide to run the configuration utility at home on their windows computer instead. This would cause a big difference in the environment of copyparty itself and its setup utility, and less assumptions can be made. Of course, we could strongly recommend to run both copyparty and its setup utility in the same environment. But it looks like having copyparty provide its own setup GUI would make this less error-prone, and also give the GUI more opportunity to see things from the correct point of view.

make it as a separate server, running on a different port: in this case, the admin has all the power of routes and firewall configuration

I see you're pointing out the same things I did in the previous paragraph just now, but you're seeing it as an advantage :) and yes, I agree it does make for a more secure approach, but also I'm worried that the additional work that would be required to access and use the setup feature would then be too much of a drawback.

making sure the admin UI is accessible to only the required subnet

I may do some prototyping with the idea of providing a setup URL with an autogenerated passphrase, and see if that would become as secure as I was hoping it would, since that is our primary concern. I'm fairly confident that there would be no way for unintended people to access it, but I'll run the idea by some friends offline too.

as well as it doesn't depend on the main app, so the app and the wizard services could be started, stopped, and restarted independently at any time

This should be possible even if the setup-UI is built into the app too, with a tiny bit of python magic :) so I'll prototype a bit around that too!

The wizard could ask for multi-instance usage, and if more than 1 profile is configured, it could show the server execution command for each profile configured, as a final step together with the path to generated configs.

Good point! This would also apply to a built-in setup UI; it should display all automatically detected profiles, and make it obvious which one you are working on.

just instruct a user to manually modify generated configs or server execution command if needed

This is a possible approach, but it would make it harder for the wizard to understand its own configuration files afterwards, since it would have to "read around" the changes made by the user somehow, and those changes would also probably be lost the next time a new configuration is generated by the wizard. I hope it will be possible to make all customizations by creating an additional configuration file which is given to copyparty in addition to the wizard-generated config, but there might be some settings which must be modified this way indeed.

How about configuring all global settings at 1st, and then clearly notify a user that now we're going to configure volume level settings, after that, on each option's step could be a table with all volumes configured in lines and the default globally-defined value for the potion pre-selected in front of each volume with the ability to change it?

This could definitely work -- it depends on what order the setup steps would be performed in, or if it would instead be a fully dynamic settings GUI rather than a wizard. Both of them make sense.

the News page is functionally the same as the release description on Github with only the difference to be closer to the users, but if the admin doesn't read and follow the changes on Github, the News might be ignored as well.

Yes, more or less -- but it would only include the information that a user "must know", which is much less than all the bells-and-whistles mentioned in the full github notes. This would be displayed in the server log as copyparty is first launched after an upgrade, but should also be displayed as full-screen warnings when you access the settings UI, pointing out which options were affected directly.

But including the version number in the generated config is a good idea, so let's do that :+1:

Looking at how other similar software does it,

These provide their own Setup GUI available in the application UI, which can be accessed and modified at runtime by logging in with the admin account: ownCloud, Nextcloud, Seafile, HFSv3, Chibisafe, kodbox, filebrowser

These provide a full Setup GUI which is only accessible locally, not over HTTP: HFSv2

These provide a textmode Setup UI which is only accessible locally, not over HTTP: rclone

These provide no configuration helpers, expecting plaintext config files: FileGator, sftpgo

ownCloud and Nextcloud also provides a very minimal setup wizard for the first-time setup, to configure the password of your own admin account, and even includes a warning that this setup wizard should not be accessible to the internet due to the risk of malicious users being able to hijack the installation. However, these setup wizards require no authentication, so if copyparty generates a passphrase on startup as mentioned in https://github.com/9001/copyparty/issues/57#issuecomment-1782731760 then this should be fine.

Some more ideas,

Maybe an approach could be to bundle the setup GUI into copyparty, but adding an option to make copyparty enter into an entirely separate "setup mode" which disables all the usual features, and only provide the setup capabilities? Just another idea to consider once we get to it.

Another idea woud be to build the configuration utility into copyparty, but it would be a textmode / terminal menu-based GUI instead, and not a web-browser UI. But, while that does solve the security-related issues with having a built-in setup feature, I'm not sure if this is a good idea, partially because many docker-based deployments will never let copyparty show an interactive console.

I will probably not be able to pick this up for a while still, so that gives us plenty of time to think :)

I also just realized that it would be safe to keep the setup GUI enabled during normal use as well, because copyparty already supports argon2-hashed passwords. That way, the correct URL/passphrase you would need to access the setup GUI would never be printed to the server log, or displayed on startup, if you provide a hashed password when launching copyparty with --setup-pwd +aCBoaaRSwNv3psWx2HSwmBj-VTI_7RYp (the hashed version of the password you wish to use).

Assuming that the server log is considered safe from prying eyes (which is usually the case), then this is not really a concern in the first place, and using the autogenerated passphrase for permanent access is probably also alright.

EDIT: one concern is that the setup GUI password could be stolen from browser history, or through xss. This can be mitigated by requiring the user to authenticate using the regular login page instead, which generates an access token (stored as a cookie) which is valid for 6 hours, allowing access to the setup GUI. Still just an idea, but one more thing to consider...

it would make it harder for the wizard to understand its own configuration files afterwards, since it would have to "read around" the changes made by the user somehow, and those changes would also probably be lost the next time a new configuration is generated by the wizard.

Well, here I have 2 co-existing points of view:

basically, the product config is not a plain text with all its flexibility of natural language, it is a pretty strictly formatted combination of known parameters with their values, so even while config wizard wouldn't be able to work with all of them covering all possible combinations, it could selectively parse the config and extract or modify only those, it is educated about.
saying that, I understand that there might be combinations of individually known parameters (or known and unknown ones), representing not yet known use cases, leading to incorrectly extracted configuration to the wizard, and because of that, corrupted config after following save. So, there is a typical business solution for such cases to notify users that after manual modification of the generated config, it is not safe to upload it back to the wizard, so once modified manually, it has to be maintained manually at least until the wizard knows all the possible settings combinations. From the technical side, we could achieve that by signing the generated configs using the server's certificate: while the private certificate is hidden from the user, every manually modified config will fail signature verification and the wizard will easily detect that.

option to make copyparty enter into an entirely separate "setup mode" which disables all the usual features, and only provide the setup capabilities

interesting approach, it could work, but in that case admin won't be able to test the config at some intermediate states without completely exiting from the wizard and switching the server to the normal state, then getting all the steps in vice versa to continue. This is why I proposed going with independent server and wizard services, as here you could make the changes in the wizard and after a simple server restart see the impact, similar to the idea of nodemon in the scope of the web development

build the configuration utility into copyparty, but it would be a textmode / terminal menu-based GUI instead, and not a web-browser UI

Definitely, it is a more secure solution, as we don't open such sensitive API to the network. However, I'd agree that the very minimal docker containers without a shell won't let the user access this wizard (as well as some cloud deployments, where there is no SSH connect to the instances of deployed solution).

it would be safe to keep the setup GUI enabled during normal use as well

well, my point is that the config UI is like a root: it is super-powerful and not needed during normal operations. There is a best practice to disable the root user on Linux at all. And here I follow the same logic: no matter how strong is the protection, if the door physically exists and thieves know about the gold behind, sooner or later it will be stolen. These days with GPU and cloud computing password bruteforcing is very fast...

9001 / copyparty

Product configuration wizard #57