Separate the GUI frontend, and allow Duplicati to run as a service

[GoogleCodeExporter]

If the user closes the Duplicati icon (or is not logged in), Duplicati will
not run. If Duplicati can run as a service (daemon on linux), these
problems will be solved.

The GUI should communicate with the service over an encrypted TCP link (or
possibly named pipes, if they are on the same host).

Original issue reported on code.google.com by kenneth....@gmail.com on 26 Feb 2009 at 12:27

[GoogleCodeExporter]

Issue 25 has been merged into this issue.

Original comment by kenneth....@gmail.com on 13 Mar 2009 at 3:34

[GoogleCodeExporter]

I target the service/client splitup for version 2.0.

Original comment by kenneth....@gmail.com on 20 Mar 2009 at 10:13

• Added labels: Milestone-Release2.0

[GoogleCodeExporter]

I think Duplicati is great; but I am waiting for it to run as a service before
deploying it for mission-critical backups.

Looking forward to v2.0 - keep up the great work.

Original comment by a.digius...@gmail.com on 8 Jul 2009 at 7:14

[GoogleCodeExporter]

A possible solution would be to start Duplicati using screen. For example:

screen -dmS Duplicati ./StartDuplicati.sh

Or integrate the screen command in the skript file.

Original comment by lothark...@gmail.com on 28 Dec 2009 at 3:53

[GoogleCodeExporter]

Can you re-connect the X session to an app started in a screen?

Anyway, good idea, but screen requires that the user logs in, and then runs the 
script.

To be fully useful, the service should be able to start without any users 
logged in,
and not require any userinterface. And I am not aware of a screen 
implementation for
windows.

Original comment by kenneth....@gmail.com on 28 Dec 2009 at 7:35

[GoogleCodeExporter]

The way I deal with this now is to use CommandLine and a scheduler.
It would be nice if CommandLine could use the GUI backup profiles.

Then you would have what would be effectively a running service.

Original comment by jhiswin on 21 Apr 2010 at 5:35

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

I haven't thought of it that way, but yes, I can see where this could be 
usefull.
The downside to that solution is that you will rely on the properties of the 
scheduler to 
see and control status.

For this issue I am planning to make this entirely integrated, so the UI looks 
close to 
what it does now, but transmits/receives data from the service.

The timeline for the completion of this issue is a bit far ahead, if the 
current 
development speed is an indicator, as there are still numerous issues awaiting 
1.2.

The implementation of commandline/gui profile combination can be developed with 
significantly less effort. If you would like such a feature, please create a 
ticket for 
it, and we can discuss what it should look like.

Original comment by kenneth....@gmail.com on 24 Apr 2010 at 8:19

[GoogleCodeExporter]

Issue 281 has been merged into this issue.

Original comment by kenneth....@gmail.com on 12 Oct 2010 at 7:42

[GoogleCodeExporter]

Wouldn't it be possible to add Duplicati.CommandLine.exe via Windows API or 
Sc.exe [1] as a service during the installation? 

It would require the addition of reading and respecting the scheduling data 
from the user's (or system's) profile, but that functionality appears to be 
implemented already in the GUI, so you could perhaps reuse that code.

[1] http://support.microsoft.com/kb/251192

Original comment by achim.j....@gmail.com on 15 Oct 2010 at 3:19

[GoogleCodeExporter]

Yes, that is a bit of a shortcut. Simply have a micro scheduler that reads the 
database and runs the commandline version at the right moments.

The service would have to accept some sort of control instructions though, so 
the user can stop/pause/throttle a backup. There is also a need for a reverse 
communication, so the user can see what backup is currently running, how far 
along it is etc.

It would also require that the GUI can access the sqlite database with the 
settings. One request is that it is possible to install a "headless" version, 
which only installs the service, and then be able to configure and inspect the 
backup from a remote location.

Original comment by kenneth....@gmail.com on 15 Oct 2010 at 3:48

[GoogleCodeExporter]

What is the best way of contributing to getting this resolved ? I don't have a 
lot of time to put in, but I am quite interested in seeing this progress.

Original comment by shinji6...@gmail.com on 9 Nov 2010 at 11:41

[GoogleCodeExporter]

I have an off-line project with some other people who should be working on a 
proposal for this. So some progress may be on the way.

The current state of the code is that the scheduler and backup process runs 
completely without being tied to the GUI. It could in theory just not create 
any UI elements and run as a service. (But that would not be manageable).

My expected approach is outlined below. If you have comments to a particular 
item, the sequence,  overall approach, or particular use case, please let me 
know. If you feel comfortable tackling one of the tasks, you can get a sandbox 
and work on a branch of the current trunk.

1) Formats
I would start out by identifying all requests being sent to the 
scheduler/runner, and define a format for those requests (preferably xml). Then 
I would identify all events being raised by the scheduler/runner, and define a 
format for those. This should be fairly easy.

2) Communication method
Next step is to define a flexible method for sending requests and events 
between the two programs. This is slightly more complicated, because there are 
multiple factors to optimize for. A simple first version could just write files 
in a pre-agreed folder. This is inefficient and causes disk-wear so should not 
be the final version. I was thinking that it could be implemented as a simple 
http server, but that does not do two-way communication well, and polling would 
not be nice. But a http interface would simplify some later development, so I 
would prefer that.

3) Authentication
When the communication channel and messages are well defined, an authentication 
system must be defined. It should be flexible enough to be portable and still 
allow a local user a login-free connection (perhaps using the GUI process 
credentials?).

I'm not entirely sure how the users are to be verified. I would prefer that the 
users are somehow extracted from the OS, so there is no local database to 
maintain. But that may be hard to make in a cross-platform compatible manner.

4) Multi-user
Next, the service must be able to start multiple instances, each in the correct 
user context, so multiple users can use the same machine without interfering 
with each other.

5) Portable support
The --portable-mode can be implemented simply by hosting the service 
in-process, and communicating internally with it through an alternate channel.

6) Remote display/editing
If the communication channel already uses http, it should be very easy to make 
a small html page that requests the status and shows it in a browser.

If the backups should be entirely editable, there are some issues with the 
custom dialog components from backends, as they are Windows.Forms only. A first 
version could present the options in a grid-like fashion, a second version 
could include a html version of the UI.

7) VSS and other security stuff
As the service will possibly run in the Administrator/root account, some extra 
options may be required to prevent non-privileged users to backup data they are 
not allowed to.

Original comment by kenneth....@gmail.com on 9 Nov 2010 at 7:08

[GoogleCodeExporter]

I just want to add my vote for this. I don't particularly care about all the 
API stuff, sending custom requests to the server etc. - I'd just like the icon 
to be gone from the task bar, so my users can't accidentally close Duplicati, 
and be able to control Duplicate via a web interface remotely.

I'm confused about point (3) and the need for a database. Isn't there already a 
database where the server stores the backup jobs it's supposed to do?

Original comment by elsdoer...@gmail.com on 25 Mar 2011 at 5:47

[GoogleCodeExporter]

Yes, there is a database with the setup, but the idea is that Duplicati becomes 
manageable from a remote location, and since you probably do not want everyone 
to monitor/edit your backups remotely, there needs to be a list of approved 
logins/users.

I have not yet decided on how to manage this on a system with multiple users; 
should they share the backup database, or have one each? If they have one each, 
how do I figure out which one the (remote) user wants? Since this is still not 
decided, I would start of by making a user-login database.

Original comment by kenneth@hexad.dk on 26 Mar 2011 at 12:03

[GoogleCodeExporter]

I would like to recommend not integrating this directly into the main program.

There are other ways of remote management, such as RDP/VNC/SSH, which will 
likely already be in place.  I also feel the same about the scheduler.  Many 
systems already have a system in place for scheduling.  If this is added as a 
feature, I would suggest making a separate program for creating scheduling 
rules, or a separate optional scheduling daemon.
Personally, I'd much prefer adding cron rules for backup myself, or creating 
Windows scheduled tasks myself.  Allows me to better organize scheduled tasks, 
and I have plenty of flexibility and control.
There are also reporting and logging tools for monitoring backups.  I could 
just create modules to parse and input logging data that Duplicati generates.

What I'm trying to say is that, scheduling and remote management would be 
separate programs in of itself.

Original comment by jhiswin on 26 Mar 2011 at 2:05

[GoogleCodeExporter]

As I see it, the current problem is that Duplicati cannot both run as a 
service/scheduled AND show a UI.

You can currently setup Duplicati to run with the system scheduler 
(cron/windows scheduler), and that option will not go away. But if you do so, 
you cannot pause Duplicati, should you want to.

For this to work, Duplicati will need to run as a separate process and accept 
commands from another process, which implies some form of inter-process 
communication.

It would be (almost) zero extra work to allow the communication channel to 
accept input from another machine, as opposed to limit it to local machine 
input.

Should you choose to only run the UI locally, you can still use VNC/RDP to view 
and manipulate the UI. This solution works for many cases, but if you are an IT 
administrator, you do not want to do this for, say 20 machines each day, to 
monitor backup status. If the inter-process communication allows external 
input, it is a matter of writing a program that queries all known hosts and 
presents the results in a sensible manner.

So I do not think there is any incentive for not making remote management a 
possibility. This will not remove the option of running the UI locally and 
using RDP/VNC, nor will it remove the option of running it "portable" or simply 
scheduled.

Should you wish not to use the remote management feature, it is a simple case 
of not opening the port in the firewall (possibly locking the listener in the 
program).

As for running it with the system scheduler, I can see some cases where this 
would be beneficial. For now I would prefer not to depend on the OS scheduler, 
as I can see some issues where the user modifies/deletes the schedule causing 
Duplicati and the scheduler to be out of sync, e.g. leaving un-used backups in 
Duplicati.

Feel free to add an issue requesting integration with the OS scheduler though, 
and we can discuss what should happen when.

Original comment by kenneth@hexad.dk on 1 Apr 2011 at 9:04

[GoogleCodeExporter]

So, something like xml or telnet-ish for remote management right?
I was thinking more along the lines of shell scripting or install package type 
management, but that makes sense.
Ya, you could just have it bind only to 127.0.0.1 or reject other IPs.  And 
authentication too.

I personally don't like programs that take over scheduling, because it tends to 
spread out configuration all over the place when I have to manage things.  But 
I guess it could be limiting for the backups.

For logging I was thinking more along the lines of things like Nagios, AWstats, 
and/or eventlog exporting.

So, would the plan be to daemonize the main program, and just make UI/cli 
interact using the remote management API?  OS scheduler integration wouldn't 
really be necessary then as anyone needing that could easily just use the cli 
and some shell scripts for that.

Original comment by jhiswin on 1 Apr 2011 at 1:18

[GoogleCodeExporter]

The remote management would be something like a webservice interface where you 
can read/write xml like responses (or probably JSON).

I had not considered it, but of course, you can just use the OS scheduler and 
have it send commands like a normal cli/script and never user the internal 
scheduler.

The log is something I expect there to be a 100 different requests for, so I 
plan to make it plugable with simple python scripts, so you can easily redirect 
messages to your favorite destination. There will probably be stock logging for 
linux log, windows eventlog, file based and email.

Original comment by kenneth@hexad.dk on 1 Apr 2011 at 6:43

• Changed title: Separate the GUI frontend, and allow Duplicati to run as a service

[GoogleCodeExporter]

Original comment by kenneth@hexad.dk on 26 Apr 2011 at 7:00

• Changed state: Started

[GoogleCodeExporter]

Is there any way to use existing duplicati profiles with the system scheduler 
on windows?

Original comment by junkmail...@int0rnet.net on 27 Apr 2011 at 10:59

[GoogleCodeExporter]

As of right now, no.  After this issue is implemented it should be possible by 
using the cli.

Original comment by jhiswin on 27 Apr 2011 at 11:03

[GoogleCodeExporter]

If your *only* requirement is to replace the Duplicati scheduler with the 
Windows scheduler, you can create a job that runs:
"C:\Program Files\Duplicati\Duplicati.exe" --run-backup="<backup-name>"

It is not entirely meant to be used this way, but it will run the backup, 
provided that it exists. It is slightly un-stable in that it still requires a 
display to be attached and that there is no return code or other non-UI 
indication of success/failure.

If Duplicati is not running, it will start. If it is running it will activate 
the backup in the running instance.

Original comment by kenneth@hexad.dk on 28 Apr 2011 at 8:02

[GoogleCodeExporter]

That was exactly what I was after, thanks!

I'm still looking forward to the resolution of this issue, however.  

And if I may put my $.02 in, I think you should punt on user management and 
just employ basic password-only authentication to the backend for now.  

Regarding the scheduler, I would defer to the OS scheduler.  That is the status 
quo for the *nix world, and I believe there are windows APIs for managing 
scheduled tasks if you wish to continue to maintain the scheduling in the UI.  

Regarding communication, if I recall .NET allows you to expose methods as 
webservice methods fairly easily - although I'm not sure how this works in 
standalone applications.  I would suggest this approach if you can get it "for 
free" with the libraries you are using.  If you want to be real hip, you can 
structure things into a REST/JSON API, but SOAP would probably get you up and 
running most quickly in a .NET environment.

Anyhow, this is a great and much-needed project.  Keep up the good work!

Original comment by junkmail...@int0rnet.net on 28 Apr 2011 at 8:23

[GoogleCodeExporter]

Ah didn't realize that duplicati would run continuously as a service in this 
model, so I suppose my OS scheduler comment is moot.  Nevermind :)

Original comment by junkmail...@int0rnet.net on 28 Apr 2011 at 8:25

[GoogleCodeExporter]

Were about 2.5 years into this request. Would be nice if more effort would go 
into this feature than adding a bunch of crazy new storage options such as s3, 
skydrive, google docs, etc. I just want a stable backup/restore solution thats 
cross platform and can easily work in headless environments. =P

Original comment by mac...@gmail.com on 20 Sep 2011 at 9:23

[GoogleCodeExporter]

I agree this feature would be quite useful. The ability to run as a service 
with a separate config GUI would definitely make Duplicati a server-class 
application.

Original comment by stan.blo...@gmail.com on 20 Sep 2011 at 9:50

[GoogleCodeExporter]

Re #26: It's a volunteer project. If that's what you want, your non-demanding, 
non-condescending help would be appreciated. I'm sure that the author would 
welcome solid patches.

Yeah, I want this too, but you get more flies with sugar than vinegar, yes yes.

Original comment by BoldAndB...@gmail.com on 20 Sep 2011 at 9:52

[GoogleCodeExporter]

Thanks for letting me know that this is still requested :)
(Warning: There is a lot of text following....)

I do what I can to keep this project running, but it is still, largely, a 
one-man-show.

If you refer to the creation date for this issue, it was back when 1.0 was just 
released, and that version had a lot of problems that I tried to address with 
1.2. I was hoping that 1.2 would be stable enough so that I could focus on 2.0 
(thus the service split), but instead I am trying to do a minor 1.3 release 
with some additional problems adressed. Unfortunately I do not work as fast as 
I had hoped, but I only work on this in my spare time, and there has to be time 
for family too.

You might wonder why I seem to have enough time for minor tasks, but not focus 
on the larger issues. The reason is that the smaller issues are much more 
immediate, and I can go in and make a GoogleDocs backend in a few hours. It 
does not matter much if I am interrupted while doing it, because it is almost 
mechanical work for me. An issue like issue #9 requires a lot of planning and 
it takes a long time where I have to keep track of all the different 
sub-issues. If I am interrupted while doing this, chances are that something 
breaks horribly, and thus I need to know that I have at least a few hours 
before I can work on this.

Just because there has been no progress externally on this issue does not mean 
that I am not working on it, but I want this issue to adress some more 
fundemental issues with Duplicati. 

First such issue is that the WinForms has to go. When I started, it looked like 
the Mono project were just about to ship themed WinForms components for Mac and 
Linux, but they have not yet, and while it mostly works, it looks like crap. 
Even if I could fix the Mac trayicon issue, there would never be more than a 
few users because it looks so horrible (worse than Win95). 

Second issue is that I would like this to open the door to buisiness users that 
want to do remote monitoring and control of the backup.

My idea to solve this is to go all html/css/ajax with the UI, and this involves 
a mini webserver that basically serves all the static html/css files and has a 
special ajax-like interface for controlling the database, the scheduler and the 
currently executing backup. I have the infrastructure for this ready, but I 
need to implement all the ajax call methods and the html/css UI. This model 
will allow me to write a tray-icon native to each platform that looks (and is) 
100% system-skinned, but basically calls the webserver (i.e. practically no 
code to maintain). The main UI (the wizard pages) will then be shown in a 
native window with an embedded browser, as that works on all platforms and 
relieves me from having to write more than one version of the pages. The 
html/css is also MUCH easier to work with, and I hope that this will allow more 
people to contribute to the Duplicati UI, as my graphical skills are limited. 
It will also make it very easy to write a remote monitor tool, or make a plugin 
to a company's existing monitor system. And html/css/ajax seems to go well with 
Win8.

To allow for a local executing Duplicati (portable mode), the server/webserver 
can be hosted in the Duplicati process (like running the server.exe). The 
authentication is still not in place in my head, but I plan to use a file to 
check user permissions when running on the same machine, and I probably have to 
write multiple authentication systems for the supported platforms to get remote 
login support.

I think it will be cool :)

While I have been doing all this thinking and making my life hard, a programmer 
called Kenneth Bergeron (incidentially with the same first name as me), has 
developed and contributed a new GUI for Duplicati, complete with a service 
backend. In short, this allows you to run Duplicati as a service, albeit with a 
slightly different GUI look. This version is built Windows-only and takes 
advantage of many Windows features, such as integrated authentication, database 
logging, etc. The source code is in trunk, under the name Duplicati.Scheduler. 
No release has been made yet because he is currently testing it on some systems.

This does not mean that I will skip my idea, because I think the two things 
cater for different user groups, but the two versions will probably merge at 
some point.

As for my limited time, I do not mind other people taking over. You are of 
course encouraged to write me if you want to contribute some code, but there is 
so much more to running a project than just code. I strive to answer reported 
issues within 24 hours, not necesarily solve them that fast (that would be 
impossible), but just let people know that I take their reports seriously and 
also make sure that I request extra information before the setup/Duplicati gets 
thrown away. You can also join the google-groups/mailing-list and respond to 
people that ask things, which will free up more of my time too. If you want to, 
you can also get access to edit/create wiki pages, describing how to do stuff, 
debugging or commandline usage for instance. I have also created a Duplicati 
wikipedia page that you may want to edit, and generally "spread the word" which 
takes up much of my time. If you are more into graphics, I could use a cool 
logo or some banners that can be displayed on supporting sites. And then there 
is planning. As you can see the roadmap is not really updated, and you can take 
on the role of "release manager" and decide what bugs are important to fix and 
which should be left for the next version. And I am sure there are a hundred 
other things you can do. If I feel that such things are taken care of, I will 
focus on producing code, otherwise it will be a cover-all-bases approach from 
me.

Thank you all for using Duplicati and being interested enough to actually 
interact with me. (Did you really read all of that text?)

Original comment by kenneth@hexad.dk on 21 Sep 2011 at 7:47

[GoogleCodeExporter]

@Kenneth,

Thanks for the detailed response. i am definitely interested in a more central 
management system for multiple systems and I am willing to pay for it, but 
definitely please do not make any type of AIR or webserver type setup that will 
run on the clients end. Some backup solutions like vembu are doing it this way 
and i cant stand it. 

Thanks for your hard work and i do appreciate it, so as i am pretty sure you 
already have done, please dont take my comments as anything more than feedback.

@boldandbusted,

I dont think made a single DEMAND. I 100% understand open source projects and I 
work actively with many of them. What i did make was a comment and request. 
Some people call that feedback, which is good for projects. I am not a paying 
customer, so i obviously have no right to make any demands.

Original comment by mac...@gmail.com on 22 Sep 2011 at 8:54

[GoogleCodeExporter]

Hello Kenneth,

I really appreciate your great effort in creating this wonderful solution, 
integrating various ways of data backup. I have been using it widely for my 
personnel use to backup my laptop with my lab work. 

My comment here is in regard to your Comment #29 where you mentioned that a 
developer named Kenneth Bergeron has created a service based implementation of 
Duplicati and might publish it to open source. Do you have any idea or did he 
mention to you, when it might become available? Please let me know if you know 
already.

Thanks and best regards.

Original comment by rishabh....@gmail.com on 22 Dec 2011 at 10:59

[GoogleCodeExporter]

Now that we released Duplicati 1.3 we focus on 2.0. As far as I know Kenneth 
has already a Duplicati service running but there is no UI yet. As soon as 
there is something we can show, we'll put something online. 

Original comment by rst...@gmail.com on 23 Dec 2011 at 9:40

[GoogleCodeExporter]

The work that Kenneth Bergeron has done is actually published as open source.
But there are no published binaries, because Bergeron wants to work/test it 
some more.

Just check out the current trunk, and open "Duplicati Scheduler.sln".
You need Visual Studio installed (VS Express works fine), to open and build the 
project.

Last I tested, there was a .ico file missing but apart from that the project 
builds and runs.

And yes, I do have a no-UI version of Duplicati running which exposes a web 
service, and I have started work on the UI, but unless you want to do actual 
development, don't look at it yet :).

Original comment by kenneth@hexad.dk on 23 Dec 2011 at 10:13

[GoogleCodeExporter]

There is a UI; but, no installation.
It has some testing done.  
I have been using it for several months.
Please download it and let me know how it works.

"Just check out the current trunk, and open "Duplicati Scheduler.sln".
You need Visual Studio installed (VS Express works fine), to open and build the 
project."

Cheers, Ken

Original comment by NormanNa...@gmail.com on 9 Feb 2012 at 7:32

[GoogleCodeExporter]

So...still nothing, hmm?  It's been a year since the last update and we're 
still waiting for a way for Duplicati to run without having to be logged in.

What's the status?

Original comment by taume...@gmail.com on 26 Feb 2013 at 2:22

[GoogleCodeExporter]

There has been quite some progress, but still not near useful status.

The server component is largely complete, and runs jobs without any UI attached.
The user interface has been through some rewrites, but is still some way from 
being able to add or edit jobs.
We have had a good contribution that helped produce the basic dialog needed to 
edit/add a backup, but it needs some work still. We've been contacted by a few 
people wanting to help out, and we do our best to let them.

I have had much less time than anticipated, so my contributions to the project 
in the last year has been fairly limited, mostly responding to requests (such 
as this). I have side-tracked a bit and started working on a new-and-improved 
storage format, that solves the issues with long chains, time-consuming full 
backups, de-duplication etc. I need this part for my own personal needs, so 
once this is done, my own priority for the project returns to the UI.

If anyone wants to work on the new UI, simply download the source, and open 
with either Visual Studio (express), or MonoDevelop. Set the 
Duplicati.GUI.TrayIcon as startup project, and click "Run".

You can now let Duplicati open the webpage that is the main user interface, and 
start editing the extjs files. If you want commit acces, or would like to 
coordinate your work, write me an email.

I have attached the mockup that René has been working on, so you can see where 
we are heading.
I have also attached some screenshots of where we are. The *** marks stuff that 
is not done.

Original comment by kenneth@hexad.dk on 26 Feb 2013 at 10:45

Attachments:

• [Main Window - Actions 2012-09-30.png](https://storage.googleapis.com/google-code-attachments/duplicati/issue-9/comment-36/Main Window - Actions 2012-09-30.png)
• Screenshot2.png
• Screenshot4.png

[GoogleCodeExporter]

Any progress since februry regarding this?

I just discover Duplicati which is a great software to make some backup but the 
service/daemon is really missing @ the moment.

Thx :)

Original comment by f.deguel...@gmail.com on 28 Aug 2013 at 8:55

[GoogleCodeExporter]

We chose to postpone the GUI work, because we found that the current
version of Duplicati has some drawbacks, and we expect the GUI work to take
quite some time. The changes that are required to address these drawbacks
has wide-reaching implications, and we decided that it would be better to
improve the base version (the CLI) before comitting to the GUI. With the
CLI you can do some basic scheduled backup runs, so it can cater for our
own personal needs, and make up for the missing service functionality.

In short: No GUI updates right now, but a completely redesigned core with a
new storage engine and a new CLI.

Regards, Kenneth

Original comment by kenneth@hexad.dk on 28 Aug 2013 at 3:26

[GoogleCodeExporter]

Do you have any ETA for version 2.0 release.

Original comment by haa...@gmail.com on 4 Sep 2013 at 7:37

[GoogleCodeExporter]

No ETA.

We work as fast and as much as possible, but we have paid day jobs.

Anyone interested in speeding things up is most welcome to join the team.
The GUI needs someone with skills in HTML/CSS/Javascript.
We are currently using Sencha (previously known as ExtJS) as the JS framework, 
but we can discuss that part.
Just drop an email directly to me or René if you want in :)

Original comment by kenneth@hexad.dk on 4 Sep 2013 at 11:31