One-shot mode

[lheckemann]

I want a tool that behaves like rsync, but with the advantages of syncthing:

Property	Syncthing currently	rsync
Exit after copying	:x:	:heavy_check_mark:
Stateless	:x:	:heavy_check_mark:
Fast	:heavy_check_mark:	:x:
Secure	:heavy_check_mark:	:question:
Connection magic	:heavy_check_mark:	:x:

Exit after copying

This provides a notion of a transfer being "complete", which can be helpful when copying to removable media for instance. "one-shot syncthing" could exit once no more differences exist between both ends, which is a little unlike rsync, but perhaps more useful (the state of the source and destination aren't necessarily consistent after running rsync!)

Stateless

rsync doesn't store state (besides the files being copied) persistently. Syncthing stores crypto keys, mutable config, things for the web interface, and folder indices. While persistent keys can be helpful (and would be useful for this tool as well), ad-hoc key generation would be nice when transferring between two machines that shouldn't have a persistent trust relationship to one another.

Fast

rsync goes through files and directories one-by-one on a single thread. This can bottleneck performance in multiple ways: CPU not being fast enough, RAID not being fully utilised because only 3 disks are accessed at a time, …

Syncthing is already capable of using multiple threads.

Secure

rsync can be tunneled via SSH or other rsh-like programs, and relies entirely on this for security.

Syncthing already uses TLS/DTLS to secure all its communications.

Connection magic

rsync requires the server address to be known, ports open in the firewall, and a direct connection between the server and the client to be possible.

Syncthing already has discovery servers and relaying functionality, which make this easier.

---

So I'm hoping the two gaps, "exit after copying" and "stateless", can be filled in syncthing — and I'd wager that these are easier problems than the others!

In terms of operation, I'm imagining maybe something like this:

[user@paris]$ syncthing-oneshot --send ./directory
My ID: MIMKZCI-KCDDFOI-PYGHL3S-5NAV6PP-XCFNVQI-OHA5SI3-OHEMJVL-LVWI2QE
Connection request from 37LJM3S-NH2OHPT-SL4L3RD-VQX3H2Y-IUUI2PM-SDHCFWP-ESALIEP-Y5VBRAF, accept (y/n)? y
[insert progress indicator here]

[user@tokyo]$ syncthing-oneshot --recv ./directory --id MIMKZCI-KCDDFOI-PYGHL3S-5NAV6PP-XCFNVQI-OHA5SI3-OHEMJVL-LVWI2QE
My ID: 37LJM3S-NH2OHPT-SL4L3RD-VQX3H2Y-IUUI2PM-SDHCFWP-ESALIEP-Y5VBRAF
[insert progress indicator here]

and afterwards, directory has the same contents on tokyo as on paris.

---

Is this functionality that $maintainer could see being part of syncthing?

[calmh]

I could see it but I think it's more difficult than you expect. The asynchronous nature of Syncthing means it's hard to be sure when we know about all remote changes, and we'd have to wait for connections to all known devices.

[AudriusButkevicius]

Agree, I think this is out of scope for syncthing. Rsync is what you want, it's fast and secure if used properly and you can use side tools for connection magic.

We should close as far as I am concerned.

[calmh]

I think this could be implemented cleanly, to the extent it's possible to implement at all. I'm thinking a new service, which monitors the sync state and somehow determines that it's happy, and then exits with an ErrFatal or whatever it's called that causes everything to shut down cleanly. Boom, one shot mode. So I think this can live. Possibly forever, but that's the way of some things.

[AudriusButkevicius]

How is that different from just running rsync? Unison? Whats the sell?

Sure, we can do things, but you'd hope you bring something new/better to the game and this is neither, it would be a hack, that I suspect work half arsed.

[lheckemann]

@AudriusButkevicius how do I use rsync properly to make it fast? It's not passed ~1600Mbit/s when running locally on a RAID that should support a lot more, which I think is due to it being limited to reading and writing one file, linearly, at a time, or to being limited on the CPU side…

[AudriusButkevicius]

I don't think you could achieve 1600Mbit/s with syncthing, so you already beat it.

Syncthing also mostly copies one or two files at a time.

You can always run multiple rsync processes.

I am still of the opinion that it is not worth implementing this in syncthing, as it would just be a slower/worse implementation of other tools that already do this, so I hope to be enlightened here.

[calmh]

I think there's a legitimate case for usually using Syncthing in the normal manner, but at some point wanting to run it in one shot mode. Just before folding up the laptop and heading out, for example. Or to get/push the latest changes while on expensive roaming internet.

[imsodin]

Providing my 2cents:

There's a plethora of sync tools, clearly none does it all, otherwise everyone would just use that. Syncthing seems to be doing pretty well in quite a few aspects, so people want it to also excel in the other aspects, in the hopes of eventually having a "Eierlegende Wollmilchsau" (reaching sync utopia). I also think this particular aspect (whatever one-shot exactly is) is a sensible thing to want.

I still think it shouldn't be part of Syncthing, at least not in a first iteration, for an entirely different reason: In practice simplicity and maintainability are a major criteria for when we accept contributions. While such functionality wouldn't affect sensitive internals of Syncthing, it would still grow the "support surface" extensively. I thus would rather see something like this as a separate project. That might even live in the Syncthing umbrella, could definitely reuse most of the code (I'd be super happy to help make ./lib/... more reusable), and maybe eventually be merged into Syncthing. It just doesn't provide the implicit benefits you get with Syncthing (I think it's fair to say our level of community and maintainer support is pretty good).

We could also have a ticket/wiki for "Extensions", where ideas like this are collected. Basically a list of things that one could do in/with/on top of Syncthing but that we don't currently consider for Syncthing itself. Or even leave tickets like this open add label them with e.g. extensions/monumental/..., might make it more discoverable and thus less likely to get duplicates issues every now andthen.

[AudriusButkevicius]

Sooo ... close is what I'm hearing ? 🙃

[calmh]

I like the idea of a contrib script or something to do this, via the api.

[Kriechi]

I have the following (very crude but working) script to do pretty much what you are describing:

When my system starts, syncthing starts as systemd service automatically, and the following script gets started separately. It will connect via the HTTP API and keep querying one specific folder that I want to fully sync, before the loop exits and I shutdown the whole system. Should be easy though to just loop around all existing folders to wait for all of them.

#!/usr/bin/env bash

HOST='http://127.0.0.1:8384'
KEY='secret-pa33phra3se'
FOLDER_ID="folder-id-to-one-shot-check"

while true ; do
  NEW=$(curl -s -H "X-API-Key: ${KEY}" "${HOST}/rest/db/status?folder=${FOLDER_ID}" | jq '"\(.state)@\(.stateChanged)"' -r)
  NEW_STATE=$(echo "${NEW}" | sed -E 's/@.+//')
  if [[ "${NEW_STATE}" == "idle" ]] ; then
    break
  fi
  sleep 0.5
done

shutdown now

[daiaji]

syncthing's sync strategy for removable devices is bugging me.

1. Syncthing still continues to read the directory on the removable device after the synchronization is complete. Under Windows, I cannot safely remove the USB flash drive without closing Syncthing.
2. For removable devices, the operating system may not use a fixed path when mounting, and syncthing should use volume labels or UUIDs to identify these devices.
3. To be honest, mobile hard drives and USB flash drives are usually produced with the worst materials (the worst flash memory particles and the worst hard disk platters), which also means that these storage media are usually more easily damaged, so timely Backing up data from removable devices to NAS is very important, and usually they are also slower, so incremental backups are also important (full backups are too slow).

[mildred]

This is an interesting enhancement idea.

The main reason I would want to use syncthing over rsync to sync using other means like that (it could be removable storage, or else) would be to have the sync respect the .stignore files and all other syncthing rules.

Could ideas like this be implemented as external tools, and use syncthing as an API? If there is an API, it shouldn't be too difficult to have a tool that instantiate a syncthing server with an in-memory fresh config, configure the two folders to sync using this in-memory config, and run the thing until the monitoring tells all is good. I don't know how that's feasable though in the current state.

[pgorod]

I am not sure if my request is exactly the same as the one described here, so I thought I'd ask. I often have to send big (2-3GB) files to people, my use case is a bit like the service provided by WeSend and similar websites.

Except that SyncThing can do it a lot better, because there is no cloud servers in between, there is no delay uploading (a huge advantage), and even downloads will likely be a lot faster than any shared server will provide in a free tier.

So what is currently lacking is just a stream-lined user-experience for the receiving side. If I could just send a link, and the app would start up, download the file, and disappear forever... that would be great. I know this is not easy to achieve but I believe it's doable, and it's such a basic thing that would have tons of people wanting to use it.

Thoughts?