search

bcpierce00 / unison

Unison file synchronizer
GNU General Public License v3.0
4.16k stars 234 forks source link

failure syncing from windows: input_value_from_block: bad object #166

Closed cepa closed 4 years ago

cepa commented 6 years ago

Hi,

I'm experimenting with syncing files between a windows host and linux host. Went already through OCaml/Unison building process and have two binaries with the same version (2.51.2) built with the same OCaml compiler (4.04.0). Unison on windows built with Mingw64 from MSYS2.

Unfortunately, I'm getting this error while trying to sync files:

unison -auto -batch -perms=0 -sshargs "-i secret.pem -o StrictHostKeyChecking=no" ./ ssh://test@test.host//home/test/ Unison 2.51.2 (ocaml 4.04.0): Contacting server... Unison server failed: Fatal error: Fatal error during unmarshaling (input_value_from_block: bad object), possibly because client and server have been compiled with differentversions of the OCaml compiler. Fatal error: Lost connection with the server

the unison version on the remote host: unison version 2.51.2 (ocaml 4.04.0)

Any ideas how can I fix this?

thanks

gdt commented 4 years ago

See #375, and purge every single copy of unison from all machines, except the one you intend, and run unison with -version. If you are still having an issue please use the list.

glondu commented 4 years ago

the unison version on the remote host: [...]

Are you sure the right binary is invoked on the remote host? I don't see any -servercmd option in your invokation...

purge every single copy of unison from all machines

Also, purge any archive (file starting with ar) and fingerprint cache (file starting with fp) from your .unison directory.

jebej commented 4 years ago

I am also running into this issue on Windows 10, with both machines running powershell-openssh. I have installed the exact same binaries on each machine from the github release.

Unison 2.51.3 (ocaml 4.11.1): Contacting server...
Unison server failed: Fatal error: Fatal error during unmarshaling (input_value_from_block: bad object),
possibly because client and server have been compiled with different versions of the OCaml compiler.
Fatal error: Lost connection with the server
gdt commented 4 years ago

Is anyone else having this issues still?

@jebej Thanks for being careful about matching binaries. Can you describe if this is initial sync, if you have old archive files, how many files, how much data, etc?

jebej commented 4 years ago

This is an initial sync. I'm not sure what "archive files" are but I suppose I don't have them. I tried two different folders and got the same error. One was a git repo with a few text files, the other a directory with PDFs.

glondu commented 4 years ago

@jebej Just to be sure: does a regular ssh connection work? (with the same ssh used by unison)

jebej commented 4 years ago

Yes. Is there any way to increase verbosity to see what is happening?

gdt commented 4 years ago

Another case of this was reported on the mailinglist. The user confirmed that "ssh linuxhost unison -version" showed the right version and that removing /usr/bin/unison led to "unison not found".

If this works for anyone, suggestions about what might be different in the working/not-working case would be helpful.

edersasch commented 4 years ago

Here is a "-debug=verbose" output of a run initiated on linux contacting windows.


sascha@mi551:~$ /home/sascha/opt/unison-v2.51.3+ocaml-4.08.1+x86_64.linux/bin/unison-gtk2 -debug=verbose

(unison-gtk2:19355): GLib-GIO-CRITICAL **: 17:12:48.178: g_dbus_proxy_new: assertion 'G_IS_DBUS_CONNECTION (connection)' failed
[startup] Preferences:
ui = graphic
host = 
server = false
prefsdocs = false
doc = 
version = false
silent = false
dumbtty = false
testserver = false
showprev = false
selftest = false
confirmmerge = false
retry = 0
repeat = 
contactquietly = false
key = 
label = 
expert = false
height = 15
auto = false
maxthreads = 0
maxsizethreshold = -1
prefer = 
force = 
sortnewfirst = false
sortbysize = false
keeptempfilesaftermerge = false
diff = diff -u CURRENT2 CURRENT1
copyonconflict = false
backupdir = 
maxbackups = 2
backups = false
backupsuffix = 
backupprefix = .bak.$VERSION.
backuploc = central
copymax = 1
copyquoterem = default
copythreshold = -1
copyprogrest = rsync --partial --append-verify --compress
copyprog = rsync --partial --inplace --compress
rsync = true
fastcheck = default
ignorelocks = false
dumparchives = false
showarchive = false
rootsName = 
ignorearchives = false
fastercheckUNSAFE = false
fat = true
allHostsAreRunningWindows = false
someHostIsRunningWindows = false
ignore = Path {Public/Pictures/S9}
ignore = Path {Public/Pictures/*unbearbeitet}
ignore = Path {Public/Documents/Alexander/boardgames/.vs}
ignore = Name {CMakeLists.txt.user}
ignore = Name {desktop.ini}
ignore = Name {Thumbs.db}
ignore = Path {Public/Music/Sample Music}
ignore = Path {Public/Documents/Eigene Videos}
ignore = Path {Public/Documents/Eigene Musik}
ignore = Path {Public/Documents/Eigene Bilder}
ignore = Path {Public/Documents/My Videos}
ignore = Path {Public/Documents/My Pictures}
ignore = Path {Public/Documents/My Music}
confirmbigdel = true
batch = false
path = Public/Documents
path = Public/Pictures
root = ssh://eders@192.168.178.54/C:/Users/
root = /home/sascha/opt/unison-prime14b
killserver = false
halfduplex = false
stream = true
addversionno = false
servercmd = "C:/Users/eders/opt/unison-v2.51.3+ocaml-4.08.1+x86_64.windows/bin/unison.exe"
sshargs = 
rshargs = 
rshcmd = rsh
sshcmd = ssh
xferbycopying = true
sshversion = 
clientHostName = mi551
ignoreinodenumbers = false
links-aux = true
links = default
times = false
group = false
owner = false
numericids = false
dontchmod = false
perms = 1023
watch = true
rsrc-aux = false
rsrc = default
maxerrors = 1
follow = Path {Public/*}
unicodeCS = false
unicodeEnc = false
unicode = default
someHostIsInsensitive = false
ignorecase = default
timers = false
terse = false
logfile = /home/sascha/unison.log
log = true
debugtimes = false
debug = verbose
addprefsto = 
[remote] Shell connection: ssh (ssh, -l, eders, 192.168.178.54, -e, none, "C:/Users/eders/opt/unison-v2.51.3+ocaml-4.08.1+x86_64.windows/bin/unison.exe", -server)
[remote_emit+] grab: Unison 2.51 with OCaml >= 4.01.2\n\000\000\000\000\017
[remote+] Waiting for next message
[remote+] Received the write permission
[remote+] Waiting for next message
[remote+] send [negociateFlowControl-args] '\132\149\166\190\000\000\000\001\000\000...' 21 bytes
[remote+] send [rsp] '\132\149\166\190\000\000\000\022\000\000...' 42 bytes
[remote+] Sending request (id: 2)
[remote+] send [canonizeOnServer-args] '\132\149\166\190\000\000\000\r\000\000...' 33 bytes
[remote+] send [rsp] '\132\149\166\190\000\000\000\018\000\000...' 38 bytes
[remote+] Sending request (id: 3)
[remote+] Sending write token
[remote_emit+] dump: \002\000\000\000\205-\000\000\000\167rsp\132\149\166\190\000\000\000\022\000\000\000\000\000\000\000\t\000\000\000\006\1464negociateFlowControl.\000\000\000\005negociateFlowControl-args\132\149\166\190\000\000\000\001\000\000\000\000\000\000\000\000\000\000\000\000@\003\000\000\000,)\000\000\000-rsp\132\149\166\190\000\000\000\018\000\000\000\000\000\000\000\b\000\000\000\006\1460canonizeOnServer6\000\000\000\248canonizeOnServer-args\132\149\166\190\000\000\000\r\000\000\000\000\000\000\000\t\000\000\000\b\160\144)C:/Users/A\000\000\000\000\017
[remote_emit+] grab: \002\000\000\000\205\024\000\000\000\233rsp\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000-\000\000\000\167negociateFlowControl-res\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\003\000\000\000,\024\000\000\000\233rsp\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000=\000\000\000\140canonizeOnServer-res\240\t\220\b\000\000\000\000P\001\166\b\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000
[remote+] Message received (id: 2)
[remote+] receive 'rsp\000\000\000\000\000\000\000...' 24 bytes
Fatal error during unmarshaling (input_value_from_block: bad object),
possibly because client and server have been compiled with different versions of the OCaml compiler.
sascha@mi551:~$
ovidiopr commented 4 years ago

I also have the same problem, connecting from a Windows machine to a Linux server. I have tried with unison v2.51.3 and several versions of OCAML (always identical in both machines) and get the error "Unison server failed: Fatal error: Fatal error during unmarshaling (input_value_from_block: bad object), possibly because client and server have been compiled with different versions of the OCaml compiler."

gdt commented 4 years ago

@edersasch @ovidiopr What is your locale setting? I have a crazy not-supported-by-evidence thought that perhaps there is some sort of unicode/utf-8/whatever translation going on by ssh. Can you try LC_ALL=C on both ends (I have no idea how to do that on Windows) and see if that changes anything?

edersasch commented 4 years ago

To exclude ssh problems I started unison from a windows git installation's msys bash as a socket server: LANG=C LC_ALL=C unison -socket 49152. Then I tried to connect to it from linux: LANG=C LC_ALL=C /home/sascha/opt/unison-v2.51.3+ocaml-4.08.1+x86_64.linux/bin/unison-gtk2 -debug=verbose bastel socket://192.168.178.54:49152/bastel. I also tried to match the default windows locale in linux, but I always get Fatal error during unmarshaling (input_value_from_block: bad object).

I can use socket and ssh on these machines with unison 2.48.4 and 2.48.15 without problems.

sascha@mi551:~$ LANG=C LC_ALL=C /home/sascha/opt/unison-v2.51.3+ocaml-4.08.1+x86_64.linux/bin/unison-gtk2 -debug=verbose bastel socket://192.168.178.54:49152/bastel
[startup] Preferences:
ui = graphic
host = 
server = false
prefsdocs = false
doc = 
version = false
silent = false
dumbtty = false
testserver = false
rest = socket://192.168.178.54:49152/bastel
rest = bastel
showprev = false
selftest = false
confirmmerge = false
retry = 0
repeat = 
contactquietly = false
key = 
label = 
expert = false
height = 15
auto = false
maxthreads = 0
maxsizethreshold = -1
prefer = 
force = 
sortnewfirst = false
sortbysize = false
keeptempfilesaftermerge = false
diff = diff -u CURRENT2 CURRENT1
copyonconflict = false
backupdir = 
maxbackups = 2
backups = false
backupsuffix = 
backupprefix = .bak.$VERSION.
backuploc = central
copymax = 1
copyquoterem = default
copythreshold = -1
copyprogrest = rsync --partial --append-verify --compress
copyprog = rsync --partial --inplace --compress
rsync = true
fastcheck = default
ignorelocks = false
dumparchives = false
showarchive = false
rootsName = 
ignorearchives = false
fastercheckUNSAFE = false
fat = false
allHostsAreRunningWindows = false
someHostIsRunningWindows = false
confirmbigdel = true
batch = false
root = socket://192.168.178.54:49152/bastel
root = bastel
killserver = false
halfduplex = false
stream = true
addversionno = false
servercmd = 
sshargs = 
rshargs = 
rshcmd = rsh
sshcmd = ssh
xferbycopying = true
sshversion = 
clientHostName = mi551
ignoreinodenumbers = false
links-aux = true
links = default
times = false
group = false
owner = false
numericids = false
dontchmod = false
perms = 1023
watch = true
rsrc-aux = false
rsrc = default
maxerrors = 1
unicodeCS = false
unicodeEnc = false
unicode = default
someHostIsInsensitive = false
ignorecase = default
timers = false
terse = false
logfile = /home/sascha/unison.log
log = true
debugtimes = false
debug = verbose
addprefsto = 

(unison-gtk2:2501): GLib-GIO-CRITICAL **: 22:00:39.385: g_dbus_proxy_new: assertion 'G_IS_DBUS_CONNECTION (connection)' failed
[remote_emit+] grab: Unison 2.51 with OCaml >= 4.01.2\n\000\000\000\000\017
[remote+] Waiting for next message
[remote+] Received the write permission
[remote+] Waiting for next message
[remote+] send [negociateFlowControl-args] '\132\149\166\190\000\000\000\001\000\000...' 21 bytes
[remote+] send [rsp] '\132\149\166\190\000\000\000\022\000\000...' 42 bytes
[remote+] Sending request (id: 2)
[remote+] send [canonizeOnServer-args] '\132\149\166\190\000\000\000\n\000\000...' 30 bytes
[remote+] send [rsp] '\132\149\166\190\000\000\000\018\000\000...' 38 bytes
[remote+] Sending request (id: 3)
[remote+] Sending write token
[remote_emit+] dump: \002\000\000\000\205-\000\000\000\167rsp\132\149\166\190\000\000\000\022\000\000\000\000\000\000\000\t\000\000\000\006\1464negociateFlowControl.\000\000\000\005negociateFlowControl-args\132\149\166\190\000\000\000\001\000\000\000\000\000\000\000\000\000\000\000\000@\003\000\000\000,)\000\000\000-rsp\132\149\166\190\000\000\000\018\000\000\000\000\000\000\000\b\000\000\000\006\1460canonizeOnServer3\000\000\000\221canonizeOnServer-args\132\149\166\190\000\000\000\n\000\000\000\000\000\000\000\b\000\000\000\007\160\144&bastelA\000\000\000\000\017
[remote_emit+] grab: \002\000\000\000\205\024\000\000\000\233rsp]\000\\\000u\000@\000\\\000h\000 \000\\\000[\000\\\0000-\000\000\000\167negociateFlowControl-resi\000c\000\000\000P\000W\000D\000=\000C\000:\000/\000U\003\000\000\000,\024\000\000\000\233rspI\000T\000E\000C\000T\000U\000R\000E\000=\000A\000MJ\000\000\000WcanonizeOnServer-res\240!\r\n\000\000\000\000P\001\138\b\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000
[remote+] Message received (id: 2)
[remote+] receive 'rsp]\000\\\000u\000@...' 24 bytes
Fatal error during unmarshaling (input_value_from_block: bad object),
possibly because client and server have been compiled with different versions of the OCaml compiler.
sascha@mi551:~$
tillschaefer commented 4 years ago

I can confirm the issue running the windows client and a gentoo server. Tested Ocaml 4.9.0 and 4.11.1 on the server side with client builds for ocaml 4.9.1 and 4.11.1. Other linux clients work fine. Previous windows build Unison 2.48.4 (Jul 28, 2016) from https://www.irif.fr/~vouillon/unison/ worked flawless, but i had to upgrade ocaml on the server side.

I am using openssh from windows 10 on the client side.

tleedjarv commented 4 years ago

From the debug logs you provided, it is clear that the received data is mangled.

To be able to investigate this further, please everyone experiencing this problem do the following: 1) Make sure you provide the verbose debug log from both machines. 2) On Windows, try the 32-bit binary and report back the results.

tillschaefer commented 4 years ago 
2. On Windows, try the 32-bit binary and report back the results.

I am unable to find any 32 bit binaries here: https://github.com/bcpierce00/unison/releases

gdt commented 4 years ago

Look at https://github.com/bcpierce00/unison/releases/download/v2.51.3/unison-v2.51.3+ocaml-4.10.1+i386.windows.zip

Keep in mind that "32-bit" is not a specific kind of computer, and only describes a CPU instruction set if you have already decided that all computers are within one family of CPUs from Intel :-) More seriously, there are multiple other CPU types that have 32-bit word sizes, including VAX, 68000, sparc, powerpc, (the aarch32 variant of) arm, mips, and so on -- so "32 bit" is not used as label to mean "Intel x86 32 bit".

gdt commented 4 years ago

Can anyone report that syncing from windows to anything else (or windows to remote windows) over ssh, or over socket, does work with the release binaries, in either i386 (32-bit) or x86_64 (64-bit) mode? I would think so, but I'm not actually seeing that stated. If so, please state your locale settings.

edersasch commented 4 years ago

The problem seems to happen only with 64 bit windows build.

Working setup, no problems here:

The broken setup starts a win 64 server and on the same machine using localhost a win 64 client:

Unfortunately the server creates no debug output:

Unison server failed: Fatal error: Fatal error during unmarshaling (input_value_from_block: bad object),
possibly because client and server have been compiled with different versions of the OCaml compiler.

The server terminates after this output. The client of course loses the connection then:

[startup] Preferences:
ui = graphic          
host =                
server = false        
prefsdocs = false     
doc =                 
version = false       
silent = false        
dumbtty = false       
testserver = false
rest = socket://127.0.0.1:49152/dest
rest = src
showprev = false
selftest = false
confirmmerge = false
retry = 0
repeat =
contactquietly = false
key =
label =
expert = false
height = 15
auto = false
maxthreads = 0
maxsizethreshold = -1
prefer =
force =
sortnewfirst = false
sortbysize = false
keeptempfilesaftermerge = false
diff = diff -u CURRENT2 CURRENT1
copyonconflict = false
backupdir =
maxbackups = 2
backups = false
backupsuffix =
backupprefix = .bak.$VERSION.
backuploc = central
copymax = 1
copyquoterem = default
copythreshold = -1
copyprogrest = rsync --partial --append-verify --compress
copyprog = rsync --partial --inplace --compress
rsync = true
fastcheck = default
ignorelocks = false
dumparchives = false
showarchive = false
rootsName =
ignorearchives = false
fastercheckUNSAFE = false
fat = false
allHostsAreRunningWindows = false
someHostIsRunningWindows = false
confirmbigdel = true
batch = false
root = socket://127.0.0.1:49152/dest
root = src
killserver = false
halfduplex = false
stream = true
addversionno = false
servercmd =
sshargs =
rshargs =
rshcmd = rsh
sshcmd = ssh
xferbycopying = true
sshversion =
clientHostName = WIN-V2KLIPVS4L5
ignoreinodenumbers = false
links-aux = true
links = default
times = false
group = false
owner = false
numericids = false
dontchmod = false
perms = 1023
watch = true
rsrc-aux = false
rsrc = default
maxerrors = 1
unicodeCS = false
unicodeEnc = false
unicode = default
someHostIsInsensitive = false
ignorecase = default
timers = false
terse = false
logfile = C:\Users\Alexander\unison.log
log = true
debugtimes = false
debug = verbose
addprefsto =
Unison 2.51.3 (ocaml 4.08.1): Contacting server...
[remote_emit+] grab: Unison 2.51 with OCaml >= 4.01.2\n\000\000\000\000\017
[remote+] Waiting for next message
[remote+] Received the write permission
[remote+] Waiting for next message
[remote+] send [negociateFlowControl-args] '\000\000\000\000\000\000\000\000\000
\000...' 21 bytes
[remote+] send [rsp] ' \220\173\b\000\000\000\000P\001...' 42 bytes
[remote+] Sending request (id: 2)
[remote+] send [canonizeOnServer-args] '\000\000\000\000\000\000\000\000\000\000
...' 28 bytes
[remote+] send [rsp] '\000\000\000\000\000\000\000\000\000\000...' 38 bytes     
[remote+] Sending request (id: 3)
[remote+] Sending write token
[remote_emit+] dump: \002\000\000\000\205-\000\000\000\167rsp \220\173\b\000\000
\000\000P\001\173\b\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\
000\000\000\000\000\000\000\000\000\000\000\000\000\000\000.\000\000\000\005nego
ciateFlowControl-args\000\000\000\000\000\000\000\000\000\000\000\000\000\000\00
0\000\000\000\000\000\000\003\000\000\000,)\000\000\000-rsp\000\000\000\000\000\
000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\
000\000\000\000\000\000\000\000\000\000\000\000\0001\000\000\000 canonizeOnServe
r-args\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\0
00\000\000\000\000\000\000\000\000\000\000\000\000\000\017
Fatal error: Lost connection with the server

Trying the windows localhost setup with the 32 bit unison.exe also works without problems.

gdt commented 4 years ago

So our current belief is:

If you have any evidence that doesn't fit the above summary, please add it here.

tleedjarv commented 4 years ago

Please test with the 64-bit binaries from PR #419.

nareddyt commented 4 years ago

Thanks for the quick fix! I can confirm the latest windows binary in PR #419 solves the issue. As expected, I didn't have to update unison on my ubuntu server, only my windows client.

edersasch commented 4 years ago

I can confirm that unison-0c1fff95.ocaml-4.08.1.windows-latest from #419 fixes the issue. I successfully tested ssh from linux without modifying the linux version and windows localhost socket. Thanks!

ovidiopr commented 4 years ago

Sorry, where can I download the binaries corresponding to this fix? Best regards, Ovidio

On Mon, Nov 2, 2020 at 6:54 PM Greg Troxel notifications@github.com wrote:

Closed #166 https://github.com/bcpierce00/unison/issues/166 via #419 https://github.com/bcpierce00/unison/pull/419.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/bcpierce00/unison/issues/166#event-3949375500, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACNOR2LUUUXY3DH74XZNPH3SN3W3TANCNFSM4ETEADEA .

tleedjarv commented 4 years ago

Sorry, where can I download the binaries corresponding to this fix?

You can always get non-release binaries at https://github.com/bcpierce00/unison/actions - just pick a completed build that suits your needs (if you start from a PR then "Checks" is a shortcut). For this fix: https://github.com/bcpierce00/unison/actions/runs/342181358

ovidiopr commented 4 years ago

Thank you very much! I can confirm that 'unison-efbb6aa5.ocaml-4.05.0.windows-latest' works flawlessly from a Windows 10 client to a Linux server.

On Mon, Nov 2, 2020 at 7:22 PM Tõivo Leedjärv notifications@github.com wrote:

Sorry, where can I download the binaries corresponding to this fix?

You can always get non-release binaries at https://github.com/bcpierce00/unison/actions - just pick a completed build that suits your needs (if you start from a PR then "Checks" is a shortcut). For this fix: https://github.com/bcpierce00/unison/actions/runs/342181358

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/bcpierce00/unison/issues/166#issuecomment-720644254, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACNOR2K3XOKWT3CZOQTC4PDSN32GNANCNFSM4ETEADEA .

schminkel commented 3 years ago

I can also confirm that this issue has been fixed with version: --> unison-v2.51.4_rc1 + ocaml-4.08.1 using Window 10 & Linux for a unison sync

I got error message: Unison server failed: Fatal error: Fatal error during unmarshaling (input_value_from_block: bad object),..

When trying this combinations: unison-v2.51.3 + ocaml-4.11.1 unison-v2.51.3 + ocaml-4.08.1

Thanks, Thorsten