search

beer4all / rclone

"rsync for cloud storage" - Google Drive, Amazon Drive, S3, Dropbox, Backblaze B2, One Drive, Swift, Hubic, Cloudfiles, Google Cloud Storage, Yandex Files
https://rclone.org
MIT License
0 stars 0 forks source link

error while copying with a local xrootd server (due to hash?) #21

Open kiderrien opened 4 years ago

kiderrien commented 4 years ago

I generated a local server with an adler32 checksum. When copying a file, this type of error is obtained :

client:

[userxroot@resticrclone3 rclone]$ rclone -vv copy /tmp/back2/adler xrootd:back2/p --checksum
2020/07/03 10:13:47 DEBUG : large: Hash:path= /tmp/back2/p/large
panic: mux: cannot find data waiter for id [0 0]

goroutine 187 [running]:
go-hep.org/x/hep/xrootd.(*cliSession).consume(0xc0002a4c30)
        /home/userxroot/go/pkg/mod/go-hep.org/x/hep@v0.26.0/xrootd/session.go:258 +0x3f7
created by go-hep.org/x/hep/xrootd.newSession
        /home/userxroot/go/pkg/mod/go-hep.org/x/hep@v0.26.0/xrootd/session.go:103 +0x2c6

Server :

00703 10:13:47 21170 XrootdXeq: .21345:25@localhost unbind 0:00:00
200703 10:13:47 21170 XrdPoll: Unable to exclude fd 25 during disc (0) event; link .21345:25@localhost; no such file or directory
200703 10:13:47 21170 XrootdXeq: .21345:31@localhost unbind 0:00:00
200703 10:13:47 21168 XrootdXeq: .21345:34@localhost disc 0:00:00
200703 10:13:47 21087 XrdLink: Unable to receive from .21345:19@localhost; connection reset by peer
200703 10:13:47 21172 XrdLink: Unable to receive from .21345:24@localhost; connection reset by peer
200703 10:13:47 21087 XrootdXeq: .21345:19@localhost disc 0:00:00 (link read error)
200703 10:13:47 21087 XrdPoll: Unable to exclude fd 19 during disc (0) event; link .21345:19@localhost; no such file or directory
200703 10:13:47 21172 XrootdXeq: .21345:24@localhost disc 0:00:00 (link read error)
200703 10:13:47 21172 XrootdXeq: .21345:23@localhost unbind 0:00:00
200703 10:13:47 21172 XrootdXeq: .21345:29@localhost unbind 0:00:00
200703 10:13:47 21172 XrdPoll: Unable to exclude fd 24 during disc (0) event; link .21345:24@localhost; no such file or directory

This error seems to come from the hash function.

When copying for a file only it works but there is a hash problem:

2020/07/03 11:09:12 DEBUG : test3: Using hash function with hash.Type= Adler32Hash
2020/07/03 11:09:12 DEBUG : test3: Hash:path= /tmp/back2/p/test3
2020/07/03 11:09:12 DEBUG : test3: Hash: o.Hash=  && Data=  XJob scheduled.

Data= XJob scheduled Normally Data = adler32 224d3ec3 This is the same problem with a local xrootd server configured with the md5 checksum

However the copy works well with the remote xrootd server using adler32.

kiderrien commented 4 years ago

It does not seem to succeed in finding the file that the path indicates when copying to the local server while the remote does not have this problem during a copy. The request for the checksum returns "�XJob scheduled" when it does not find the file.

kiderrien commented 4 years ago

ok. the issue is that currently go-hep/xrootd doesn't handle async responses (ie: kXR_attn server responses) at all.

See here.