search

haiwen / seafile

High performance file syncing and sharing, with also Markdown WYSIWYG editing, Wiki, file label and other knowledge management features.
http://seafile.com/
Other
12.25k stars 1.54k forks source link

HTTP/HTTPS sync #975

Closed nandovici closed 9 years ago

nandovici commented 9 years ago

Hi,

I am trying to run the https synchronization.

This is a slice of the ccnet.log on the client:

[12/17/14 18:35:07] ccnet-daemon.c(193): starting ccnet client 4.0.3
[12/17/14 18:35:07] ccnet-daemon.c(195): ccnet source code version    8bca42a29e944898c2be645e7c8a3d326f8c4084
[12/17/14 18:35:07] ../common/session.c(395): Listen on 127.0.0.1 13419
[12/17/14 18:35:07] ../common/session.c(267): Update pubinfo file
[12/17/14 18:35:08] ../common/session.c(375): Accepted a local client
[12/17/14 18:35:08] ../common/session.c(375): Accepted a local client
[12/17/14 18:35:08] ../common/session.c(375): Accepted a local client
[12/17/14 18:35:08] ../common/session.c(375): Accepted a local client
[12/17/14 18:35:08] ../common/session.c(375): Accepted a local client
[12/17/14 18:35:08] ../common/session.c(375): Accepted a local client
[12/17/14 18:35:08] ../common/processors/rcvcmd-proc.c(492): Add server 2e0924b1 cdm.seafile.cidiemme.net:10001
[12/17/14 18:35:08] ../common/connect-mgr.c(364): [Conn] Start outgoing connect to (null) 2e0924b194) 217.27.88.11:10001
[12/17/14 18:35:18] ../common/connect-mgr.c(220): [Conn] peer (null)(2e0924b194) connection fails

Thank you in advance.

kgbvax commented 9 years ago

Same story here, apparently.

freeplant commented 9 years ago

The ccnet.log is in-relavant. Please check seafile.log at the client side.

killing commented 9 years ago

There are 2 possible cause:

  1. Your libraries are created before 3.0 version
  2. You use a self-signed certificate on the server. You need to check the "Disable verify certificate" option in the client. And restart the client.
killing commented 9 years ago

As answered in http://stackoverflow.com/questions/12941703/use-curl-with-sni-server-name-indication using SNI requires an up-to-date version of libcurl and openssl. If you're using Linux, just check your system's library version. If you're using Windows or Mac, the bundled version is already new enough.

nandovici commented 9 years ago

I solved.

My libraries were created before 3.0 version!

Creating a fresh new library I can syncronize via HTTPS with no problem.

Thank you very much!

fenderle commented 9 years ago

Why do older libraries not sync through http? Is it possible to convert them or fix the problem somehow using config changes?

killing commented 9 years ago

@fenderle The old format lacks information to support the new protocol, like file timestamps. You can migrate the data to a new library by copying data on web. But history would be lost...

TimWolla commented 9 years ago

@fenderle The old format lacks information to support the new protocol, like file timestamps. You can migrate the data to a new library by copying data on web. But history would be lost...

Is it possible to provide an automated migration path? e.g. by setting the change time of every file to the commit time. Redownloading 20+ gigabytes using a small pipe is not fun :-( But I don't care about an automated migration on the server taking several hours.

palday commented 9 years ago

The automated path would be very useful, especially since the claim on the website is "Now we use HTTP/HTTPS sync only. The old TCP sync protocol will no longer be used. The using of port 10001 and 12001 becomes history. They are not needed at all." In light of this thread, this is patently false, and this is even noted in the source code.

TimWolla commented 9 years ago

@palday https://forum.seafile-server.org/t/migration-of-old-library-format-to-the-new-library-format/2094?u=timwolla

raimue commented 7 years ago

The linked guide for migrating an old Seafile library to the new format is no longer reachable. I am being stuck with some older libraries, but unable to find any more information on this. Was this forum post archived somewhere?

TimWolla commented 7 years ago

@raimue I don't have the contents of the post any more, but I can provide you the link of the migration software I wrote, but without giving any support or help whatsoever: https://github.com/TimWolla/seaf-fix

raimue commented 7 years ago

@TimWolla Thank you very much for sharing the tool, that allowed me some insight in what is required. While I got the tool to build, in the end I still decided it would be the future-proof solution to just give up on history and migrate these libraries by re-uploading the files. The only library I could not get converted was the "My Library Template". As I am not using it at the moment, I'll will just leave it like that for now.

shoeper commented 7 years ago

I found the text from the post in my email archive. Here is is:

This is from @TimWolla 

I finished converting my complete seafile installation a few days before and polished my program a little as well.
Manual
What my program can do:

    Converting an unencrypted library from the „V0“ to the „V1“ library format while preserving the version history

What my program cannot do:

    Converting an encrypted library (I do not need this myself)
    Brew some coffee
    Everything else

Please note:

    While I successfully converted my own unencrypted libraries I cannot guarantee that it won't delete all your precious data, keep backups and play safe.
    You have to shutdown seafile before starting a conversion (or at least ensure that the library that's being converted is not modified).
    It is pretty slow (this is an inherent property of the conversion process). Depending on the exact size of structure of the library it may take a few hours or even days.

How to use?

    Find the UUID of the library you want to convert (it's in the address bar of the web interface)
    Shut down seafile
    Find the root commit via: SELECT commit_id FROM Branch WHERE repo_id = '…' (insert your UUID here)
    Run ./seaf-fix /srv/seafile/seafile-data/storage /srv/seafile/seaf-fix/target/ UUID commit_id (you'll know that your paths are)
    Wait
    Wait a little bit more until something like this appears: Done, your new root is: b0d58206779cc1299a4dde13950dfe091cc1a704
    Ensure your backup is good, once again
    UPDATE Branch SET commit_id = 'b0d58206779cc1299a4dde13950dfe091cc1a704' WHERE repo_id = 'UUID' (adapt as needed)
    Repeat step 4 to 8 for every sub library: SELECT repo_id FROM VirtualRepo WHERE origin_repo = 'UUID'.
    You have to symlink /srv/seafile/seafile-data/storage/fs/SUBLIBRARY_UUID/ to /srv/seafile/seafile-data/storage/fs/UUID/ (as both share the same FS directory)
    rm -rf /srv/seafile/seafile-data/storage/commits/UUID/ (for every library you converted)
    rm -rf /srv/seafile/seafile-data/storage/fs/UUID/ (for every library you converted)
    Move every folder from /srv/seafile/seaf-fix/target/commits/ to /srv/seafile/seafile-data/storage/commits/
    If you have sublibraries: Merge the contents of the /srv/seafile/seaf-fix/fs/ folders into the folder with the UUID of the root library
    If you have sublibraries: Fix the base_commit inside VirtualRepo (you have to search the output of the parent library for that, sorry).
    Move /srv/seafile/seaf-fix/target/fs/UUID to /srv/seafile/seafile-data/storage/fs/UUID.
    Run seafile-fsck.sh (it should not find any issues)
    Run seafile-gc.sh --dry-run (it should not find any issues either)
    Start seafile again and check whether your library looks fine.
    Party

License?

MIT License.
Where to get it?

It comes in three prebuilt flavors. One version for servers with lots of RAM to spare (it uses up to 25 GB of RAM on my box, YMMV), another one for smaller boxes (about 2 GB of RAM, YMMV) and a third with only the minimal amount of RAM. You can adapt it yourself in the source code on this line: https://github.com/TimWolla/seaf-fix/blob/d5c362c34eb4f6c5bd88cf58d0d9eef3b137043b/seaf-fix.hs#L279 (6 Millionen = 25 GB, 400k = 2 GB). Obviously the more RAM you have to spare the faster it is. The binaries are static binaries built using GHC 7.10.1.20150710 inside a Docker container (Debian Jessie), they should work on every Linux out there, though.

Both the git tags, as well as the binaries can be verified using this GPG key:

pub   4096R/094168AF 2013-10-04
      Key fingerprint = 3033 55CF 8E33 D090 86CB  28F6 8FF7 5566 0941 68AF
uid                  Tim Düsterhus (git signing) <timwolla@bastelstu.be>
sub   4096R/127D597F 2013-10-04

Git repository: https://github.com/TimWolla/seaf-fix

25G RAM version: https://github.com/TimWolla/seaf-fix/releases/download/0.1.0.0/seaf-fix-6000000
2G RAM version: https://github.com/TimWolla/seaf-fix/releases/download/0.1.0.0/seaf-fix-400000
Minimal version: https://github.com/TimWolla/seaf-fix/releases/download/0.1.0.0/seaf-fix-0

Hashes:

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

6166144eb0eb6b68f7b94aa6ea5e7904c1494d5f72c3dde97a4c67e2cb8e9a97  seaf-fix-0
f131ad42c8940e9e4bdbc4814420e0ecc0876d70cf700cc5143b25f994b3246f  seaf-fix-400000
c44818d20cec4622bf04a59ce5f1fa6e5f235c1703dc4242f9eea381b33ebb4e  seaf-fix-6000000
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1

iQIcBAEBCAAGBQJWZNkyAAoJEI/3VWYJQWiv5W0P/ifns/qmvXWZ/tZr7zsYlkgG
mwAnh7iqeWrwFw0EAS9F2LwvVSffTdJenG6tSUsC1SQ5NnD7JI89zUxk1wN13Y+3
kO3bFaq5UE6GQbkpPVtSK9zi1F6TYWNdmkygkTE3L1/3wqIKRvuP2hGvyoI8ydY3
xqExbnpwSd6kI2+yy1OONJVR+A1ZmK6ZNdROP6AOcfxvPtYbqN8Zd6qptMeHbfQ0
4ujnwXtPB+zvleT7KfrfWyTZbEW6BL624Ox/FQzu9Dvn9S/uDQZPxQL5uMtfvWY2
A3OaZFCPqaZ+pyJYHz6cp/1iuaPQT94gV2BFEHXXgJNqtCc/5CS99Y3VAhDxpJ6U
aCKQEeRKkkjJXkBE2edd+KCOqnw+FJlgIzi4yTagziZ73Gtq+br8Dpv9raw2ypQv
xTE0dYR6zjKMvhPKKZ+59FwbZfWLfpqfyJnXSudnAN66jJyJr4RtGKZT/WF047Pf
Nm7HP5W3mJ7ftXZFqAi9TEbsdOdrd/DE1GA0KVVwC4wUJKvXq3Nq091C4/uL4Rkt
tpJQ0B+GVoYMYpBoJeZqUO5lZs2Mv4QjL8/DRF2GEl0FRDTfjC9bGv1YWfywCHob
guI1JEKY1WkWchMavOTO2jIZoLZZokhZq2RQDdaaBVODFnPkW4e/O4blD7cVD0xe
+xS1IZQOQQy1Oqp/TI3S
=v3hu
-----END PGP SIGNATURE-----

And here is one of the more relevant replies: (also by @TimWolla)

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Christian,

sure. I just uploaded a `seaf-fix-infty` with the following SHA-256 checksum to GitHub:

    6c5b77ba1a860d7aef9fdb87bc1dac788d053e3341d15b584781164f209fddc4  seaf-fix-infty

This version completely removes the clearing of the internal cache, therefore it’s memory
usage will grow without bounds. Make sure to not overwhelm your server and stop services
that should not be killed by the OOM killer (such as your database server).

Best regards
Tim
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1

iQIcBAEBCAAGBQJXctVTAAoJEI/3VWYJQWivM/8P/jBvUHEkO2NMJFpUxJl37BZc
kqPoSUo+E6cfi6gm2np1n3bU5f3Pt0fy0jTOO2MeeZehuO1bpsKsNYencN5Y5i9v
3M14JaMDYnKluoALWnJCGcew/Ncnf44/MseQpuSSyQogPmuq/Ww02sRw1N1wj54g
EWTAK7Kcv3hd5VUZb+fFRd66ivbqCdbeG5t2psZqsgziukdR2RidQcaJAeqjfXNX
oRpaCceQC/jzTX+1HGLjTVRGLHqE1/1dNEty9gv3cGjJX7rSZjuAi0C+bfxqjOEb
QEv38157SP3whEVsNVC7cQAHKA7bMvXV2nTVxREIj2OrZjFbuFNtQxoUFJK0n/NS
3ao+xthRCpP6etLpPv8g0pqnhXci9FlG+puYuTZk530E84Eb0C3E3ehOBn586WYF
zVaqYlmZETharo5UlEWXPRCagtJvj7hm1h8LX5H8UUpm2ukZm3UEkzi0aTcB2T2K
2N/lndCSqN12ZzdL01B5D3C4VUZ/67YotVIi89iV7VzwfhCi+jpe0ygko3iYjsOC
606SeMOtdDioIGpAFNVutXeALwXX7qSB4ZafDlTLIWIOL7g2Z0pVsE9RvU2KobRV
efxcl32ZEw09IzQjbWBfbBpCS/Xjzj2y+ajzcV6QyevHMfYJXT/ZPnCNbptfMmmg
2QKfqkwfoKK70flT5cFw
=nv14
-----END PGP SIGNATURE-----