Synology: XML error when server sends large (response-) XML (HTTPS only)

[danielbierstedt]

Your environment

Synology NAS with built in CardDAV server

TbSync version: 0.7.22 DAV-4-TbSync version: Provider für CalDAV & CardDAV 0.10 Thunderbird version: 60.3.3

[ ] Yes, I have installed the latest available (beta) version from

• https://github.com/jobisoft/TbSync/releases
• https://github.com/jobisoft/DAV-4-TbSync/releases and my issue is not yet fixed, I can still reproduce it.

Expected behavior

Contacts synced

Actual behavior

Throws XML error when syncing a contact with a profile pic

Steps to reproduce

...

To help resolving your issue, enable debug logging (TbSync Account Manager -> Help) and send me the debug.log via e-mail (use the title of your issue as subject of the email). Done

[jobisoft]

Just to make sure it is not already fixed, please try the beta: tbsync.jobisoft.de

[danielbierstedt]

Sorry to say that the beta version does not work at all. It looks a bit different than the images on your website, but however. When I want to enter the settings, TB freezes. I then have to force close.

[jobisoft]

Do you have installed the tbsyncBetaInstaller and now see 3 modules with [beta release channel] in your AddOn manager? I have heard from one other user, that not all modules have been updated by the betaInstaller. In that case you have to manually install all 3 XPI modules from here:

tbsync.jobisoft.de/beta

The beta is know to work.

[danielbierstedt]

Ok, I removed everything and installed the 3 modules from that website. I NOW see the [BETA] text. Sadly, it still does not work. It does not like the picture embedded in the vcard.

[jobisoft]

Ok, I will get back to this soon.

[jobisoft]

OK, I want to work on this starting this afternoon. I want to release a new stable version next week and this issue is one of the last bugs that needs to be fixed.

Any chance you could provide access to a test account which has such a contact which fails, so I can reproduce this 100%? Can you send me credentials via email?

[jobisoft]

I also would like to ask if this error occurs with HTTP connection to your server as well with HTTPS connection. Can you try with both?

[danielbierstedt]

Sorry for the delay. Will do the tests and provide a login with necessary detail tomorrow.

[jobisoft]

Ok, after all these investigations it looks like there is a general problem between Thunderbird and Synology.

Synology uses an HTTP 1.1 feature called chunked transfer encoding and while this seems to work on HTTP connections, it fails on HTTPS connections, if the XML send by the server is rather large.

From what I have seen, Thunderbird kills the connection before the final (?) chunk has arrived and thus the received XML is incomplete and cannot be evaluated.

There is no fix available yet and I was not able to trigger this error with a generic server which is sending chunks.

For the time being, the only "fix" for this is to use HTTP connections, which of course is not a valid option if you access your synology from outside (assuming inhouse data transfer does not need transport layer encryption, because your family members do not intend to spy on you).

[danielbierstedt]

ok, understood. Thanks for all your work. I opened a ticket at Synology to investigate. There is chinese new year until 10'th of Feb, so could take some time until the first answer. I'll keep you posted.

[jobisoft]

I think this is an issue with Thunderbird. I am trying to setup a test server to trigger this error, so I can report it.

[lweidmueller]

Hi, I send you today my logfiles from Win10 and MacOS. http doesn't work also. If you need testing support on Synology CardDAV, I could do some testing for the project.

Edit by jobisoft: Your log files indicate that your Server redirected your HTTP requests back to HTTPS, which of course triggered the same error. So the status is still, that it fails only with HTTPS.

[jobisoft]

I am one step further. I changed the internal implementation so that I can actually see the chunks and this is what I get from one of my Synology Testservers:

[onStartRequest] 
[onDataAvailable] 15872
[onDataAvailable] 15872
[onDataAvailable] 15872
[onDataAvailable] 15872
[onDataAvailable] 15872
[onDataAvailable] 15872
[onDataAvailable] 15872
[onDataAvailable] 15872
[onStopRequest] ‭0x804B0014‬

So I get 8 chunks of 15kb and than it stops with error 0x804B0014‬, which is NS_ERROR_NET_RESET, as stated here: https://developer.mozilla.org/en-US/docs/Mozilla/Errors

When I look at the actual chunks, the last one is not the expected end of an XML file/stream. So this confirms the situation: Something is resetting the connection, before all chunks have arrived.

But why?

[jobisoft]

And this is how it looks with HTTP:

[onStartRequest] 
[onDataAvailable] 974
[onDataAvailable] 2824
[onDataAvailable] 1412
[onDataAvailable] 2824
[onDataAvailable] 1412
[onDataAvailable] 15532
[onDataAvailable] 7060
[onDataAvailable] 8472
[onDataAvailable] 5648
[onDataAvailable] 12708
[onDataAvailable] 1412
[onDataAvailable] 15532
[onDataAvailable] 1412
[onDataAvailable] 5648
[onDataAvailable] 1412
[onDataAvailable] 15532
[onDataAvailable] 1412
[onDataAvailable] 2824
[onDataAvailable] 1412
[onDataAvailable] 2824
[onDataAvailable] 1412
[onDataAvailable] 5648
[onDataAvailable] 2824
[onDataAvailable] 1412
[onDataAvailable] 15532
[onDataAvailable] 1412
[onDataAvailable] 15532
[onDataAvailable] 1412
[onDataAvailable] 4236
[onDataAvailable] 2824
[onDataAvailable] 1412
[onDataAvailable] 2824
[onDataAvailable] 1412
[onDataAvailable] 15532
[onDataAvailable] 9884
[onDataAvailable] 1412
[onDataAvailable] 2824
[onDataAvailable] 4236
[onDataAvailable] 1412
[onDataAvailable] 2369
[onStopRequest] 0

Lots of chunks. All fine. And it is obviously not the last chunk which is missing in the HTTPS transmission.

I have no idea, what is causing this.

[jobisoft]

Is it possible to look at log files of the Synology? Are they accessible?

[danielbierstedt]

Had a quick look this morning, no luck. Will try to find them.

[jobisoft]

To make it even more confusing: With the new alternative implementation which allows me to see chunks, I can see that other servers also use the chunked method and it does not fail there. For example iCloud.

[jobisoft]

I am really running out of ideas. I use a very low level Mozilla method for the network communication (nsIHttpChannel), but today I played around with the fetch() API, which is a high level implementation and I also get an AbortError.

For further investigations, this is the request I send on the address book URL:

PROPFIND : <d:propfind xmlns:d="DAV:"><d:prop><d:getetag /></d:prop></d:propfind>

[danielbierstedt]

So you think it's the Synology implementation? Let's see if their support answers to my request. I will continue to look for the logs, but I would not wait for it...

[jobisoft]

I don‘t know. I have confirmation from a user who tried to use curl to make the same request and there it worked. But this is all very confusing.

[jobisoft]

It would be a huge help, if someone from Synology could look over this.

[danielbierstedt]

Yes, this is confusing, indeed. I see that it somehow works to a certain point when I migrate slowly. Max 5 contacts per sync, no pics etc. Then I see that, once failed, I cannot make it work again. I have to delete everything on the server and start over. Weird...

[jobisoft]

Is someone of you able to install stunnel on your synology?

If stunnel is used, we will not use the default ssl code from the carddav server (he thinks the connection is http) but we will be use the ssl implementation from stunnel. I remember the setup of stunnel was rather simple.

[jobisoft]

You of course than have to use a different port for the stunnel connection.

[jobisoft]

Based on this I created a step by step guide to install stunnel for this issue on a synology:

Perform all operations below as root.

Install stunnel

#sudo su
#ipkg install stunnel

Installing stunnel (4.26-2) to root… Downloading http://ipkg.nslu2-linux.org/feeds/optware/syno-e500/cross/unstable/stunnel_4.26-2_powerpc.ipk Installing zlib (1.2.5-1) to root… Downloading http://ipkg.nslu2-linux.org/feeds/optware/syno-e500/cross/unstable/zlib_1.2.5-1_powerpc.ipk Configuring stunnel Creating /opt/etc/stunnel/stunnel.pem (server certificate) … Generating a 1024 bit RSA private key ………….++++++ ……………………………………………………………++++++ writing new private key to ‘/opt/etc/stunnel/stunnel.pem’ —– You are about to be asked to enter information that will be incorporated into your certificate request. What you are about to enter is what is called a Distinguished Name or a DN. There are quite a few fields but you can leave some blank For some fields there will be a default value, If you enter ‘.’, the field will be left blank. —– Country Name (2 letter code) [PL]:KK State or Province Name (full name) [Some-State]:Your City Locality Name (eg, city) []:Your Location Organization Name (eg, company) [Stunnel Developers Ltd]:deadcode.net Organizational Unit Name (eg, section) []:stunnel Common Name (FQDN of your server) [localhost]:stunnel subject= /C=KK/ST=Your City/L=You Location/O=deadcode.net/OU=stunnel/CN=stunnel notBefore=May 19 13:57:47 2015 GMT notAfter=May 18 13:57:47 2016 GMT SHA1 Fingerprint=04:E0:55:BF:53:05:43:02:8C:07:A0:56:95:58:71:01:9C:BE:D4:18 postinst script returned status 1 ERROR: stunnel.postinst returned 1 Configuring zlib Successfully terminated.

Configuring stunnel

By default, the SSL cert generated is only valid for 1 year. That means it will fail one year later. To avoid a yearly fix for the certificate issue, we modify the config and generate a new cert again, which is valid for 10 years:

# vi /opt/etc/stunnel/stunnel-cert.cnf

add following line to the top of the file:

default_days = 3650

Then generate a new cert again.

# cd /opt/etc/stunnel
# mv stunnel.pem stunnel.pem.bak
# /opt/bin/openssl req -new -x509 -newkey rsa:2048 -keyout key.pem -out stunnel.pem -config /opt/etc/stunnel/stunnel-cert.cnf

Generating a 2048 bit RSA private key …………………………………………………..+++ ……………………………………………………………+++ writing new private key to ‘key.pem’ Enter PEM pass phrase: Verifying – Enter PEM pass phrase: —– You are about to be asked to enter information that will be incorporated into your certificate request. What you are about to enter is what is called a Distinguished Name or a DN. There are quite a few fields but you can leave some blank For some fields there will be a default value, If you enter ‘.’, the field will be left blank. —– Country Name (2 letter code) [PL]:HK State or Province Name (full name) [Some-State]:Hong-Kong Locality Name (eg, city) []:Hong-Kong Organization Name (eg, company) [Stunnel Developers Ltd]:deadcode.net Organizational Unit Name (eg, section) []:stunnel Common Name (FQDN of your server) [localhost]:stunnel

Remove passpharse from key file

# cd /opt/etc/stunnel
# /opt/bin/openssl rsa -in key.pem -out newkey.pem
# cat newkey.pem >> stunnel.pem
# chmod 400 stunnel.pem

Edit stunnel.conf

# vi /opt/etc/stunnel/stunnel.conf

Comment all [pop3s], [imaps], [ssmtp] sections. Also comment chroot, setuid, setgid, socket Add following lines to the end of the file

[carddav] accept = 8009 connect = 8008

With 8008 being your http port your carddav server is running at and 8009 the new port we want to use for https carddav via stunnel.

Activate stunnel

I do not know if stunnel is allreay running after the install. If not, it must be started with

/etc/init.d/stunnel.sh start

or restarted otherwise

/etc/init.d/stunnel.sh restart

Does that work?

[danielbierstedt]

Good morning,

this is just a feedback from my side. Why I opened this thicket was because I wanted to switch from Nextcloud (running on my Syno NAS) to the Syno inbuilt systems and get rid of Nextcloud. I then discovered the issues with contacts, had a short break and continued with the calendars. Turns out then, that the calendar solution does not work that good, too. The Outlook plugin re-creates schedules with every sync, really annoying.

So, I decided to stay with Nextcloud, where everything worked fine from the start. Don't think I'm not thankfull, but whith this second, real basic issue, I don't think its a good idea to move on.

I haven't heard anything from the Synology support, btw. Last action was to send them logs, silence since then.

PS: stunnel would not be ok with me. Its another system to take care of, hard work to maintain it on Synology and I want to stay as basic as possible.

[jobisoft]

Totally fine! You can continue to use TbSync with Nextcloud :-)

[jobisoft]

I take this off the bug list for now and mark it as wontfix. Feedback on the stunnel solution is still appreciated.

[HaRoHum]

I got stunnel up and running, but I spotted the next problem:

On my DS216+ I installed IPKG as described in https://chattim.wordpress.com/2016/05/08/install-ipkg-on-synology-via-gui/ and I installed stunnel as described above. The only valid lines in my stunnel.conf are root@Diskstation:/opt/etc/stunnel# egrep -v '^$|^;' stunnel.conf cert = /opt/etc/stunnel/stunnel.pem pid = /stunnel.pid [carddav] accept = 8009 connect = 8008 To (re-)start stunnel, I had to call /opt/etc/init.d/S68stunnel After this netstat -atu|grep 8009 showed, that stunnel was listening on Port 8009, so on my Diskstation everything was fine.

During the following steps I used wireshark to monitor the network traffic. Note: to decrypt the SSL trafic, I had to import synologies private key into wireshark as described her: https://support.citrix.com/article/CTX116557 On my DS216+, that key is found at /usr/local/etc/certificate/CardDAVServer/carddav/privkey.pem

First, I captured, what is going on, if I use the standard https communication as offered by Synology (port 8443). Wireshark shows the usual TLS handshake as described here: https://www.ibm.com/support/knowledgecenter/en/SSFKSJ7.1.0/com.ibm.mq.doc/sy10660.htm After that, TbSync sends its first request PROPFIND /addressbooks/users/xxxxxx/addressbook and Synology responds 301 Moved Permanently. At the same packet, Synology sends an Alert (Close Notify) to shut down the SSL session. TbSync responds properly by sending FIN, ACK. It then starts a new TLS handshake as described above and sends the next request ( PROPFIND /addressbooks/users/xxxxxx/addressbook/ (Note the trailing "/")). Note: many packets later, I captured that large XML response, this is all about here, but until now, I was not able to trace down what's wrong there.

Now, what is going on, if I use stunnel for https communication (port 8009) ? Again, wireshark shows the usual TLS handshake, followed by the first request PROPFIND /addressbooks/users/xxxxxx/addressbook. Again, Synology responds 301 Moved Permanently, but this time the Alert (Close Notify) is not sent in common, but as a separate packet. Again my PC responds properly by sending FIN, ACK, but this time a new TLS handshake is NOT initiated. Instead, the PC sends the next request ( PROPFIND /addressbooks/users/xxxxxx/addressbook/). As Synology had closed the SSL session before, it does not accept that and sends a RST. That's the end of the game. After several retrys, my PC gives up and reports an error. I have attached two screenshots here.

[jobisoft]

That is sad, so stunnel itself is now causing this? I am out of ideas...

the only thing you could try now is to skip the redirecting and give TbSync the redirected url directly.

[jobisoft]

There were some changes to the redirect code in TbSync. Are you on TbSync 1.7? Released yesterday?

[HaRoHum]

No, I did this on Feb 20th. I'll check again using 1.7

[HaRoHum]

sorry, 1.7 shows same behaviour as before

[jobisoft]

:-( sorry to hear that. I thank there is nothing more we can do...

[HaRoHum]

Yep, I agree well, staying on http is not that great problem for me: As long as I am working in my local network, I think this does not cause a security risk. If I am not at home, (e.g. on a public WLAN), I put up a VPN tunnel, so the traffic is already encrypted, no need for extra https.

[jobisoft]

I want to try one last thing, can I have access to your stunnel port with a test account on your Synology?

[jobisoft]

For some of you it might be enough to open the advanced Thunderbird config and set

extensions.dav4tbsync.maxitems = 1

This will reduce the data send per request and thus mitigate this bug. However, it does not work if your address book contains a lot of contacts, because during the very first request, I get all UIDs of your contacts and - currently - I do not know how to ask your server to split up that list. So if you have lots of contacts (> 1000 ??) that initial list will blow up the connection already.

[jobisoft]

@HaRoHum I made another change to the redirect code. Can you update your addons and try again? No need for new traces, just a quick run to see if it survives the redirect.

[HaRoHum]

Sorry, debug.log reports Error, see attachments

debug.log Request#1.txt Request#2.txt Response#1.txt Response#2.txt

[HaRoHum]

Sorry, I picked the wrong profile, pls. wait a minute

[jobisoft]

@HaRoHum Is it now not working at all for you?

[HaRoHum]

Hi John, yes, exactly.

One of my profiles uses TbSync (Beta) 1.7.1 Provider for CalDAV & CardDAV (Beta) 0.15.4 On this profile, search for folders and sync is operational (except for the problem with large xml files)

I copied this profile completely, and upgraded the add-ons to TbSync (Beta) 1.7.2 Provider for CalDAV & CardDAV (Beta) 0.15.6

Now, I get a HTTP 403 as described above

[HaRoHum]

sorry for those big letters. I should have used Preview before sending

[plcstop]

I use a simple workaround for contact pictures: reduce the picture size to 400px for the long size and it works (and 400px is more than enough for a contact thumbnail).

[patrick-hessinger]

Just experienced the same issue and came up with the following workaround: The Synology DSM offers a Reverse Proxy-Functionality (Control Panel > Application Portal > Reverse Proxy). I have created a Reverse Proxy Rule, that has localhost and the HTTP port (default 8008) of the CardDAV-Server as destination.

The reverse proxy will wrap the result in HTTPS, so communication to the client is secure. Also, with this config it does not drop the connection. Even more interesting: It also works if you have the HTTPS port as destination. So whatever Synology's cardDAV Server does wrong, the Reverse Proxy fixes it.

Without full investigation, I would say this is an Synology and not an TbSync issue then. Hope this workaround helps others getting there from the help button in the message.

[LapinFou]

Great Tips !!! Thanks you so much. 💯

[jobisoft]

I will soon create a dedicated Synology page with this information linked from TbSync. Thanks for this great news!

[jobisoft]

I created a new issue with the information https://github.com/jobisoft/DAV-4-TbSync/issues/104

Which I will later link from within TbSync.

@patrick-hessinger For a complete understanding, could you explain, what needs to be put into the two fields, that you blacked out? A theoretical example? What needs to be put into TbSync, than? You can add it here or into the new issue.

Thanks again for finding this!

[patrick-hessinger]

Glad I could help. I added instructions on the new issue: https://github.com/jobisoft/DAV-4-TbSync/issues/104#issuecomment-529086768

[jobisoft]

Thank you so much!