search

acmesh-official / acme.sh

A pure Unix shell script implementing ACME client protocol
https://acme.sh
GNU General Public License v3.0
39.23k stars 4.96k forks source link

Report bugs for Synology DSM deploy hook #2727

Open tresni opened 4 years ago

tresni commented 4 years ago

Please report bugs for the Synology DSM hook here. Remember to include debug logs

acme.sh  --deploy --deploy-hook synology_dsm .....   --debug 2
lippertmarkus commented 4 years ago

@tresni thank's for your work! This doesn't support automatically renewing the cert, does it?

tresni commented 4 years ago

@lippertmarkus If you mean will the Synology automatically renew the certs, no. However, renewed certificates will be updated on the synology.

If you want to do renewals on your synology, I do this using a cronjob. Control Panel -> Task Scheduler -> Create -> Scheduled Task -> User-defined Script. Under General, set User to root. Under Schedule, set a schedule you feel is appropriate (I do monthly.) Under Task Settings, put the following in the Run Command box (assuming you installed to /usr/local/share/acme.sh):

/usr/local/share/acme.sh/acme.sh --cron --home /usr/local/share/acme.sh/
lippertmarkus commented 4 years ago

@tresni Thanks for the quick response. I do basically the same, see https://github.com/lippertmarkus/synology-le-dns-auto-renew I just think searching for all the directories of all apps where the certificate is used like the script in the Wiki and my script does is a bit ugly.

When I saw that you use the web API in your deployment hook, I wondered if we could use the this API also for the renewing process?

tresni commented 4 years ago

This does both issuance and renewal. No weird directory handling, it just uses the API for everything. Maybe an example is better then trying to explain :)

ash-4.3# acme.sh --issue -d example.us --dns dns_cf
...
[Fri Mar  6 22:21:47 MST 2020] Your cert is in  /usr/local/share/acme.sh/example.us/example.us.cer
[Fri Mar  6 22:21:47 MST 2020] Your cert key is in  /usr/local/share/acme.sh/example.us/example.us.key
[Fri Mar  6 22:21:47 MST 2020] The intermediate CA cert is in  /usr/local/share/acme.sh/example.us/ca.cer
[Fri Mar  6 22:21:47 MST 2020] And the full chain certs is there:  /usr/local/share/acme.sh/example.us/fullchain.cer
ash-4.3# export SYNO_Username=... SYNO_Password=...
ash-4.3# SYNO_Create=1 SYNO_Certificate=example.us acme.sh -d example.us --deploy --deploy-hook synology_dsm
[Fri Mar  6 22:22:40 MST 2020] Logging into localhost:5000
[Fri Mar  6 22:22:47 MST 2020] Getting certificates in Synology DSM
[Fri Mar  6 22:22:47 MST 2020] Generate form POST request
[Fri Mar  6 22:22:47 MST 2020] Upload certificate to the Synology DSM
[Fri Mar  6 22:22:53 MST 2020] http services were NOT restarted
[Fri Mar  6 22:22:53 MST 2020] Success
ash-4.3# acme.sh -d example.us --renew --force
[Fri Mar  6 22:24:03 MST 2020] Renew: 'example.us'
[Fri Mar  6 22:24:04 MST 2020] Single domain='example.us'
[Fri Mar  6 22:24:04 MST 2020] Getting domain auth token for each domain
[Fri Mar  6 22:24:06 MST 2020] Getting webroot for domain='example.us'
[Fri Mar  6 22:24:06 MST 2020] example.us is already verified, skip dns-01.
[Fri Mar  6 22:24:06 MST 2020] Verify finished, start to sign.
[Fri Mar  6 22:24:06 MST 2020] Lets finalize the order, Le_OrderFinalize: https://acme-v02.api.letsencrypt.org/acme/finalize/.../...
[Fri Mar  6 22:24:07 MST 2020] Download cert, Le_LinkCert: https://acme-v02.api.letsencrypt.org/acme/cert/...
[Fri Mar  6 22:24:08 MST 2020] Cert success.
...
[Fri Mar  6 22:24:08 MST 2020] Your cert is in  /usr/local/share/acme.sh/example.us/example.us.cer
[Fri Mar  6 22:24:08 MST 2020] Your cert key is in  /usr/local/share/acme.sh/example.us/example.us.key
[Fri Mar  6 22:24:08 MST 2020] The intermediate CA cert is in  /usr/local/share/acme.sh/example.us/ca.cer
[Fri Mar  6 22:24:08 MST 2020] And the full chain certs is there:  /usr/local/share/acme.sh/example.us/fullchain.cer
[Fri Mar  6 22:24:09 MST 2020] Logging into localhost:5000
[Fri Mar  6 22:24:09 MST 2020] Getting certificates in Synology DSM
[Fri Mar  6 22:24:10 MST 2020] Generate form POST request
[Fri Mar  6 22:24:10 MST 2020] Upload certificate to the Synology DSM
[Fri Mar  6 22:24:10 MST 2020] http services were NOT restarted
[Fri Mar  6 22:24:10 MST 2020] Success
ash-4.3#
lippertmarkus commented 4 years ago

@tresni great, works like a charm! I added OTP/2-factor auth support in #2782 :)

lippertmarkus commented 4 years ago

@tresni sorry for hijacking this issue for discussing. Please tell me if you want to discuss elsewhere :)

While implementing #2782 I found that there is a binary which allows executing API requests without requiring authentication:

sudo /usr/syno/bin/synowebapi --exec api=SYNO.Core.Certificate.CRT method=list version=1
[Line 259] Exec WebAPI:  api=SYNO.Core.Certificate.CRT, version=1, method=list, param={}, runner=
{
   "data" : {
      "certificates" : [
         {
...

Maybe this could be used to implement a second way for importing the certificates? This wouldn't require any access information, so no scheme, host, port, username, password or device id environment variables. On the other hand this can only be used with a locally installed version of acme.sh and requires root. What do you think? Did you already try that?

I just briefly looked into that but couldn't figure out how to prepare a request for sending the binary certificate data with this method, but I think it should work somehow.

tresni commented 4 years ago

This module is designed where it can be run from any system, not just the Synology device itself, in line with what I believe acme.sh is attempting to accomplish. So using the WebApi allows for that, using the binary would not. Definitely a good find though!

tatablack commented 4 years ago

Nice work with the hook, much cleaner than looping over folders and certs. 👍 🙂

I was trying it today, but I ran into two separate issues:

Password The password is not escaped when hitting login.cgi in this line, and mine (aA#99B$dIix!) contained characters which broke the authentication in multiple ways (# has meaning in URLs, $ has meaning in shell scripts). I also had an exclamation mark, which however seemed fine once I exported the env variable using single instead of double quotes.

Changing the password to an alphanumeric one fixed the issue, as the script proceeded further.

Certificate upload Here's the (edited) output of ./acme.sh -d "*.subdomain.domain.net" --deploy --deploy-hook synology_dsm --debug 3 --home $PWD.

In short, uploading fails (after quite some time) and the response has an error code of 119. I'd welcome any suggestions here... 🤔

(EDIT: trying to upload the same files through the DSM Web UI works perfectly)

DSM: 6.2.2-24922 Update 4 Wget: GNU Wget 1.19.2

[Tue Apr 28 17:37:41 IST 2020] Upload certificate to the Synology DSM
[Tue Apr 28 17:37:41 IST 2020] ./acme.sh:_post:1733                     POST
[Tue Apr 28 17:37:41 IST 2020] ./acme.sh:_post:1734                     _post_url='http://localhost:5000/webapi/entry.cgi?api=SYNO.Core.Certificate&method=import&version=1&SynoToken=MYSYNOTOKEN'
[Tue Apr 28 17:37:41 IST 2020] ./acme.sh:_post:1735                     body='----------------------------20200428163741
Content-Disposition: form-data; name="key"; filename="*.subdomain.domain.net.key"
Content-Type: application/octet-stream

-----BEGIN RSA PRIVATE KEY-----
<PRIVATE KEY REMOVED>
-----END RSA PRIVATE KEY-----

----------------------------20200428163741
Content-Disposition: form-data; name="cert"; filename="*.subdomain.domain.net.cer"
Content-Type: application/octet-stream

-----BEGIN CERTIFICATE-----
<CERTIFICATE REMOVED>
-----END CERTIFICATE-----

----------------------------20200428163741
Content-Disposition: form-data; name="inter_cert"; filename="ca.cer"
Content-Type: application/octet-stream

-----BEGIN CERTIFICATE-----
<CERTIFICATE REMOVED>
-----END CERTIFICATE-----

----------------------------20200428163741
Content-Disposition: form-data; name="id"

----------------------------20200428163741
Content-Disposition: form-data; name="desc"

Letsencrypt wildcard certificate for my internal network
----------------------------20200428163741
Content-Disposition: form-data; name="as_default"

false
----------------------------20200428163741--
'
[Tue Apr 28 17:37:41 IST 2020] ./acme.sh:_post:1736                     _postContentType='multipart/form-data; boundary=--------------------------20200428163741'
[Tue Apr 28 17:37:42 IST 2020] ./acme.sh:_exists:494                    curl exists=0
[Tue Apr 28 17:37:42 IST 2020] ./acme.sh:_exists:494                    mktemp exists=0
[Tue Apr 28 17:37:42 IST 2020] ./acme.sh:_exists:494                    wget exists=0
[Tue Apr 28 17:37:42 IST 2020] ./acme.sh:_post:1794                     _WGET='wget -q -d  --content-on-error '
[Tue Apr 28 17:40:08 IST 2020] Please refer to https://www.gnu.org/software/wget/manual/html_node/Exit-Status.html for error code: 4
[Tue Apr 28 17:40:08 IST 2020] ./acme.sh:_sed_i:867                     options='s/^ *//g'
[Tue Apr 28 17:40:08 IST 2020] ./acme.sh:_sed_i:869                     Using sed  -i
[Tue Apr 28 17:40:09 IST 2020] ./acme.sh:_post:1843                     _ret='4'
[Tue Apr 28 17:40:09 IST 2020] deploy/synology_dsm.sh:synology_dsm_deploy:142 response='{"error":{"code":119},"success":false}'
[Tue Apr 28 17:40:09 IST 2020] Unable to update certificate, error code {"error":{"code":119},"success":false}
[Tue Apr 28 17:40:09 IST 2020] Error deploy for domain:*.subdomain.domain.net
[Tue Apr 28 17:40:09 IST 2020] Deploy error.
tatablack commented 4 years ago

Actually, according to the manual, Wget does not currently support multipart/form-data for transmitting POST data... 😖

I tried this again using cURL and it works.

So I guess the options are either doing the upload in a way that works with Wget, or bail out with a message if that's not possible and Wget is being used.

tresni commented 4 years ago

Exit code 4 is a network error which I don't think is related to the form data itself. Also, the documentation for wget is just saying that it cannot create multipart uploads itself. We construct the payload ourself and send it as data, I don't think this has any impact on wget, but I'll double check.

JoJota commented 4 years ago

I'm using default cURL for transferring the certificates, but receiving error code 119 too.

As the default synology certificate is not verified, I first got cURL error 60 with the combination of "SSL certificate problem: unable to get local issuer certificate". After resolving this problem with setting export HTTPS_INSECURE=1 I received the synology error 119, which I am stuck with now. Further investigation seemed that there is not very much known what causes this error in detail, at least I couldn't find anything helpful.

Apart from that after not getting any further, I tried importing the certificates with a different reload script which was linked here, but this broke my DSM certificate system completely (no certificates were shown and even after regenerating the nginx certificate folder all the services which use the certificates are gone and I am not able to restore them). So I will now restore the DSM system to factory settings to see if that somehow helps getting rid of the 119er.

Device: DS216j DSM Version: DSM 6.2.2-24922 Update 4

Error 119 debug log using cURL (cleared of private data):

[Mon  4 May 10:37:42 CEST 2020] _postContentType='multipart/form-data; boundary=--------------------------20200504083742'
[Mon  4 May 10:37:42 CEST 2020] _CURL='curl -L --silent --dump-header /home/pi/.acme.sh/http.header  --trace-ascii /tmp/tmp.DOAhb8GRzk  -g  --insecure  '
[Mon  4 May 10:37:42 CEST 2020] Please refer to https://curl.haxx.se/libcurl/c/libcurl-errors.html for error code: 92
[Mon  4 May 10:37:42 CEST 2020] Here is the curl dump log:
[Mon  4 May 10:37:42 CEST 2020] == Info: Expire in 0 ms for 6 (transfer 0x3fa880)
== Info: Expire in 1 ms for 1 (transfer 0x3fa880)
== Info: Expire in 0 ms for 1 (transfer 0x3fa880)
(...)
== Info: Expire in 0 ms for 1 (transfer 0x3fa880)
== Info:   Trying 192.168.222.21...
== Info: TCP_NODELAY set
== Info: Expire in 200 ms for 4 (transfer 0x3fa880)
== Info: Connected to example.com (192.168.222.21) port 443 (#0)
== Info: ALPN, offering h2
== Info: ALPN, offering http/1.1
== Info: successfully set certificate verify locations:
== Info:   CAfile: none
  CApath: /etc/ssl/certs
=> Send SSL data, 5 bytes (0x5)
0000: .....
== Info: TLSv1.3 (OUT), TLS handshake, Client hello (1):
=> Send SSL data, 512 bytes (0x200)
0000: ......x..*79..%.).$.J.G..}...<..&..r.# ......b..j.G>....};...-
0040: O.,.....>.......,.0.........+./...$.(.k.#.'.g.....9.....3.....=.
<= Recv SSL data, 5 bytes (0x5)
0000: ....b
== Info: TLSv1.3 (IN), TLS handshake, Server hello (2):
<= Recv SSL data, 98 bytes (0x62)
0000: ...^../*.-.....Y.J.......C3]n~........ ..1.!.$..Nz.M.}d...lH...<
0040: .pY}..../.......................h2
<= Recv SSL data, 5 bytes (0x5)
0000: ....(
== Info: TLSv1.2 (IN), TLS handshake, Certificate (11):
<= Recv SSL data, 808 bytes (0x328)
0000: ...$..!...0...0.......|..{My.0...*.H........0Q1.0...U....TW1.0..
0040: .U....Taipei1.0...U....Synology Inc.1.0...U....Synology Inc. CA0
0080: ...200503181130Z..400119181130Z0M1.0...U....TW1.0...U....Taipei1
00c0: .0...U....Synology Inc.1.0...U....synology.com0.."0...*.H.......
0100: ......0..............K.u.v..^Ht.#..:.4......4.w..;..B.d.......
0140: (.'....W.>..(.........qlD.3.'B....EO3..Y.CvT.....X)i%?GE.DE..c
0180: (.]#........RbSn.hL~....1.A.....;..#....k..F...1....]w.G0.....^.
01c0: .......4X.....*..\.^..b_....5.K..u<s...D.-.....FU....._...V..(..
0200: W=4y.......m.-#.....0...*.H..............^.........e....E..O.MiB
0240: /n.....eC!...p..v.C7G...*n...."..N.|....E.).....s..C..t6.bx;..!.
0280: .(.iy!.*...~_p},{.<.R.........Lm.=mu..Q......U.%+H.)u6...T......
02c0: ..e...........9!..6..2.c.V..o.x..-iN2:.e...`....|.*+k.b..4.nc+..
0300: =........t.....h.....X..`xD.....q....*D6
<= Recv SSL data, 5 bytes (0x5)
0000: ....M
== Info: TLSv1.2 (IN), TLS handshake, Server key exchange (12):
<= Recv SSL data, 333 bytes (0x14d)
0000: ...I...A...1...E.sNL..,8.5.....\l.ui.3. q.F..9E.3y1C.....}..3...
0040: ....... .......N$.?./.y5.......> ....e..b..}Cc....:..3.....@....
0080: ..%D.]...{.....V.......H!...#M". ...y.4.<..|.2.i^...;'J..UW..M
00c0: y......4..1..)T.dS.&...=....I..c@W....sM....N.|.$U$.B'.M........
0100: .K..@.......z.|6G...d~..Un......=.....9..Fk........W...s.r{...W.
0140: P..O.u2....MD
<= Recv SSL data, 5 bytes (0x5)
0000: .....
== Info: TLSv1.2 (IN), TLS handshake, Server finished (14):
<= Recv SSL data, 4 bytes (0x4)
0000: ....
=> Send SSL data, 5 bytes (0x5)
0000: ....F
== Info: TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
=> Send SSL data, 70 bytes (0x46)
0000: ...BA..*~b.#.}>i.....?..G..Y.*....l.3..'..T...l6.h.&../..@.I...@
0040: {}.=.E
=> Send SSL data, 5 bytes (0x5)
0000: .....
== Info: TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
=> Send SSL data, 1 bytes (0x1)
0000: .
=> Send SSL data, 5 bytes (0x5)
0000: ....(
== Info: TLSv1.2 (OUT), TLS handshake, Finished (20):
=> Send SSL data, 16 bytes (0x10)
0000: ........=.b....w
<= Recv SSL data, 5 bytes (0x5)
0000: .....
<= Recv SSL data, 5 bytes (0x5)
0000: ....(
== Info: TLSv1.2 (IN), TLS handshake, Finished (20):
<= Recv SSL data, 16 bytes (0x10)
0000: .....%..S.......
== Info: SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
== Info: ALPN, server accepted to use h2
== Info: Server certificate:
== Info:  subject: C=TW; L=Taipei; O=Synology Inc.; CN=synology.com
== Info:  start date: May  3 18:11:30 2020 GMT
== Info:  expire date: Jan 19 18:11:30 2040 GMT
== Info:  issuer: C=TW; L=Taipei; O=Synology Inc.; CN=Synology Inc. CA
== Info:  SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
== Info: Using HTTP2, server supports multi-use
== Info: Connection state changed (HTTP/2 confirmed)
== Info: Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
=> Send SSL data, 5 bytes (0x5)
0000: ....0
=> Send SSL data, 5 bytes (0x5)
0000: ....3
=> Send SSL data, 5 bytes (0x5)
0000: ....%
== Info: Using Stream ID: 1 (easy handle 0x3fa880)
=> Send SSL data, 5 bytes (0x5)
0000: .....
=> Send header, 375 bytes (0x177)
0000: POST /webapi/entry.cgi?api=SYNO.Core.Certificate&method=import&v
0040: ersion=1&SynoToken=ObIUVFgjj3w66 HTTP/2
0069: Host: example.com:port
008b: User-Agent: acme.sh/2.8.6 (https://github.com/acmesh-official/ac
00cb: me.sh)
00d3: Accept: */*
00e0: Content-Type: multipart/form-data; boundary=--------------------
0120: ------20200504083742
0136: Cookie: ;
0142: X-SYNO-TOKEN: ObIUVFgjj3w66
015f: Content-Length: 6185
0175:
<= Recv SSL data, 5 bytes (0x5)
0000: ....@
== Info: Connection state changed (MAX_CONCURRENT_STREAMS == 128)!
=> Send SSL data, 5 bytes (0x5)
0000: ....!
=> Send SSL data, 5 bytes (0x5)
0000: ....J
=> Send data, 6185 bytes (0x1829)
0000: ----------------------------20200504083742
002c: Content-Disposition: form-data; name="key"; filename="my.key"
007e: Content-Type: application/octet-stream
00a6:
00a8: -----BEGIN RSA PRIVATE KEY----------END RSA PRIVAT
0728: E KEY-----.
0735: ----------------------------20200504083742
0761: Content-Disposition: form-data; name="cert"; filename="my.cer"
07b4: Content-Type: application/octet-stream
07dc:
07de: -----BEGIN CERTIFICATE----------END C
0f9e: ERTIFICATE-----.
0fb0: ----------------------------20200504083742
0fdc: Content-Disposition: form-data; name="inter_cert"; filename="ca.
101c: cer"
1022: Content-Type: application/octet-stream
104a:
104c: .-----BEGIN CERTIFICATE----------END CERTIFICATE-----.
16be: ----------------------------20200504083742
16ea: Content-Disposition: form-data; name="id"
1715:
1717:
1719: ----------------------------20200504083742
1745: Content-Disposition: form-data; name="desc"
1772:
1774: acme.sh certificate
1793: ----------------------------20200504083742
17bf: Content-Disposition: form-data; name="as_default"
17f2:
17f4: false
17fb: ----------------------------20200504083742--
== Info: We are completely uploaded and fine
<= Recv SSL data, 5 bytes (0x5)
0000: ....!
<= Recv SSL data, 5 bytes (0x5)
0000: ....O
<= Recv header, 13 bytes (0xd)
0000: HTTP/2 200
<= Recv header, 15 bytes (0xf)
0000: server: nginx
<= Recv header, 37 bytes (0x25)
0000: date: Mon, 04 May 2020 08:37:42 GMT
<= Recv header, 43 bytes (0x2b)
0000: content-type: text/plain; charset="UTF-8"
<= Recv header, 23 bytes (0x17)
0000: vary: Accept-Encoding
<= Recv header, 25 bytes (0x19)
0000: x-request-error: unauth
<= Recv header, 33 bytes (0x21)
0000: x-content-type-options: nosniff
<= Recv header, 33 bytes (0x21)
0000: x-xss-protection: 1; mode=block
<= Recv header, 63 bytes (0x3f)
0000: cache-control: max-age=0, no-cache, no-store, must-revalidate
<= Recv header, 18 bytes (0x12)
0000: pragma: no-cache
<= Recv header, 12 bytes (0xc)
0000: expires: 0
<= Recv header, 73 bytes (0x49)
0000: strict-transport-security: max-age=15768000; includeSubdomains;
0040: preload
<= Recv header, 2 bytes (0x2)
0000:
<= Recv data, 38 bytes (0x26)
0000: {"error":{"code":119},"success":false}
<= Recv SSL data, 5 bytes (0x5)
0000: ....%
== Info: HTTP/2 stream 0 was not closed cleanly: INTERNAL_ERROR (err 2)
== Info: stopped the pause stream!
== Info: Connection #0 to host example.com left intact
[Mon  4 May 10:37:42 CEST 2020] _ret='92'
[Mon  4 May 10:37:42 CEST 2020] Unable to update certificate, error code {"error":{"code":119},"success":false}
[Mon  4 May 10:37:43 CEST 2020] Error deploy for domain:example.com
[Mon  4 May 10:37:43 CEST 2020] Deploy error.
tresni commented 4 years ago

I've fixed the URL encoding issue for passwords, I'll open a PR for that shortly. I have confirmed there appears to be an issue using wget. Comparing differences between wget and curl now.

tresni commented 4 years ago

@JoJota and @tatablack both of your issues should be fixed by #2935 . Thank you for the reports. @JoJota I'm not entirely sure but I believe this is actually the same issue that is causing wget to fail all the time. curl always worked for me, but looking at the code and digging through everything it looks like it should have been failing with the 119 error. (That error is not actually defined but it appears to be the id cookie is missing.) Anyway, should be resolved, please let me know if you continue to have any issues.

ghost commented 4 years ago

The Synology DSM deploy hook feature is broken - on master and dev branches.

The script does not detect or read exported SYNO_Create variable/data. Only when it is added to the synology_dsm.sh script are you able to progress up to the deployment function which after authentication and reading of cert data fails to add and apply the Let's Encrypt cert.

[Fri May 22 18:49:39 +03 2020] _postContentType='multipart/form-data; boundary=--------------------------20200522154939'
[Fri May 22 18:49:39 +03 2020] _CURL='curl -L --silent --dump-header /root/.acme.sh/http.header  --trace-ascii /tmp/tmp.liqxVE82'
[Fri May 22 18:49:39 +03 2020] _ret='0'
[Fri May 22 18:49:39 +03 2020] Unable to update certificate, error code {"error":{"code":5529},"success":false}
[Fri May 22 18:49:39 +03 2020] Error deploy for domain:xxx.xxxxxxx.xxx
[Fri May 22 18:49:39 +03 2020] Deploy error.

Note: xxx.xxxxxxx.xxx replaces actual cert domain

There's also the issue of the missing sudo command which is not mentioned in the wiki - https://github.com/acmesh-official/acme.sh/wiki/Synology-NAS-Guide

No issues to report with the issuance of Let's Encrypt certs, acme.sh itself or other exported variables ie. CERT_DOMAIN

tresni commented 4 years ago

@AtAM1 Please give some additional information. Are you attempting to run this on Synology NAS device or from a remote computer? If on a Synology NAS, what device and what version of DSM? I've just tested from my personal machine to a virtualized DS3615xs running DSM 6.1.7 with no issues (only the domain has been changed.)

With latest dev:

 bash-5.0$ ./acme.sh --test -d example.com --deploy --deploy-hook synology_dsm
[Fri May 22 10:34:34 MDT 2020] Using stage ACME_DIRECTORY: https://acme-staging-v02.api.letsencrypt.org/directory
[Fri May 22 10:34:34 MDT 2020] Logging into localhost:5000
[Fri May 22 10:34:35 MDT 2020] Getting certificates in Synology DSM
[Fri May 22 10:34:35 MDT 2020] Unable to find certificate: What In The World and $SYNO_Create is not set
[Fri May 22 10:34:35 MDT 2020] Error deploy for domain:example.com
[Fri May 22 10:34:35 MDT 2020] Deploy error.
bash-5.0$ SYNO_Create=1 ./acme.sh --test -d example.com --deploy --deploy-hook synology_dsm
[Fri May 22 10:34:45 MDT 2020] Using stage ACME_DIRECTORY: https://acme-staging-v02.api.letsencrypt.org/directory
[Fri May 22 10:34:45 MDT 2020] Logging into localhost:5000
[Fri May 22 10:34:45 MDT 2020] Getting certificates in Synology DSM
[Fri May 22 10:34:46 MDT 2020] Generate form POST request
[Fri May 22 10:34:46 MDT 2020] Upload certificate to the Synology DSM
[Fri May 22 10:34:46 MDT 2020] http services were NOT restarted
[Fri May 22 10:34:46 MDT 2020] Success

With master:

bash-5.0$ ./acme.sh --test -d example.com --deploy --deploy-hook synology_dsm
[Fri May 22 10:40:56 MDT 2020] Using stage ACME_DIRECTORY: https://acme-staging-v02.api.letsencrypt.org/directory
[Fri May 22 10:40:56 MDT 2020] Logging into localhost:5000
[Fri May 22 10:40:56 MDT 2020] Getting certificates in Synology DSM
/Users/bhartvig/Projects/acme.sh/deploy/synology_dsm.sh: line 113: SYNO_Create: parameter null or not set
[Fri May 22 10:40:56 MDT 2020] Deploy error.
bash-5.0$ SYNO_Create=1 ./acme.sh --test -d example.com --deploy --deploy-hook synology_dsm
[Fri May 22 10:41:04 MDT 2020] Using stage ACME_DIRECTORY: https://acme-staging-v02.api.letsencrypt.org/directory
[Fri May 22 10:41:04 MDT 2020] Logging into localhost:5000
[Fri May 22 10:41:04 MDT 2020] Getting certificates in Synology DSM
[Fri May 22 10:41:04 MDT 2020] Generate form POST request
[Fri May 22 10:41:04 MDT 2020] Upload certificate to the Synology DSM
[Fri May 22 10:41:04 MDT 2020] http services were NOT restarted
[Fri May 22 10:41:04 MDT 2020] Success

Additionally, the hook does not use sudo, so I'm not sure what you mean by "missing sudo command" at all.

tresni commented 4 years ago

I just tested master on the same virtual machine (DS3615sx running DSM 6.1.7) and had no issues:

tresni@testdsm:/$ sudo -i

We trust you have received the usual lecture from the local System
Administrator. It usually boils down to these three things:

    #1) Respect the privacy of others.
    #2) Think before you type.
    #3) With great power comes great responsibility.

Password:
root@testdsm:~# wget https://github.com/Neilpang/acme.sh/archive/master.tar.gz
--2020-05-22 09:49:18--  https://github.com/Neilpang/acme.sh/archive/master.tar.gz
Resolving github.com... 140.82.113.4
Connecting to github.com|140.82.113.4|:443... connected.
HTTP request sent, awaiting response... 301 Moved Permanently
Location: https://github.com/acmesh-official/acme.sh/archive/master.tar.gz [following]
--2020-05-22 09:49:18--  https://github.com/acmesh-official/acme.sh/archive/master.tar.gz
Reusing existing connection to github.com:443.
HTTP request sent, awaiting response... 302 Found
Location: https://codeload.github.com/acmesh-official/acme.sh/tar.gz/master [following]
--2020-05-22 09:49:18--  https://codeload.github.com/acmesh-official/acme.sh/tar.gz/master
Resolving codeload.github.com... 140.82.113.10
Connecting to codeload.github.com|140.82.113.10|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 193181 (189K) [application/x-gzip]
Saving to: 'master.tar.gz'

100%[====================================================================================================================>] 193,181      753KB/s   in 0.3s

2020-05-22 09:49:19 (753 KB/s) - 'master.tar.gz' saved [193181/193181]

root@testdsm:~# tar xvf master.tar.gz
[trimmed]
root@testdsm:~# cd acme.sh-master/
root@testdsm:~/acme.sh-master# ./acme.sh --install --nocron --home /usr/local/share/acme.sh --accountemail "brian.andrew@brianandjenny.com"
[Fri May 22 09:49:59 MST 2020] It is recommended to install socat first.
[Fri May 22 09:49:59 MST 2020] We use socat for standalone server if you use standalone mode.
[Fri May 22 09:49:59 MST 2020] If you don't use standalone mode, just ignore this warning.
[Fri May 22 09:49:59 MST 2020] Installing to /usr/local/share/acme.sh
[Fri May 22 09:49:59 MST 2020] Installed to /usr/local/share/acme.sh/acme.sh
[Fri May 22 09:49:59 MST 2020] Installing alias to '/root/.profile'
[Fri May 22 09:49:59 MST 2020] OK, Close and reopen your terminal to start using acme.sh
[Fri May 22 09:50:00 MST 2020] Good, bash is found, so change the shebang to use bash as preferred.
[Fri May 22 09:50:00 MST 2020] OK
root@testdsm:~/acme.sh-master# cd /usr/local/share/acme.sh
root@testdsm:/usr/local/share/acme.sh# export HE_Username=[USERNAME]
root@testdsm:/usr/local/share/acme.sh# export HE_Password=[PASSWORD]
root@testdsm:/usr/local/share/acme.sh# export SYNO_Username=[USERNAME]
root@testdsm:/usr/local/share/acme.sh# export SYNO_Password=[PASSWORD]
root@testdsm:/usr/local/share/acme.sh# . acme.sh.env
root@testdsm:/usr/local/share/acme.sh# acme.sh --test --issue -d example.com --dns dns_he
It seems that you are using sudo, please read this link first:
https://github.com/acmesh-official/acme.sh/wiki/sudo
root@testdsm:/usr/local/share/acme.sh# exit
logout
tresni@testdsm:/$ sudo su
ash-4.3# cd /usr/local/share/acme.sh
ash-4.3# . acme.sh.env
ash-4.3# export HE_Username=[USERNAME]
ash-4.3# export HE_Password=[PASSWORD]
ash-4.3# export SYNO_Username=[USERNAME]
ash-4.3# export SYNO_Password=[PASSWORD]
ash-4.3# export SYNO_Certificate="This is a test"
ash-4.3# acme.sh --test --issue -d example.com --dns dns_he
[Fri May 22 09:55:05 MST 2020] Using stage ACME_DIRECTORY: https://acme-staging-v02.api.letsencrypt.org/directory
[Fri May 22 09:55:06 MST 2020] Single domain='example.com'
[Fri May 22 09:55:06 MST 2020] Getting domain auth token for each domain
[Fri May 22 09:55:07 MST 2020] Getting webroot for domain='example.com'
[Fri May 22 09:55:07 MST 2020] Adding txt value: kdSTp4Gi5_88zn6WQFSUe9kxv1tlSQGpJzzkpkonbgk for domain:  _acme-challenge.example.com
[Fri May 22 09:55:07 MST 2020] Using DNS-01 Hurricane Electric hook
[Fri May 22 09:55:08 MST 2020] TXT record added successfully.
[Fri May 22 09:55:08 MST 2020] The txt record is added: Success.
[Fri May 22 09:55:08 MST 2020] Let's check each dns records now. Sleep 20 seconds first.
[Fri May 22 09:55:29 MST 2020] Checking example.com for _acme-challenge.example.com
[Fri May 22 09:55:29 MST 2020] Domain example.com '_acme-challenge.example.com' success.
[Fri May 22 09:55:29 MST 2020] All success, let's return
[Fri May 22 09:55:29 MST 2020] Verifying: example.com
[Fri May 22 09:55:32 MST 2020] Success
[Fri May 22 09:55:32 MST 2020] Removing DNS records.
[Fri May 22 09:55:32 MST 2020] Removing txt: kdSTp4Gi5_88zn6WQFSUe9kxv1tlSQGpJzzkpkonbgk for domain: _acme-challenge.example.com
[Fri May 22 09:55:32 MST 2020] Cleaning up after DNS-01 Hurricane Electric hook
[Fri May 22 09:55:33 MST 2020] Record removed successfully.
[Fri May 22 09:55:33 MST 2020] Removed: Success
[Fri May 22 09:55:33 MST 2020] Verify finished, start to sign.
[REMOVED LINKS TO CERT]
[Fri May 22 09:55:34 MST 2020] Cert success.
-----BEGIN CERTIFICATE-----
[TRIMMED]
-----END CERTIFICATE-----
[Fri May 22 09:55:34 MST 2020] Your cert is in  /usr/local/share/acme.sh/example.com/example.com.cer
[Fri May 22 09:55:34 MST 2020] Your cert key is in  /usr/local/share/acme.sh/example.com/example.com.key
[Fri May 22 09:55:34 MST 2020] The intermediate CA cert is in  /usr/local/share/acme.sh/example.com/ca.cer
[Fri May 22 09:55:34 MST 2020] And the full chain certs is there:  /usr/local/share/acme.sh/example.com/fullchain.cer
ash-4.3# acme.sh --test -d example.com --deploy --deploy-hook synology_dsm
[Fri May 22 09:56:16 MST 2020] Using stage ACME_DIRECTORY: https://acme-staging-v02.api.letsencrypt.org/directory
[Fri May 22 09:56:16 MST 2020] Logging into localhost:5000
[Fri May 22 09:56:16 MST 2020] Getting certificates in Synology DSM
/usr/local/share/acme.sh/deploy/synology_dsm.sh: line 113: SYNO_Create: parameter null or not set
[Fri May 22 09:56:16 MST 2020] Deploy error.
ash-4.3# SYNO_Create=1 acme.sh --test -d example.com --deploy --deploy-hook synology_dsm
[Fri May 22 09:56:30 MST 2020] Using stage ACME_DIRECTORY: https://acme-staging-v02.api.letsencrypt.org/directory
[Fri May 22 09:56:30 MST 2020] Logging into localhost:5000
[Fri May 22 09:56:30 MST 2020] Getting certificates in Synology DSM
[Fri May 22 09:56:31 MST 2020] Generate form POST request
[Fri May 22 09:56:31 MST 2020] Upload certificate to the Synology DSM
[Fri May 22 09:56:31 MST 2020] http services were NOT restarted
[Fri May 22 09:56:31 MST 2020] Success
ash-4.3#
ghost commented 4 years ago

@tresni Hi Brian.

Yes, I am connected to the synology box via ssh from my computer (mac pro).

On sudo - if I run the command without sudo:

root@xxxx:/usr/local/share/acme.sh# ./acme.sh --deploy -d "$CERT_DOMAIN" --deploy-hook synology_dsm --ecc --debug 2
It seems that you are using sudo, please read this link first:
https://github.com/acmesh-official/acme.sh/wiki/sudo

The same goes for script installation. Once you execute 'sudo -i' and after entering password you are logged in as root. I did not try to install the script under an administrator account as 'sudo -i' was used in the instructions.

As for SYNO_Create, I just tried executing the same command line you used but with sudo and after exporting SYNO_Unsername, SYNO_Password, CERT_DOMAIN, SYNO_Certificate and the result:

root@xxxx:/usr/local/share/acme.sh# SYNO_Create=1 sudo ./acme.sh --test -d xxx.xxxxxxx.xxx --deploy --deploy-hook synology_dsm --ecc
[Fri May 22 19:59:00 +03 2020] Using stage ACME_DIRECTORY: https://acme-staging-v02.api.letsencrypt.org/directory
[Fri May 22 19:59:00 +03 2020] Logging into xxx.xxxxxxx.xxx:14080
[Fri May 22 19:59:01 +03 2020] Getting certificates in Synology DSM
[Fri May 22 19:59:01 +03 2020] Unable to find certificate:  and $SYNO_Create is not set
[Fri May 22 19:59:01 +03 2020] Error deploy for domain:xxx.xxxxxxx.xxx
[Fri May 22 19:59:01 +03 2020] Deploy error.

acme.sh created and installed the cert and its files to /root/.acme.sh/xxx.xxxxxxx.xxx_ecc/

Also, another issue I had with the master branch was SYNO_Scheme, SYNO_Hostname, SYNO_Port were not being picked up by the deployment script so I manually saved them in /root/.acme.sh/xxx.xxxxxxx.xxx_ecc/xxx.xxxxxxx.xxx.conf ... haven't tested dev yet.

tresni commented 4 years ago

On sudo - if I run the command without sudo:

A shell started with sudo -i is detected as using sudo. Doing sudo su is not. YMMV.

As for SYNO_Create, I just tried executing the same command line you used but with sudo and after exporting SYNO_Unsername, SYNO_Password, CERT_DOMAIN, SYNO_Certificate and the result:

The command is incorrect. By doing SYNO_Create=1 sudo you are passing SYNO_Create in the environment to sudo, but it won't (normally) be passed on to the command you are actually running (./acme.sh).

asciicast

If you are going to use sudo, the appropriate command would be sudo SYNO_Create=1 ./acme.sh. Any environmental variables you want to pass need to come after sudo and before the command you want to execute.

Just an additional note that I've update the Synology NAS guide to give steps that work with no workarounds necessary. Please let me know if you have problems after following the steps exactly as outlined there.

ghost commented 4 years ago

@tresni Alright. I'm going to start fresh and install after executing sudo su instead of sudo -i and see where that takes me.

Appreciate your help!

Thanks

ghost commented 4 years ago

@tresni Hello again Brian.

I gave this another try and had followed the updated wiki. While acme.sh can now be executed without sudo and the deployment script is detecting/reading the data for SYNO_xxxxxx, it's still complaining and failing when deploying the cert to DSM.

ash-4.3# SYNO_Create=1 ./acme.sh --deploy -d xxx.xxxxxxx.xxx --deploy-hook synology_dsm --ecc
[Fri May 22 23:15:37 +03 2020] Logging into xxx.xxxxxxx.xxx:14080
[Fri May 22 23:15:38 +03 2020] Getting certificates in Synology DSM
[Fri May 22 23:15:38 +03 2020] Generate form POST request
[Fri May 22 23:15:38 +03 2020] Upload certificate to the Synology DSM
[Fri May 22 23:15:39 +03 2020] Unable to update certificate, error code {"error":{"code":5529},"success":false}
[Fri May 22 23:15:39 +03 2020] Error deploy for domain:xxx.xxxxxxx.xxx
[Fri May 22 23:15:39 +03 2020] Deploy error.

What else can be causing this?

Thank you!

ghost commented 4 years ago

Adding debug output:

[Sat May 23 00:13:16 +03 2020] ./acme.sh:_post:1735                     _postContentType='multipart/form-data; boundary=--------------------------20200522211316'
[Sat May 23 00:13:16 +03 2020] ./acme.sh:_exists:494                    curl exists=0
[Sat May 23 00:13:16 +03 2020] ./acme.sh:_exists:494                    mktemp exists=0
[Sat May 23 00:13:16 +03 2020] ./acme.sh:_exists:494                    wget exists=0
[Sat May 23 00:13:16 +03 2020] ./acme.sh:_post:1747                     _CURL='curl -L --silent --dump-header /root/.acme.sh/http.header  --trace-ascii /tmp/tmp.4U3Sf34XpF  -g '
[Sat May 23 00:13:16 +03 2020] ./acme.sh:_post:1842                     _ret='0'
[Sat May 23 00:13:16 +03 2020] deploy/synology_dsm.sh:synology_dsm_deploy:139 response='{"error":{"code":5529},"success":false}'
[Sat May 23 00:13:16 +03 2020] Unable to update certificate, error code {"error":{"code":5529},"success":false}
[Sat May 23 00:13:16 +03 2020] Error deploy for domain:xxx.xxxxxxx.xxx
[Sat May 23 00:13:16 +03 2020] Deploy error.

Tested with synology_dsm.sh from both master and dev branches

tresni commented 4 years ago

What Synology device are you using? What version of DSM are you using? I can't find error code 5529 defined. The surrounding defined errors that I could find are:

5521:_T("certificate","err_connection"),
5522:_T("certificate","err_server_not_match"),
5523:_T("certificate","err_too_many_reg"),
5524:_T("certificate","err_too_many_req"),
5525:_T("certificate","err_mail"),
5526:_T("s2s","err_invalid_param_value")

I've been able to successfully deploy an ECC certificate as follows (DS3615sx and DSM 6.1.7):

./acme.sh --test --issue -d example.com --dns dns_he --keylength ec-256
./acme.sh --test -d example.com --deploy --deploy-hook synology_dsm --ecc

With the information you've provided, I can't replicate what you are seeing.

ghost commented 4 years ago

What Synology device are you using? What version of DSM are you using? I can't find error code 5529 defined. The surrounding defined errors are that I could find are:

918+ with DSM 6.2.3 rel 25426 (latest release)

5521:_T("certificate","err_connection"),
5522:_T("certificate","err_server_not_match"),
5523:_T("certificate","err_too_many_reg"),
5524:_T("certificate","err_too_many_req"),
5525:_T("certificate","err_mail"),
5526:_T("s2s","err_invalid_param_value")

I've been able to successfully deploy an ECC certificate as follows (DS3615sx and DSM 6.1.7):

./acme.sh --test --issue -d example.com --dns dns_he --keylength ec-256
./acme.sh --test -d example.com --deploy --deploy-hook synology_dsm --ecc

With the information you've provided, I can't replicate what you are seeing.

Could the form fields/options have changed between versions as it is failing when uploading the new cert? The error code 5529 is very near to the error codes you posted so I am guessing it has something to do with the cert or its fields.

tresni commented 4 years ago

What Synology device are you using? What version of DSM are you using? I can't find error code 5529 defined. The surrounding defined errors are that I could find are:

918+ with DSM 6.2.3 rel 25426 (latest release)

Unfortunately I am not home to test with my DSM running latest and having trouble getting xpenology to run DSM 6.2 or later on VirtualBox.

Could the form fields/options have changed between versions as it is failing when uploading the new cert? The error code 5529 is very near to the error codes you posted so I am guessing it has something to do with the cert or its fields.

You could check the synoSDSjslib/sds.js file to see if it contains the the string '5529' similar to the ones I posted. Should be available at http://[synology_host]:[synology_port]/synoSDSjslib/sds.js .

ghost commented 4 years ago

You could check the synoSDSjslib/sds.js file to see if it contains the the string '5529' similar to the ones I posted. Should be available at http://[synology_host]:[synology_port]/synoSDSjslib/sds.js .

I checked and it's 5529:_T("certificate","not_support_ecc")

How does it not support ecc when with the previous acme/synology method (prior to the dsm deployhook) I was able to create and install ec-256 and ec-384 certs.

tresni commented 4 years ago

I can't answer that as I don't work for Synology. That's a response from their API. Can you add the certificates manually via the DSM webui?

I was able to get something setup to access my DS1515+ at home, it's running DSM 6.2.2 and supports ECC certificates. So either it's something new in 6.2.3 or something specific to the DS918. Reviewing the DS918+ Release Notes, the closest I see is "Added support for Let's Encrypt wildcard certificates for Synology DDNS."

ghost commented 4 years ago

@tresni This must be a regression then as I was only able to install ecc certs via shell. The webui is refusing to accept ecc certs. The previous deployment method was done via shell which is why it worked.

Thanks for your help Brian... saved me a few grey hairs.

BartSiwek commented 4 years ago

I discovered a problem today with my DS218+ - the fields in the server response are lower case and the hook uses case sensitive greps to filter the response. Net result - authentication fails.

My response (edited of tokens and ids):

deploy/synology_dsm.sh:synology_dsm_deploy:83 response='HTTP/2 200
server: nginx
date: Fri, 17 Jul 2020 16:45:23 GMT
content-type: text/html; charset="UTF-8"
vary: Accept-Encoding
x-content-type-options: nosniff
x-xss-protection: 1; mode=block
x-frame-options: SAMEORIGIN
x-syno-token: VALUEVALUEVALUE
p3p: CP="IDC DSP COR ADM DEVi TAIi PSA PSD IVAi IVDi CONi HIS OUR IND CNT"
content-security-policy: base-uri 'self';  connect-src ws: wss: *; default-src 'self' 'unsafe-eval' data: blob: https://*.synology.com https://www.synology.cn/; font-src 'self' data: https://*.googleapis.com https://*.gstatic.com; form-action 'self'; frame-ancestors 'self' https://gofile.me http://gofile.me; frame-src 'self' data: blob: https://*.synology.com https://www.synology.cn/ http://*.synology.com http://*.synology.cn; img-src 'self' data: blob: https://*.google.com https://*.googleapis.com http://*.googlecode.com https://*.gstatic.com; media-src 'self' data: about:;  script-src 'self' 'unsafe-eval' data: blob: https://*.synology.com https://www.synology.cn/ https://*.google.com https://*.googleapis.com; style-src 'self' 'unsafe-inline' https://*.googleapis.com;
set-cookie: id=VALUE;path=/
set-cookie: smid=VALUEVALUE;expires=Sat, 17-Jul-2021 16:45:22 GMT;path=/;HttpOnly
strict-transport-security: max-age=15768000; includeSubdomains; preload
'

Fix is to add -i to greps and I (capital i) to the sed:

_syno_get_cookie_data() {
  grep -i "\W$1=" | grep -i "^Set-Cookie:" | _tail_n 1 | _egrep_o "$1=[^;]*;" | tr -d ';'
}

and 

token=$(echo "$response" | grep -i "X-SYNO-TOKEN:" | sed -n 's/^X-SYNO-TOKEN: \(.*\)$/\1/pI' | tr -d "\r\n")

There might be a few more spots, but this fixed the auth for me.

BartSiwek commented 4 years ago

One more that I can't really show logs for - both the password and the certificate name can contain special character that bash does not like. When they are stored in the .conf file for the domain they cause parsing errors. This could be improved, since as of now it is required to not have a password with special chars for the hook to work.

alxwolf commented 4 years ago

DSM 6.2.3 Update 2 - When HTTP/2 is activated (under Control Panel -> Network -> DSM Settings, deployment fails:

<= Recv data, 38 bytes (0x26) 0000: {"error":{"code":119},"success":false} <= Recv SSL data, 5 bytes (0x5) 0000: ....% == Info: HTTP/2 stream 0 was not closed cleanly: INTERNAL_ERROR (err 2) == Info: stopped the pause stream! == Info: Connection #0 to host example.com left intact [Thu 23 Jul 08:22:15 CEST 2020] _ret='92' [Thu 23 Jul 08:22:15 CEST 2020] Unable to update certificate, error code {"error":{"code":119},"success":false} [Thu 23 Jul 08:22:15 CEST 2020] Error deploy for domain:example.com [Thu 23 Jul 08:22:15 CEST 2020] Deploy error.

Deactivating HTTP/2 will fix this.

tresni commented 4 years ago

@BartSiwek

Fix is to add -i to greps and I (capital i) to the sed:

Should be addressed in tresni@5f5096e (did a rebase so I can get it merged to dev)

both the password and the certificate name can contain special character that bash does not like

Yes, you have to either use a password that is bash-safe or properly escape it for use within bash.

@gittyhub2

Deactivating HTTP/2 will fix this.

Thanks, I'm looking into this. For now, I'll update the wiki to reflect this. No idea what the cause would be, curl supports HTTP/2 and I would expect it to "just work." Does the log indicate whether it was curl or wget?

tresni commented 4 years ago

@gittyhub2 I've got HTTP/2 enabled on my DS1515+ running DSM 6.2.3-25426 Update 2 and renewal is working fine with tresni@5f5096e. This sounds like the issue fixed in #2935 . That's not part of tagged release yet, but is part of master. A ./acme.sh --upgrade should get you the latest synology_dsm deployhook.

alxwolf commented 4 years ago

I've got HTTP/2 enabled on my DS1515+ running DSM 6.2.3-25426 Update 2 and renewal is working fine with tresni/acme.sh@5f5096e.

Thanks @tresni for looking into that.

I have a DiskStation DS718+ here with DSM 6.2.3-25426 Update 2, with acme.sh running on

pi@pihole:~ $ acme.sh --version https://github.com/acmesh-official/acme.sh v2.8.7`

When I compare the trace files from "initial contact" between hook and DiskStation (_CURL='curl -L --silent --dump-header /home/pi/.acme.sh/http.header --trace-ascii /tmp/tmp.tnV9KZVy0Y -g ', so I guess it's curl not wget with regards to your other question) using diff init_without_H2.log init_with_H2.log it looks to me that with HTTP/2, the reply is all lower-case, while with HTTP/1.1 the reply is mixed-case.

I can confirm the same behaviour for a DS920+ with same DSM version.

< <= Recv header, 17 bytes (0x11)
< 0000: HTTP/1.1 200 OK
---
> <= Recv header, 13 bytes (0xd)
> 0000: HTTP/2 200

[...]

227c246
< 0000: Vary: Accept-Encoding
---
> 0000: vary: Accept-Encoding
229c248
< 0000: X-Content-Type-Options: nosniff
---
> 0000: x-content-type-options: nosniff
231c250
< 0000: X-XSS-Protection: 1; mode=block
---
> 0000: x-xss-protection: 1; mode=block
233c252
< 0000: X-Frame-Options: SAMEORIGIN
---
> 0000: x-frame-options: SAMEORIGIN
235c254
< 0000: X-SYNO-TOKEN: blablabla
---
> 0000: x-syno-token: blablablabla
237c256
< 0000: P3P: CP="IDC DSP COR ADM DEVi TAIi PSA PSD IVAi IVDi CONi HIS OU
---
> 0000: p3p: CP="IDC DSP COR ADM DEVi TAIi PSA PSD IVAi IVDi CONi HIS OU
240c259
< 0000: Content-Security-Policy: base-uri 'self';  connect-src ws: wss: 
---
> 0000: content-security-policy: base-uri 'self';  connect-src ws: wss:
Profex commented 4 years ago

Tested this locally on my DS918+ running the new DSM7 preview and it fails at this step. Seems like the script is unable to retrieve the X-SYNO-TOKEN

[Wed Sep 9 14:41:02 CEST 2020] ./acme.sh:_get:1963 ret='0' [Wed Sep 9 14:41:02 CEST 2020] deploy/synology_dsm.sh:synology_dsm_deploy:83 response='HTTP/2 200 server: nginx date: Wed, 09 Sep 2020 12:41:02 GMT content-type: text/html; charset="UTF-8" x-content-type-options: nosniff x-xss-protection: 1; mode=block x-frame-options: SAMEORIGIN p3p: CP="IDC DSP COR ADM DEVi TAIi PSA PSD IVAi IVDi CONi HIS OUR IND CNT" content-security-policy: base-uri 'self'; connect-src data: ws: wss: *; default-src 'self' 'unsafe-eval' data: blob: https://*.synology.com https://www.synology.cn/ https://help.synology.cn/; font-src 'self' data: https://*.googleapis.com https://*.gstatic.com https://fonts.gstatic.com https://*.gstatic.com; form-action 'self'; frame-ancestors 'self' https://gofile.me http://gofile.me; frame-src 'self' data: blob: https://*.synology.com https://www.synology.cn/ https://account.synology.cn/ https://accounttest.synology.cn/ https://help.synology.cn/ https://www.youtube.com http://www.youtube.com http://*.synology.com http://*.synology.cn; img-src 'self' data: blob: https://*.google.com https://*.googleapis.com http://*.googlecode.com https://*.gstatic.com https://*.gstatic.com https://*.googleapis.com https://*.google.com https://i.ytimg.com https://*.gstatic.com https://*.googleapis.com https://*.google.com http://*.baidu.com https://*.bdstatic.com https://*.bdimg.com; media-src 'self' data: about: https://*.synology.com https://help.synology.cn; script-src 'self' 'unsafe-eval' data: blob: https://*.synology.com https://www.synology.cn/ https://help.synology.cn https://*.google.com https://*.googleapis.com https://*.google.com https://*.googleapis.com https://*.google.com https://*.googleapis.com https://*.baidu.com https://*.bdstatic.com https://*.bdimg.com; style-src 'self' 'unsafe-inline' https://*.googleapis.com https://*.googleapis.com https://*.googleapis.com https://api.map.baidu.com; set-cookie: id=IDREPLACED;expires=Sat, 12-Sep-2020 00:41:02 GMT;path=/;secure set-cookie: _SSID=SSIDREPLACED;path=/ ' [Wed Sep 9 14:41:02 CEST 2020] deploy/synology_dsm.sh:synology_dsm_deploy:84 token

Executing the URL manually in by browser shows the following:

{ "SynoToken" : "TOKENCODEREPLACED", "result" : "success", "success" : true }

Checked a bit more and the reason is probably simple: the http header does not include the token, can't see it in the temp file created by acme.sh nor in my brwoser when checking with developer tools. So Synology must have changed something here.

throttlemeister commented 4 years ago

Getting an error on DSM7, unable to authenticate:

$ acme --deploy -d .domain.tld --deploy-hook synology_dsm --debug 2 [Wed Sep 23 14:00:13 CEST 2020] Lets find script dir. [Wed Sep 23 14:00:13 CEST 2020] SCRIPT='/var/services/homes//.acme.sh/acme.sh' [Wed Sep 23 14:00:13 CEST 2020] _script='/volume1/homes//.acme.sh/acme.sh' [Wed Sep 23 14:00:13 CEST 2020] _script_home='/volume1/homes//.acme.sh' [Wed Sep 23 14:00:13 CEST 2020] Using default home:/var/services/homes//.acme.sh [Wed Sep 23 14:00:13 CEST 2020] Using config home:/var/services/homes//.acme.sh [Wed Sep 23 14:00:13 CEST 2020] LE_WORKING_DIR='/var/services/homes//.acme.sh' https://github.com/acmesh-official/acme.sh v2.8.8 [Wed Sep 23 14:00:13 CEST 2020] Running cmd: deploy [Wed Sep 23 14:00:13 CEST 2020] Using config home:/var/services/homes//.acme.sh [Wed Sep 23 14:00:13 CEST 2020] default_acme_server [Wed Sep 23 14:00:13 CEST 2020] ACME_DIRECTORY='https://acme-v02.api.letsencrypt.org/directory' [Wed Sep 23 14:00:13 CEST 2020] _ACME_SERVER_HOST='acme-v02.api.letsencrypt.org' [Wed Sep 23 14:00:13 CEST 2020] DOMAIN_PATH='/var/services/homes//.acme.sh/.domain.tld' [Wed Sep 23 14:00:13 CEST 2020] _deployApi='/volume1/homes//.acme.sh/deploy/synology_dsm.sh' [Wed Sep 23 14:00:13 CEST 2020] _cdomain='.domain.tld' [Wed Sep 23 14:00:13 CEST 2020] SYNO_Username='' [Wed Sep 23 14:00:13 CEST 2020] SYNO_Password='[hidden](please add '--output-insecure' to see this value)' [Wed Sep 23 14:00:13 CEST 2020] SYNO_Scheme='https' [Wed Sep 23 14:00:13 CEST 2020] SYNO_Hostname='host.domain.tld' [Wed Sep 23 14:00:13 CEST 2020] SYNO_Port='5001' [Wed Sep 23 14:00:13 CEST 2020] SYNO_Certificate='acme.sh certificate' [Wed Sep 23 14:00:13 CEST 2020] _base_url='https://host.domain.tld:5001' [Wed Sep 23 14:00:13 CEST 2020] Logging into host.domain.tld:5001 [Wed Sep 23 14:00:13 CEST 2020] GET [Wed Sep 23 14:00:13 CEST 2020] url='https://host.domain.tld:5001/webman/login.cgi?username=&passwd=&enable_syno_token=yes&device_id='DID_COOKIE_TOKEN' [Wed Sep 23 14:00:13 CEST 2020] timeout= [Wed Sep 23 14:00:13 CEST 2020] _CURL='curl -L --silent --dump-header /var/services/homes//.acme.sh/http.header --trace-ascii /tmp/tmp.9pJRH5nk4X -g ' [Wed Sep 23 14:00:15 CEST 2020] ret='0' [Wed Sep 23 14:00:15 CEST 2020] token [Wed Sep 23 14:00:15 CEST 2020] Unable to authenticate to host.domain.tld:5001 using https. [Wed Sep 23 14:00:15 CEST 2020] Check your username and password. [Wed Sep 23 14:00:15 CEST 2020] Error deploy for domain:.domain.tld [Wed Sep 23 14:00:15 CEST 2020] Deploy error.

throttlemeister commented 4 years ago

Did some investigating. As stated by @Profex the X-SYNO-TOKEN is lacking from DSM7 pages., but digging through the source gives me a token of "--------". Hardcoding this value into the token variable from synology_dsm.sh makes it pass the login, but now it craps out on curl.

If I have http/2 turned on in DSM, it fails with a curl error 51, which is:

CURLE_PEER_FAILED_VERIFICATION (60)

The remote server's SSL certificate or SSH md5 fingerprint was deemed not OK. This error code has been unified with CURLE_SSL_CACERT since 7.62.0. Its previous value was 51.

And from teh debug: == Info: HTTP/2 stream 0 was not closed cleanly: INTERNAL_ERROR (err 2)

Disabling http/2, results in a curl error 18, which is:

CURLE_PARTIAL_FILE (18)

A file transfer was shorter or larger than expected. This happens when the server first reports an expected transfer size, and then delivers data that doesn't match the previously given size.

And from the debug: == Info: transfer closed with outstanding read data remaining

I am pretty much done with what I can; I am not a developer. However, I hope that maybe this helps a bit.

throttlemeister commented 4 years ago

Oh, and if I disable http compression, but with http/2 I get error 92:

CURLE_HTTP2_STREAM (92)

Stream error in the HTTP/2 framing layer.

And disabling http/2 and compression makes it go back to error 18 above.

tresni commented 4 years ago

My device does not support DSM7 (yet?) so I'm looking to see if I can setup a virtualized instance like I have previously.

tresni commented 4 years ago

Looks like the whole reason we have d15c14ab939682e0327202a78ef2470df7472181 is why this is not going to work on DSM 7. The issue is that we need something returned in the body (SynoToken) and in the headers (previously the id and smid cookies.) In DSM6 and earlier, the token was also returned as the X-Syno-Token header, but that no longer seems to be the case. Additionally, the smid cookie looks to be replaced by the _SSID cookie (this is an open question as I don't have headers to compare with.)

tresni commented 4 years ago

Okay, I found a way to get the login to work using POST. If you would like to see if this works for you, it's currently available at tresni@bb95164. I haven't opened a PR for this yet as I want to see if there is more we need to do for DSM 7.

Interesting side investigation: DSM 6.2.3 actually uses client-side encryption (AES/RSA) if you login via your browser, but it's not required. It gets the necessary information from /webapi/encryption.cgi using a POST with the payload being format=module&api=SYNO.API.Encryption&method=getinfo&version=1. That returns a JSON object with details like the public key to use, server time, and how to return the encrypted data. Checkout /webman/desktop.js and look for onEncryptParams.

throttlemeister commented 4 years ago

Thanks @tresni .

I see the post_url and body with correct username and password and DID, however it errors with: [Sun Sep 27 12:04:35 CEST 2020] Unable to authenticate to host.domain.tld:5001 using https. [Sun Sep 27 12:04:35 CEST 2020] Check your username and password. [Sun Sep 27 12:04:35 CEST 2020] Error deploy for domain:*.domain.tld [Sun Sep 27 12:04:35 CEST 2020] Deploy error.

Profex commented 4 years ago

Facing exactly the same issue, I can see the success message in debug, but no token

[Sun Sep 27 10:32:41 CEST 2020] ./acme.sh:_post:1899 _ret='0' [Sun Sep 27 10:32:41 CEST 2020] deploy/synology_dsm.sh:synology_dsm_deploy:83 response='{ "SynoToken" : "", "result" : "success", "success" : true }' [Sun Sep 27 10:32:41 CEST 2020] deploy/synology_dsm.sh:synology_dsm_deploy:84 token [Sun Sep 27 10:32:41 CEST 2020] Unable to authenticate to xxxxxx:5281 using HTTPS. [Sun Sep 27 10:32:41 CEST 2020] Check your username and password. [Sun Sep 27 10:32:41 CEST 2020] Error deploy for domain:*.xxxx.yy [Sun Sep 27 10:32:41 CEST 2020] Deploy error.

tresni commented 4 years ago

@Profex that seems to indicate the login was successful, but, unlike previous posts, there is no SynoToken in the response. Did you edit it out?

If someone is comfortable with Chrome or Firefox developer tools, it would be really handy to get a HAR file of the login sequence to see if something has changed.

Profex commented 4 years ago

@tresni no, nothing was edited out in this area, I just did edit the URL and domain name. So this is exactly the output I get in console debug trace for Synotoken and also in the temporary dump file created by the script

When I call the very same URL in Chrome after authenticating to DSM 7 via web gui the same content is returned, with the difference that a SynoToken is included

Need to look into the HAR thing, did not use it before

Normal web browser call { "SynoToken" : "EDITEDEDITED", "result" : "success", "success" : true }

Console debug level 3 [Sun Sep 27 20:49:45 CEST 2020] ./acme.sh:_post:1790 _post_url='HTTPS://EDITED:5281/webman/login.cgi?enable_syno_token=yes' [Sun Sep 27 20:49:45 CEST 2020] ./acme.sh:_post:1791 body='username=admin&passwd=EDITED&device_id=EDITED' [Sun Sep 27 20:49:45 CEST 2020] ./acme.sh:_post:1792 _postContentType [Sun Sep 27 20:49:45 CEST 2020] ./acme.sh:_exists:514 curl exists=0 [Sun Sep 27 20:49:45 CEST 2020] ./acme.sh:_exists:514 mktemp exists=0 [Sun Sep 27 20:49:45 CEST 2020] ./acme.sh:_exists:514 wget exists=0 [Sun Sep 27 20:49:45 CEST 2020] ./acme.sh:_post:1804 _CURL='curl -L --silent --dump-header ./http.header --trace-ascii /tmp/tmp.LESOZt57G4 -g ' [Sun Sep 27 20:49:45 CEST 2020] ./acme.sh:_post:1899 _ret='0' [Sun Sep 27 20:49:46 CEST 2020] deploy/synology_dsm.sh:synology_dsm_deploy:83 response='{ "SynoToken" : "", "result" : "success", "success" : true }' [Sun Sep 27 20:49:46 CEST 2020] deploy/synology_dsm.sh:synology_dsm_deploy:84 token [Sun Sep 27 20:49:46 CEST 2020] Unable to authenticate to EDITED:5281 using HTTPS. [Sun Sep 27 20:49:46 CEST 2020] Check your username and password. [Sun Sep 27 20:49:46 CEST 2020] Error deploy for domain: EDITED [Sun Sep 27 20:49:46 CEST 2020] Deploy error.

file /tmp/tmp.LESOZt57G4 <= Recv header, 70 bytes (0x46) 0000: set-cookie: _SSID=EDITED 0040: th=/ <= Recv header, 2 bytes (0x2) 0000: <= Recv data, 70 bytes (0x46) 0000: {. "SynoToken" : "",. "result" : "success",. "success" : t 0040: rue.}. <= Recv SSL data, 5 bytes (0x5) 0000: ..... == Info: TLSv1.3 (IN), TLS app data, [no content] (0): <= Recv SSL data, 1 bytes (0x1) 0000: .

throttlemeister commented 4 years ago

looking at debvug 3 output, that is the exact same response I get as well.

[Mon Sep 28 08:56:20 CEST 2020] /var/services/homes//.acme.sh/acme.sh:_post:1789 POST [Mon Sep 28 08:56:20 CEST 2020] /var/services/homes//.acme.sh/acme.sh:_post:1790 _post_url='https://host.domain.tld:5001/webman/login.cgi?enable_syno_token=yes' [Mon Sep 28 08:56:20 CEST 2020] /var/services/homes//.acme.sh/acme.sh:_post:1791 body='username=&passwd=&device_id=' [Mon Sep 28 08:56:20 CEST 2020] /var/services/homes//.acme.sh/acme.sh:_post:1792 _postContentType [Mon Sep 28 08:56:20 CEST 2020] /var/services/homes//.acme.sh/acme.sh:_exists:514 curl exists=0 [Mon Sep 28 08:56:20 CEST 2020] /var/services/homes//.acme.sh/acme.sh:_exists:514 mktemp exists=0 [Mon Sep 28 08:56:20 CEST 2020] /var/services/homes//.acme.sh/acme.sh:_exists:514 wget exists=0 [Mon Sep 28 08:56:20 CEST 2020] /var/services/homes//.acme.sh/acme.sh:_post:1804 _CURL='curl -L --silent --dump-header /var/services/homes//.acme.sh/http.header --trace-ascii /tmp/tmp.ov3O9nhBk7 -g ' [Mon Sep 28 08:56:22 CEST 2020] /var/services/homes//.acme.sh/acme.sh:_post:1899 _ret='0' [Mon Sep 28 08:56:22 CEST 2020] deploy/synology_dsm.sh:synology_dsm_deploy:83 response='{ "SynoToken" : "", "result" : "success", "success" : true }' [Mon Sep 28 08:56:22 CEST 2020] deploy/synology_dsm.sh:synology_dsm_deploy:84 token [Mon Sep 28 08:56:22 CEST 2020] Unable to authenticate to host.domain.tld:5001 using https. [Mon Sep 28 08:56:22 CEST 2020] Check your username and password. [Mon Sep 28 08:56:22 CEST 2020] Error deploy for domain:*.domain.tld [Mon Sep 28 08:56:22 CEST 2020] Deploy error.

BartSiwek commented 4 years ago

I had a similar issue just now. In my case the two factor authentication SYNO_DID token expired. When I updated it with the new one everything went through correctly. If these tokens expire, is there maybe a better way to handle that?

throttlemeister commented 3 years ago

@tresni any news on this Brian? You already got links for DSM7 in other threads, but we can get you any file for use in VirtualDSM if you want so you can play with DSM7 as well.

tresni commented 3 years ago

I'm an idiot and didn't realize I had a free license for Virtual DSM... I've got that setting up now and will look at this soon.

BartSiwek commented 3 years ago

Once again had this deploy hook fail due to expired SYNO_DID token. Is there a way to configure this so they don't expire? Or maybe a better approach to 2-factor auth?

throttlemeister commented 3 years ago

Once again had this deploy hook fail due to expired SYNO_DID token. Is there a way to configure this so they don't expire? Or maybe a better approach to 2-factor auth?

How about creating a separate admin account without 2FA and an insane 30-character or worse random password and use that for your scripts?

Your issue has nothing to do with the deploy-hook, but everything to do with 2FA. It is supposed to expire. In fact, it having the option to cache it for a period of time is already a security risk. The fact you can copy the token and carry it over to another session and a even different computer and actually use it, makes my want to ask Synology "WTF"? It's a security breach that should not be possible, And it looks to be fixed in DSM7.