nnue file during build is empty

[Ghotrix]

Default net at the moment is nn-82215d0fd0df.nnue. During make process, download of it results in empty file.

[vondele]

not here. Can you try again? You need to delete the empty file before starting (probably a previous kill of the make process could result in that empty file).

[Ghotrix]

tried again from scratch, it fails on 627 line of Makefile with error 77, I assume that's because curl doesn't like chain certificate and exits

[vondele]

possibly, what does curl --version say ? (and what's the OS / version)

second question, can you get it with wget -qO-

[vondele]

Not unrelated, we'll at least check the downloaded file in the future (https://github.com/official-stockfish/Stockfish/pull/3000).

[Ghotrix]

curl --version
curl 7.68.0 (x86_64-conda_cos6-linux-gnu) libcurl/7.68.0 OpenSSL/1.1.1d zlib/1.2.11 libssh2/1.8.2
Release-Date: 2020-01-08
Protocols: dict file ftp ftps gopher http https imap imaps pop3 pop3s rtsp scp sftp smb smbs smtp smtps telnet tftp 
Features: AsynchDNS GSS-API HTTPS-proxy IPv6 Kerberos Largefile libz NTLM NTLM_WB SPNEGO SSL TLS-SRP UnixSockets

$ wget -qO- https://tests.stockfishchess.org/api/nn/nn-82215d0fd0df.nnue > nn-82215d0fd0df.nnue 
$ stockfish 
Stockfish 130820 by the Stockfish developers (see AUTHORS file)
uci
id name Stockfish 130820
id author the Stockfish developers (see AUTHORS file)
...
uciok
setoption name Use NNUE value true
setoption name EvalFile value nn-82215d0fd0df.nnue
go depth 5
info string NNUE evaluation using nn-82215d0fd0df.nnue enabled.
info depth 1 seldepth 1 multipv 1 score cp 0 nodes 20 nps 20000 tbhits 0 time 1 pv d2d4
info depth 2 seldepth 2 multipv 1 score cp 43 nodes 42 nps 42000 tbhits 0 time 1 pv d2d4 a7a6
info depth 3 seldepth 3 multipv 1 score cp 8 nodes 152 nps 152000 tbhits 0 time 1 pv a2a3 a7a6 d2d4
info depth 4 seldepth 4 multipv 1 score cp 7 nodes 368 nps 184000 tbhits 0 time 2 pv g1f3 c7c5 d2d4 c5d4
info depth 5 seldepth 5 multipv 1 score cp 24 nodes 688 nps 344000 tbhits 0 time 2 pv c2c4 c7c5 g1f3
bestmove c2c4 ponder c7c5

[Ghotrix]

Interesting, it seems something wrong with conda's curl. After deactivation of conda's env build from scratch finishes successfully.

[vondele]

OK, definitely not an old curl. It would be interesting to figure out how we can identify this problem, or if there is something server-side that is causing this.

[Ghotrix]

Runing conda's curl mannualy gives:

curl https://tests.stockfishchess.org/api/nn/nn-82215d0fd0df.nnue                       
curl: (77) error setting certificate verify locations:
  CAfile: /tmp/trizen-ghotrix/anaconda/pkg/anaconda/opt/anaconda/ssl/cacert.pem
  CApath: none

it seems it's purely anaconda related bug. It cannot get cacert because it (shell, curl?) builds the wrong path to it. Closing the issue. Thanks for the help!

[vondele]

OK, thanks, I also verified that the same version of curl@7.68.0 does the job locally.

[snicolet]

I get the same behavior (empty nnue file after make net) on one of my Mac machines, while on the same machine downloading https://tests.stockfishchess.org/api/nn/nn-82215d0fd0df.nnue works.

It is related to certificate problems for curl (error 60).

[snicolet]

curl --version output :

curl 7.30.0 (x86_64-apple-darwin13.0) libcurl/7.30.0 SecureTransport zlib/1.2.5
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtsp smtp smtps telnet tftp 
Features: AsynchDNS GSS-Negotiate IPv6 Largefile NTLM NTLM_WB SSL libz 

[snicolet]

Still investigating the problem: curl https://tests.stockfishchess.org/api/nn/nn-82215d0fd0df.nnuegives a redirection to https://data.stockfishchess.org/nn/nn-82215d0fd0df.nnue, this is probably OK.

The resource was found at https://data.stockfishchess.org/nn/nn-82215d0fd0df.nnue; you should be redirected automatically.

But curl https://data.stockfishchess.org/nn/nn-82215d0fd0df.nnue gives the certificate problem:

curl: (60) SSL certificate problem: Invalid certificate chain
More details here: http://curl.haxx.se/docs/sslcerts.html
curl performs SSL certificate verification by default, using a "bundle"
 of Certificate Authority (CA) public keys (CA certs). If the default
 bundle file isn't adequate, you can specify an alternate file
 using the --cacert option.
If this HTTPS server uses a certificate signed by a CA represented in
 the bundle, the certificate verification probably failed due to a
 problem with the certificate (it might be expired, or the name might
 not match the domain name in the URL).
If you'd like to turn off curl's verification of the certificate, use
 the -k (or --insecure) option.

[vondele]

The redirection is as planned (that should be the -L option).

If I build curl 7.30 it replies with

curl: (1) Protocol https not supported or disabled in libcurl

but that's probably on my side, probably 7years ago https wasn't enabled by default.

[snicolet]

@vondele Adding the -k option to curl in line 626 of Makefile solved the problem for me. I don't know if the -k option is specific to Mac versions of curl, or is universal (I suppose universal).

The cost in security is probably negligeable, especially if we make SHA256 verification of the downloaded evaluation network file :-)

[vondele]

I can make that part of the sha verification commit.

I guess this old version of curl has a problem with the verification, since newer versions consider the connection secure.

So you don't mind if people sniff on which networks you're using ;-)

[vondele]

@ppigazzini just FYI.

[snicolet]

I can make that part of the sha verification commit.

Nice. I would suggest to also switch on the relevant "insecure download" flag in wget.

[vondele]

I would leave it like that, since I'm rather sure the connection is actually secure, but there is bug in the old curl preventing it from realizing that.