search

searxng / searx-space

Statistics of the public SearX(NG) instances
https://searx.space/
GNU Affero General Public License v3.0
170 stars 27 forks source link

TLS grade having a hard time to update #58

Open Zaczero opened 3 years ago

Zaczero commented 3 years ago

This is a follow up to #50.

So once again I am having an issue with my instance's TLS grade as apparently my ciphers are too modern. Talking about searx.monicz.pl here. And here is a result from cryptcheck itself: https://cryptcheck.fr/https/searx.monicz.pl

I believe this is related to me using x25519 curve for a handshake. And here is an issue posted on cryptcheck's repository: https://github.com/aeris/cryptcheck/issues/30

+here are some extra TLS details from the ssllabs guys https://www.ssllabs.com/ssltest/analyze.html?d=searx.monicz.pl

A few words from me: x25519 is not an unusual curve to choose. It has been widely supported for a few good years now. From the ssllabs result you may find that my encryption is valid for all modern (and not) browsers like Chrome 69, Firefox 62.

My opinion is that cryptcheck is currently unable to process modern encryption thus an alternative should be found. Fortunately there are a few open-source projects which focus on bring ssllabs API to life. Learn more at https://www.ssllabs.com/projects/ssllabs-apis/index.html Some of them are developed in python so I believe that the implentation itself should not be a big of a hassle.

Ssllabs has been keeping up with the latest TLS improvements and vulnerabilities. I would say that it is a service of choice when it comes to testing your website's TLS configuration. And it also provides a TLS grading similar to cryptcheck's one.

unixfox commented 3 years ago

We won't switch to SSLLabs (see https://github.com/searx/searx-stats2/issues/10#issuecomment-572003677).

You may try to bump the issue that got opened on cryptcheck if you want, it seems like there was no response for 2 years: https://github.com/aeris/cryptcheck/issues/30

Or if you find an alternative that support x25519 curve for the handshake and is as good as cryptcheck then we can consider to switch to it.

Zaczero commented 3 years ago

So I believe that SSL Labs API with a custom scoring engine (similar to cryptcheck) would be accepted as a solution (since it comes only to their grading policy). Correct? @unixfox

unixfox commented 3 years ago

Why not, if you are willing to contribute about it, we can try to implement this custom scoring engine.

Zaczero commented 3 years ago

Alright. It is going to be an interesting python experience for me. Even though I know a theory it has never been my language of choice - wish me a good luck :-). Also currently my schedule is a little busy so I am not going to give any ETA on the contribution. Any suggestion to which repo should I make a pull request with my change? Or should it be a brand new project?

unixfox commented 3 years ago

If I'm not mistaken it is the same repo as you created the issue. No need for a separate repo if you are able to incorporate the custom scoring inside searx-stats2.

Maybe @dalf will know more about this.

dalf commented 3 years ago

Yes, you can fork this repository and follow the instructions in README.md

The purpose is to change this cryptcheck_backend.py (and rename it).

More precisely, and readling https://github.com/TrullJ/ssllabs/blob/master/ssllabsscanner.py and https://github.com/ssllabs/ssllabs-scan/blob/master/ssllabs-api-docs-v3.md, for each host:

In cryptcheck_backend.py, the function analyse(host) is called for each valid searx instance: https://github.com/searx/searx-stats2/blob/04f62fb9b3f80c12de64566eab3521c7f622090c/searxstats/fetcher/cryptcheck_backend.py#L37-L38

Currently, there are 2 call in parallel, see limit=2: https://github.com/searx/searx-stats2/blob/04f62fb9b3f80c12de64566eab3521c7f622090c/searxstats/fetcher/cryptcheck_backend.py#L70

The output of analyse(host) is a tuple: (rank, url).

You can make HTTP request: 

async with new_client() as session:
  response = await session.get(api_url, timeout=HTTP_REQUEST_TIMEOUT)

The new_client function returns a HTTPX client. The API is very similar to requests (a lot of Python code use requests).

You can check for one or few URL from the command line: python3 -m searxstats --cache /usr/local/searx-stats2/cache --upgrade-https-grade https://a.instance https://another.instance

Then you can check the output and / or the file html/data/instance.json.

Zaczero commented 3 years ago

@dalf That is a one, very detailed response. Thank you for that! It will help me for sure 😃. As I said, don't expect any pull requests yet. I still have a few things to work on.

Zaczero commented 3 years ago

Hey @dalf

I finally feel confident enough in my Python skills to take care of this issue. However, I have a question regarding the structure of the instance cache/output file (more detail on that later).

First of all, I have decided to use the https://github.com/devolo/ssllabs package as it seems to be the most actively maintained one.

This package requires me to update the httpx package from 0.11 to 0.16 which breaks a few things, namely:

After spending some time with the code I found out that dropping this function would break the fetcher/basic.py which I think is responsible for gathering Certificate details (name, altname, country, org etc.) visible on the searx.space website.

Now I want to fill that missing details with the new fetcher/ssllabs... fetcher.

The only issue I am having right now is that the dictionary structure is not very clear to me and it would help a lot (+save time) if you could simply provide me the cache/output file with dumped dictionary structure. This way when implementing certificate details I would be sure not to break any rendering on the searx.space website. I could generate the file myself but it would require me to get cryptcheck docker instance running and I think there would be many unnecessary steps to it.

I suppose it looks something like:

...
tls {
  version: "..",
  serialNumber: "..",
  ...
  grade: "A+",
  gradeUrl: "https://userendpoint"
}
...

The full file would be very handy in making sure I have not made any mistakes and would help me to visualize things. And you, most likely, have the whole setup ready to go.

Zaczero commented 3 years ago

Bump @dalf

dalf commented 3 years ago

I'm sorry for long delay to answer.

First of all, I have decided to use the https://github.com/devolo/ssllabs package as it seems to be the most actively maintained one.

See https://github.com/searx/searx-stats2/issues/10#issuecomment-572003677 Perhaps SSL labs has been updated since last year ? If not, we will stick to cryptcheck.

About the SSL certificates, you are right. It is possible with a hook, see https://github.com/encode/httpcore/issues/107#issuecomment-643201471