Problems reported when using NGINX by Ondrej PPA

franbuehler commented 3 months ago

The CRS team believes that strange CRS problems are reported when users use NGINX Ondrej PPA.

Reports:

SO link to follow FIXME

What we think the problem is: FIXME

We would like to solve this problem (if any) or have it documented so users understand how to follow.

Statements from chat so that we don't lose them:

We believe that the Ondrej PPA is responsible for the very strange errors we've seen reported over the past 2 years. E.g. "200 is not a number" and very odd things like that. (bearbeitet) (LINK?) We've had, I think, 4 separate reports.
Yes, I think one remembers issues with that PPA and some bad dependencies. (LINK)
yeah, there were some PCRE issue (LINK)
Ondrey likes to use his own PCRE package (because of the PHP) (clarified the why below)

dune73 commented 3 months ago

We have a history of poor cooperation with the Ondrej PPA lead developer. I think the best thing we can do is a blog post where we explain the situation and that it's hard for us to support this.

fzipi commented 2 months ago

Do we want to transfer this one to the website repo?

dune73 commented 2 months ago

Sounds like a plan

oerdnj commented 1 month ago

We have a history of poor cooperation with the Ondrej PPA lead developer. I think the best thing we can do is a blog post where we explain the situation and that it's hard for us to support this.

I’m all ears…

fzipi commented 1 month ago

Now that we have @oerdnj attention 😄 (I'm surprised and glad you are here, 👋 btw), we probably should add what we think the problems (FIXME) were in the original description to see how/if things can be fixed, or just need better documentation on our side (or just the next step).

oerdnj commented 1 month ago

Now that we have @oerdnj attention 😄 (I'm surprised and glad you are here, 👋 btw), we probably should add what we think the problems (FIXME) were in the original description to see how/if things can be fixed, or just need better documentation on our side (or just the next step).

You know you could have tagged me and asked me before you start saying things about bad cooperation…. But let’s start afresh…

The PCRE2 library isn’t actually part of the nginx repository. I know that the nginx packages were (maybe still are) a mess in the past few months - it comes from the fact that Debian upstream maintainer unbundled the plugins and made some decisions that were not compatible with the PPAs. Unfortunately, this was inevitable as Ubuntu 24.04 picked on that, so I had to follow the suit, and it took me a while to figure out all the nifty details. And there’s apparently still something elusive that’s causing people problems on upgrades. I’m not omnipotent and I make mistakes.

And I am usually uncooperative only in case when people come and demand I do things for them for free. But I do try to listen to a solid argument.

fzipi commented 1 month ago

Of course, I assumed exactly that. This is, IMHO, the opposite of being "unresponsive" 😄 😄 .

Let's figure it out what happens... but as you said clearly here, and I totally agree: let's get a solid argument. I do not like not having a way to reproduce the problem. 💪

airween commented 1 month ago

The PCRE2 library isn’t actually part of the nginx repository.

It isn't. But it's part of your PHP repository.

In most cases (perhaps in all cases) the users who reported issues used some PHP application, and we assume all the used components installed from your repository. But it is also possible that the two have nothing to do with each other.

But the fact that there is a situation: if users install Nginx from your repository, then libmodsecurity3 produces the same weird behavior:

and some other similar reports on mailing list/StackOverflow forums.

If users switch to Debian's version everything is solved.

The weird behavior is that the @rx operator (which does a simple regular expression pattern match) does not work as expected. Therefore, we can only think that the PCRE library used has some error.

I risk in all cases the reporter built the library (libmodsecurity3) from source, therefore the build flow used the system's PCRE library - so this is why we think that the user installed a non-default PCRE library.

oerdnj commented 1 month ago

The weird behavior is that the @rx operator (which does a simple regular expression pattern match) does not work as expected. Therefore, we can only think that the PCRE library used has some error.

So, what I am actually hearing is that it has nothing to do with NGINX repository, and it affects mod_security only when PCRE2 (I assume this is PCRE2, right?) is updated to the latest version? The PCRE2 maintainer is usually very responsive if there’s a test case.

Anyway, this should be fairly easy to test - Debian stable has PCRE2 10.42-1, Debian testing and Ubuntu 24.04 has PCRE2 10.42-4. My repositories has 10.42-3 (and the only difference between -3 and -4 is riscv64 support)

Can you observe the problematic behavior on those systems? Perhaps this boils down to - do you have means to reproduce the weird behavior?

airween commented 1 month ago

So, what I am actually hearing is that it has nothing to do with NGINX repository, and it affects mod_security only when PCRE2 (I assume this is PCRE2, right?) is updated to the latest version? The PCRE2 maintainer is usually very responsive if there’s a test case.

Unfortunately we don't know any details about that.

Here is a link that points the reporter's howto, but that's unavailable now. In the next comment I tried the request with same config, but it worked for me - but I was using Ubuntu's Nginx.

Btw I'm not sure it was built with PCRE2, because in Ubuntu 22.04 - as I remember - the default PCRE engine version was the 3, not the 2.

Anyway, this should be fairly easy to test - Debian stable has PCRE2 10.42-1, Debian testing and Ubuntu 24.04 has PCRE2 10.42-4. My repositories has 10.42-3 (and the only difference between -3 and -4 is riscv64 support)

Can you observe the problematic behavior on those systems? Perhaps this boils down to - do you have means to reproduce the weird behavior?

Based on the most valuable issue and its comment, I would try to follow this one. Install Nginx from your repository (onto an Ubuntu 22.04 - just for sure), install libmodsecurity3 from source - perhaps you should try with PCRE3 first - then install the connector. After that install CRS, and try to send the mentioned request.

If you need any help to configure/build the WAF just let us know.

oerdnj commented 1 month ago

Btw I'm not sure it was built with PCRE2, because in Ubuntu 22.04 - as I remember - the default PCRE engine version was the 3, not the 2.

That might be the problem - PCRE (libpcre3) is obsolete for couple of years now and it doesn't receive any updates and fixes anymore.

I also see this:

We had the same problem. It turned out that we did not supply --with-pcre2 during configuration.

And I see issues like this in modsecurity:

So, instead of saying the repository is at fault, I would suggest that ModSecurity uses PCRE2 by default, and for all bugreports where people compile libmodsecurity3 by hand, first suggest to use PCRE2 (libpcre2) instead of PCRE1 (libpcre3).

I understand it's very confusing that libpcre3 is PCRE1 (unsupported) and libpcre2 is PCRE2, but that's related to the SONAMEs of the libraries and is a different number than PCRE vs PCRE2.

oerdnj commented 1 month ago

Hmm, I am looking at https://github.com/coreruleset/coreruleset/issues/3181#issuecomment-1490624870 and this says the user compiled nginx from the source.

airween commented 1 month ago

That might be the problem - PCRE (libpcre3) is obsolete for couple of years now and it doesn't receive any updates and fixes anymore.

I know, but if I'm right the most packages upgraded in Debian 12 - before that every application used PCRE (libpcre3), eg. Nginx. This is why was (and it's still) the default in libmodsecurity3.

We had the same problem. It turned out that we did not supply --with-pcre2 during configuration.

And I see issues like this in modsecurity:
* [PCRE2 support still requires PCRE1 owasp-modsecurity/ModSecurity#2966](https://github.com/owasp-modsecurity/ModSecurity/issues/2966)

* [PCRE2 support still requires PCRE1 owasp-modsecurity/ModSecurity#2750](https://github.com/owasp-modsecurity/ModSecurity/issues/2750)

yes, there was an issue, but I'm not sure that had any effect.

So, instead of saying the repository is at fault, I would suggest that ModSecurity uses PCRE2 by default, and for all bugreports where people compile libmodsecurity3 by hand, first suggest to use PCRE2 (libpcre2) instead of PCRE1 (libpcre3).

There is a intention that we will upgrade the source in case of both engines and the default regex engine will be the PCRE2.

I understand it's very confusing that libpcre3 is PCRE1 (unsupported) and libpcre2 is PCRE2, but that's related to the SONAMEs of the libraries and is a different number than PCRE vs PCRE2.

I know this, but here I think it's unrelated too. But may be I'm wrong.

Just a question: with which regex engine you used to built your Nginx packages?

airween commented 1 month ago

Hmm, I am looking at coreruleset/coreruleset#3181 (comment) and this says the user compiled nginx from the source.

Yes, in this case he did. But in most cases users used your repository.

oerdnj commented 1 month ago

Just a question: with which regex engine you used to built your Nginx packages?

Good question - I actually don't do anything specific and nginx (rather ./configure ... --with-pcre-jit) picks up system PCRE2 library. This is nginx 1.26.0 in ppa:ondrej/nginx and nginx 1.27.x in ppa:ondrej/nginx-mainline now.

airween commented 1 month ago

Good question - I actually don't do anything specific and nginx (rather ./configure ... --with-pcre-jit) picks up system PCRE2 library. This is nginx 1.26.0 in ppa:ondrej/nginx and nginx 1.27.x in ppa:ondrej/nginx-mainline now.

On each systems? I mean Debian 11 and "older" (but still supported) Ubuntus? Just because on Debian 11 the default pcre engine was the old one (PCRE3), and Nginx package depends on libpcre3:

$ ldd /usr/sbin/nginx
        ...
    libpcre.so.3 => /lib/x86_64-linux-gnu/libpcre.so.3 (0x00007f6080635000)
        ...

This changed in Debian 12:

$ ldd /usr/sbin/nginx 
        ...
    libpcre2-8.so.0 => /lib/x86_64-linux-gnu/libpcre2-8.so.0 (0x00007fe10637f000)
        ...

(note: replacing the old PCRE had started somewhere here: https://lists.debian.org/debian-devel/2021/11/msg00176.html)

So if you always built your packages with pcre2, and users built the library with pcre3, then - I don't know why/how - it could cause the behavior....

oerdnj commented 1 month ago

yes, there was an issue, but I'm not sure that had any effect.

I actually can't even reproduce the linking problem, but that could be effect of default Debian linker (Debian bookworm) that removes unused libraries as I see linkage to -lpcre -lpcre2-8, but dynamic linker ldd libmodsecurity.so.3 doesn't show libpcre.so.3 in the dependencies.

oerdnj commented 1 month ago

On each systems?

Yes, nginx configure script actually picks up the PCRE2 as default library if that's installed on the system.

So if you always built your packages with pcre2, and users built the library with pcre3, then - I don't know why/how - it could cause the behavior....

Yes, that's pretty much confusing. Perhaps the discrepancy in the regex processing causes an error on a different layer? I don't know, I know literally nothing about ModSecurity.

oerdnj commented 1 month ago

Do you have any recent case? Or perhaps if there's a new one, can you point me to it, and I'll poke the user around?

I don't think this is directly related to the PPAs (and Debian repositories), but is rather some side-effect of changes to nginx. And if that's the case, I am guessing it is not going to be isolated problem.

BTW if you install nginx-dev, it pulls the pcre packages used to build nginx.

oerdnj commented 1 month ago

I can also take a look and perhaps I can start providing libnginx-mod-http-modsecurity and libmodsecurity3 directly from the ppa:ondrej/nginx, ppa:ondrej/nginx-mainline and libapache2-mod-security2 from ppa:ondrej/apache2.

I do have a feature request for this already: https://github.com/oerdnj/deb.sury.org/issues/1192

airween commented 1 month ago

Do you have any recent case? Or perhaps if there's a new one, can you point me to it, and I'll poke the user around?

Unfortunately (or fortunately? :smile:) not. There are too much impulses around here, I just looked for GH issue search mechanism and could find the mentioned three. But we were constantly following the SO (StackOveflow) forums, we have a mailing list and a Slack channel - as far as I remember, we got very similar problems on almost every channel.

I don't think this is directly related to the PPAs (and Debian repositories), but is rather some side-effect of changes to nginx. And if that's the case, I am guessing it is not going to be isolated problem.

Sounds not so good.

BTW if you install nginx-dev, it pulls the pcre packages used to build nginx.

Nginx-dev exists only since Bookworm: https://packages.debian.org/search?keywords=nginx-dev

As I see the previous versions of Nginx (eg. the last non-modular version) depends on libpcre3: https://salsa.debian.org/nginx-team/nginx/-/blob/bullseye/debian/control?ref_type=heads#L18

I assume the mentioned Ubuntu systems (like 22.04) also followed this build config.

But for sure: we should try to reproduce this behavior. Well, perhaps we need to try it on an older system (Debian 11 or Ubuntu 22.04).

coreruleset / website

Problems reported when using NGINX by Ondrej PPA #113