search

openwrt / packages

Community maintained packages for OpenWrt. Documentation for submitting pull requests is in CONTRIBUTING.md
GNU General Public License v2.0
3.94k stars 3.45k forks source link

Migration from PCRE to PCRE2 #22006

Closed BKPepe closed 4 months ago

BKPepe commented 1 year ago

Due to recent findings described in https://github.com/openwrt/packages/pull/21800, we should track somewhere (= here) packages that are still using PCRE instead of PCRE2 to propose their developers to look into PCRE2 and prepare the support (Assignee: Here maintainers of relevant packages).

Reasons to switch to PCRE - Perl Compatible Regular Expressions 2:

Packages, which needs to be taken care of:

Packages in other feeds, which needs to be taken care of:

I ask anyone to be responsible and look into it, sooner than later. Otherwise, if there is no response to the package, which you are maintaining, the package will be removed at the beginning of the next month - 1st October 2023, so we have a soft deadline. I am fully aware that if there is no deadline, no further actions will be taken. You are not doing it just for me but also for others, including OpenWrt users.

Thank you for your cooperation, guys and make sure that it is run-tested on the router! ;)

1715173329 commented 1 year ago

shadowsocks-libev

I guess it's time to switch to shadowsocks-rust, which has much better performance (10x faster on aarch64 platform).

stintel commented 1 year ago

https://github.com/net-snmp/net-snmp/issues/420

BKPepe commented 1 year ago

Thanks, @stintel, I included your referenced issue to OP.

padre-lacroix commented 1 year ago

For the package kismet, I originally become the maintainer to prevent this package to be abandoned. But, I do not wish to maintain it anymore, mostly because this is a package that I do not use and also because I have no way to test it. If somebody wants to become the maintainer, then I am OK with that, otherwise it can be dropped.

jefferyto commented 1 year ago

slang2 is done in #22020.

neheb commented 11 months ago

I looked into kismet before. seems to now be a convoluted project with protocol buffers and a bunch of other dependencies.

edit: also if I had to guess, kismet being C++ would mean pcre is not used.

Ansuel commented 11 months ago

@BKPepe is it ok to include patch while we wait for a release? (for pcre2 support)

nxhack commented 11 months ago

@BKPepe mg editor done.

BKPepe commented 11 months ago

I looked into kismet before.

@neheb The question is do we care about quality over quantity? Many packages here are not maintained for various reasons. Almost everyone can add a new package to this repository, no matter what. No votes, if it has purpose or if it makes any sense. I know it will sound bad, but I would rather remove those packages, which could be updated by upstream but not here. This should hopefully result in end users or someone else complaining that the package is missing within OpenWrt and that they can help us. I'm not saying that there are not enough maintainers. We (existing ones) are doing it for some purpose. We care about it. We are trying to update it, we want to have the latest versions on our routers, etc.

I might be strict, but I would simply remove those packages. You know, it is very pleasant of you that you will take into kismet, but how many users are using it?

I fail to see such unmaintained packages here: https://downloads.openwrt.org/stats/awstats.downloads.openwrt.org.allextra2.html

Is anyone really using them? If we are doing that for a minority <5-10 users then it is probably not worth all the efforts we are putting into it. They can compile themselves. They can use GitHub Actions (simply by forking this repository), if they are not familiar with the OpenWrt build system and from the CI, download the package and install it or even prepare their own feed.

With the removing these packages, we will as someone from OpenWrt core team is saying: "Less using compute resources, faster build time, happy users"

BKPepe commented 11 months ago

@BKPepe is it ok to include patch while we wait for a release? (for pcre2 support)

@Ansuel: You have there two packages, where you are listed as maintainer. We can probably backport uwsgi pending pull request as it seems that for 3 weeks, there has been no activity. Regarding nginx, it can be done immediately because PCRE2 support is there for many weeks.

Ansuel commented 11 months ago

Was talking about other packages since the deadline is 1 October. For nginx I will check today what are the required thing to do. The blocker was lua support

Il Ven 22 Set 2023, 09:45 Josef Schlehofer @.***> ha scritto:

@BKPepe https://github.com/BKPepe is it ok to include patch while we wait for a release? (for pcre2 support)

You have there two packages, where you are listed as maintainer. We can probably backport uwsgi pending pull request as it seems that for 3 weeks, there has been no activity. Regarding nginx, it can be done immediately because PCRE2 support is there for many weeks.

— Reply to this email directly, view it on GitHub https://github.com/openwrt/packages/issues/22006#issuecomment-1730959891, or unsubscribe https://github.com/notifications/unsubscribe-auth/AE2ZMQQ24PDSTHEQ6SCQD43X3U623ANCNFSM6AAAAAA4JHKF2M . You are receiving this because you were mentioned.Message ID: @.***>

BKPepe commented 11 months ago

Having a deadline is great, but I think we need to put these packages to some categories:

  1. OpenWrt packages maintainer is not active
  2. The package is not maintained anymore.
  3. Many people do not use the package, whereas 1st point applies.
  4. PCRE2 support is on its way and it would be too pity to remove it
  5. PCRE2 is already there, but 1st point apply.

So... it is not easy as it seems. :-)

I don't currently have the answer what are you looking for. It's open discussion, so any comment is appreciated. Time will tell us.

Ansuel commented 11 months ago

@BKPepe I like deadlines... force things to move and honestly we should add them all around our project. For me it's ok to apply patch and then drop them on release bump... (assuming they are mature enough and just stalled)

BKPepe commented 11 months ago

Yeah, I like deadlines as well that's why I gave it in the first place. Exactly for the reasons what are you saying. It moves things to move.

neheb commented 11 months ago

kismet can go away. I don’t use it.

padre-lacroix commented 11 months ago

I am the maintainer of kismet and I do not use it myself: as I mentioned in a previous message, I took it to prevent at the time (2015-2016) that it will disappear.

So, kismet can disappear as far as I am concerned: I cannot and will not maintain it as I cannot even test it or use it myself.

oskarirauta commented 11 months ago

@msva

Proposal for zsh libpcre2 usage: #22197

BKPepe commented 11 months ago

It seems that we do have a lot of progress here! Glad to see it, and thanks anyone, who looked into it.

trippleflux commented 11 months ago

@BKPepe

Trying my best about Aircrack-ng, hmm not to be sound rude but the bad news is I don't yet feel the like or knowledgeable enough in aircrack-ng to maintain or use it.

Ansuel commented 11 months ago

@BKPepe https://github.com/micropython/micropython-lib/pull/737 wasted way too much time on this but eheheheh

Ansuel commented 11 months ago

Well tomorrow is the deadline. I would drop pcre library and mark the remaining package as @BROKEN while we wait for the remaining package to be update. For the one where we don't have any info I would move them to package-abbandoned https://github.com/openwrt/packages-abandoned

stintel commented 11 months ago

Well tomorrow is the deadline. I would drop pcre library and mark the remaining package as @BROKEN while we wait for the remaining package to be update. For the one where we don't have any info I would move them to package-abbandoned https://github.com/openwrt/packages-abandoned

Please don't. You can't seriously consider dropping net-snmp from a distribution that mainly targets network devices.

Ansuel commented 11 months ago

@stintel if you want I can waste also today on migrating another package to pcre2.

stintel commented 11 months ago

@stintel if you want I can waste also today on migrating another package to pcre2.

The upstream issue hasn't seen any updates since January. Don't see any references to a branch where this might be worked on, and no idea how much work it's going to be. I currently don't have the time to dig in myself.

But dropping net-snmp would be a bad thing. I would actually consider finding an alternative to OpenWrt if that happens.

Ansuel commented 11 months ago

@stintel well this is main/master so it's expected to have some package broken for some time. These changes won't affect stable...

Aside from this... I'm checking the package with a quick grep and I can see there is actual support for pcre2... https://github.com/net-snmp/net-snmp/commit/d3e95c87b32397815f6d5bcfc844259f2552697a

Let me prepare a pr and see how it goes.

stintel commented 11 months ago

Aside from this... I'm checking the package with a quick grep and I can see there is actual support for pcre2... net-snmp/net-snmp@d3e95c8

Nice find! Tag me in a PR and I can make some time to do some testing. Thanks for looking at this!

BKPepe commented 11 months ago

As you can see, I put in a few extra days over the deadline. I'm just joking. I didn't look at it recently. I see that everyone did a great job over here. Mostly @Ansuel, who was working on it. Appreciated! 👏

Maybe some of you wouldn't like me, that's understandable, but most of those switches to PCRE2 were backported to the stable branches. You know, it is quite simple on this one. We would not get rid of the deprecated PCRE really fast, and it would be quite boring once the master is ready, but stable releases are not. This helps us that users can report it to us and we can really fast investigate it and fix it. OpenWrt does not release quite often as I was used to do with Turris OS (yep... they are still based on OpenWrt 21.02 so far with Turris OS 6.x. Thus some fixes landed there as well)

I am not using legacy PCRE on my router anymore, but some of you might. We are getting to the end. 🤞 There are just a few packages remaining to be moved/migrated to PCRE2.

I haven't heard anything about shadowsocks-libev, I will create the pull request to remove it from the master branch very soon.

Ansuel commented 11 months ago

@BKPepe

with shadowsocks-libev getting dropped i think pcre can be dropped.

Also consider that uwsgi is currently broken due to a problem with python and PATH but another member is working on fixing that so that should be handled indirectly. (also wasn't catch by CI test as they don't compile every sub package)

yousong commented 11 months ago

@BKPepe @Ansuel Though not a user of the package myself, I tend to believe that shadowsocks-libev seems to be still used by quite a few people (stats).

Before an alternative is available, I will try to keep it here by modifying the source code to drop dependency on pcre. Hopefully it's just acl support only recently enabled by @fertkir in #20647

Ansuel commented 11 months ago

@yousong or... https://github.com/openwrt/packages/pull/22356

BKPepe commented 11 months ago

Oh no, this did not make it to the final stable version of 23.05.0, never mind. So, it be 23.05.1.

Ansuel commented 11 months ago

@BKPepe mhhh in theory we can still continue to backport packages and continue this. Just people will update their packages

crza commented 10 months ago

Hi,

there still seem to be some packages with missing migration to pcre2, which makes my build for openwrt-23.05 branches fail:

stintel commented 10 months ago

In addition to the comment above, http://lists.openwrt.org/pipermail/openwrt-devel/2023-October/041689.html suggests some more packages were missed.

Ansuel commented 10 months ago

Oh no... More package to convert... Already converted postfix

Il Dom 29 Ott 2023, 02:17 Stijn Tintel @.***> ha scritto:

In addition to the comment above, http://lists.openwrt.org/pipermail/openwrt-devel/2023-October/041689.html suggests some more packages were missed.

  • fdm
  • haproxy
  • libndpi
  • privoxy
  • tvheadend
  • zabbix

— Reply to this email directly, view it on GitHub https://github.com/openwrt/packages/issues/22006#issuecomment-1783968869, or unsubscribe https://github.com/notifications/unsubscribe-auth/AE2ZMQSJCOPCLYTRTWTSN3TYBWOCJAVCNFSM6AAAAAA4JHKF2OVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTOOBTHE3DQOBWHE . You are receiving this because you were mentioned.Message ID: @.***>

crza commented 10 months ago

Oh no... More package to convert... Already converted postfix Il Dom 29 Ott 2023, 02:17 Stijn Tintel @.> ha scritto: … In addition to the comment above, http://lists.openwrt.org/pipermail/openwrt-devel/2023-October/041689.html suggests some more packages were missed. - fdm - haproxy - libndpi - privoxy - tvheadend - zabbix — Reply to this email directly, view it on GitHub <#22006 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AE2ZMQSJCOPCLYTRTWTSN3TYBWOCJAVCNFSM6AAAAAA4JHKF2OVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTOOBTHE3DQOBWHE . You are receiving this because you were mentioned.Message ID: @.>

Can someone backport to openwrt-23.05 after PR is pushed / merged?

crza commented 10 months ago

@stintel according to https://www.zabbix.com/documentation/current/en/manual/installation/requirements#:~:text=libpcre3%27%20or%20%27libpcre1%27.-,PCRE%20v8.x%20and%20PCRE2%20v10.x%20(from%20Zabbix%206.0.0)%20are%20supported.,-libevent zabbix 6.0+ already has pcre2 support, so maybe no patches are needed?

Ansuel commented 10 months ago

Yep I will check each missing package. I just need the list

BKPepe commented 10 months ago

Hmm, I think what I did wrong, uhh! Yep, it's just a few more packages. Guys, I am so sorry for this! I am feeling bad, because of that. I will add those packages in OP in a short manner.

Dne ne 29. 10. 2023 17:14 uživatel Christian Marangi < @.***> napsal:

Yep I will check each missing package. I just need the list

— Reply to this email directly, view it on GitHub https://github.com/openwrt/packages/issues/22006#issuecomment-1784155923, or unsubscribe https://github.com/notifications/unsubscribe-auth/AA7IDVF7HL52LCQS7N4J2XLYBZXFNAVCNFSM6AAAAAA4JHKF2OVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTOOBUGE2TKOJSGM . You are receiving this because you were mentioned.Message ID: @.***>

Ansuel commented 10 months ago

@BKPepe taking care of checking each suggested package and moving them... for now only 2 require special care (aka me having to convert the package I think...)

BKPepe commented 10 months ago

Can someone backport to openwrt-23.05 after PR is pushed / merged?

Hmm, I think this question is answered here in this thread already. ;) You might want to check also the stable branches. Of course, it will be backported over the time, no need to worry about it.

Ansuel commented 10 months ago

@BKPepe did you find other package that still use pcre?

systemcrash commented 10 months ago

freeswitch kamailio rtpengine sipgrep snort (not v3)

http://lists.openwrt.org/pipermail/openwrt-devel/2023-October/041689.html

BKPepe commented 10 months ago

Dont confuse it with other feeds. They are on their own. This is only for packages feed, where are the most packages. For other feeds something like this needs to be created or they will notice that once libpcre is gone.

Dne po 30. 10. 2023 23:35 uživatel Paul Dee @.***> napsal:

freeswitch kamailio rtpengine sipgrep snort (not v3)

http://lists.openwrt.org/pipermail/openwrt-devel/2023-October/041689.html

— Reply to this email directly, view it on GitHub https://github.com/openwrt/packages/issues/22006#issuecomment-1786152920, or unsubscribe https://github.com/notifications/unsubscribe-auth/AA7IDVCQJJ5VVYKT6T4FWD3YCATURAVCNFSM6AAAAAA4JHKF2OVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTOOBWGE2TEOJSGA . You are receiving this because you were mentioned.Message ID: @.***>

Ansuel commented 10 months ago

@BKPepe @systemcrash Is the list now complete? only 3 package to do... freeswitch is ""easy"" and I will manually convert snort and snort3 is HELL ON EARTH.

systemcrash commented 10 months ago

Those are the ones that pop up when I run make. Nothing further to add.

Ansuel commented 10 months ago

Welp only snort3 is missing... might take a while as I won't be at pc in the next few days :(

Ansuel commented 10 months ago

And with also snort3 manually converted, there isn't any more package that still require pcre.... Now it's really a matter of testing and merge the pending pr...

systemcrash commented 10 months ago

How is this migration coming along? Any objections to merging these?

Ansuel commented 10 months ago

@systemcrash nmap are backport for the other 3 package.. pr is open but no feedback from maintainers