search

ntop / ntopng

Web-based Traffic and Security Network Traffic Monitoring
http://www.ntop.org
GNU General Public License v3.0
6.27k stars 656 forks source link

[FreeBSD package PFSense] packagesite.pkg: Not Found #5814

Closed nobless closed 2 years ago

nobless commented 3 years ago

Hello, when installed in PFsense, and running pkg update it gets result:

Updating ntop repository catalogue... pkg: https://packages.ntop.org/FreeBSD/FreeBSD:12:amd64/latest/packagesite.pkg: Not Found ntop repository is up to date.

Why it is looking for pkg extension, seems there only txz extension package.

Installation procedure is as described in your manual:

[2.6.0-DEVELOPMENT][admin@fw.eu]/root: pkg add https://packages.ntop.org/FreeBSD/FreeBSD:12:amd64/latest/ntop-1.0.txz Fetching ntop-1.0.txz: 100% 872 B 0.9kB/s 00:01 Installing ntop-1.0... Extracting ntop-1.0: 100%

nobless commented 3 years ago

Still seeing upgradable packages: n2disk: 3.7.210821 -> 3.7.210827 [ntop] nprobe: 9.7.210821 -> 9.7.210827 [ntop] ntopng: 5.1.210821 -> 5.1.210827 [ntop]

nobless commented 3 years ago

Also it breaks system upgrades with such errors:

Updating repositories metadata... Updating ntop repository catalogue... Certificate verification failed for /CN=packages.ntop.org 34369335296:error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:/var/jenkins/workspace/pfSense-build-release-tarballs/BUILD_NODE/pkg-amd64-ce/OS_MAJOR_VERSION/freebsd12/PLATFORM/aws/crypto/openssl/ssl/statem/statem_clnt.c:1915:

lucaderi commented 3 years ago

We followed the instructions at https://packages.ntop.org/FreeBSD/ and using the txz package worked for us. I am unable to reproduce the certificare error. CAn you please report how I can obtain that?

simonemainardi commented 3 years ago

@nobless any news?

nobless commented 3 years ago

upgraded the system to the latest, can't reproduce the problem currently anymore.

but the problem is still left with package: pkg: https://packages.ntop.org/FreeBSD/FreeBSD:12:amd64/latest/packagesite.pkg: Not Found

maybe it freebsd related ? Why it searching for packagesite.pkg Beside this, upgrade works:

[2.6.0-DEVELOPMENT][admin@firewall]/root:  pkg upgrade
Updating ntop repository catalogue...
pkg: https://packages.ntop.org/FreeBSD/FreeBSD:12:amd64/latest/packagesite.pkg: Not Found
ntop repository is up to date.
Updating pfSense-core repository catalogue...
pfSense-core repository is up to date.
Updating pfSense repository catalogue...
pfSense repository is up to date.
All repositories are up to date.
Checking for upgrades (9 candidates): 100%
Processing candidates (9 candidates):  88%
pkg: nprobe has a missing dependency: librdkafka
Processing candidates (9 candidates): 100%
The following 9 package(s) will be affected (of 0 checked):

Installed packages to be UPGRADED:
        n2disk: 3.7.210905 -> 3.7.210908 [ntop]
        nprobe: 9.7.210905 -> 9.7.210908 [ntop]
        ntopng: 5.1.210905 -> 5.1.210908 [ntop]
        pfSense: 2.6.0.a.20210905.0500 -> 2.6.0.a.20210908.0500 [pfSense]
        pfSense-base: 2.6.0.a.20210905.0100 -> 2.6.0.a.20210908.0100 [pfSense-core]
        pfSense-default-config: 2.6.0.a.20210905.0100 -> 2.6.0.a.20210908.0100 [pfSense-core]
        pfSense-kernel-pfSense: 2.6.0.a.20210905.0100 -> 2.6.0.a.20210908.0100 [pfSense-core]
        pfSense-rc: 2.6.0.a.20210905.0100 -> 2.6.0.a.20210908.0100 [pfSense-core]
        pfSense-repo: 2.6.0.a.20210905.0500 -> 2.6.0.a.20210908.0500 [pfSense]

Number of packages to be upgraded: 9

The operation will free 2 MiB.
150 MiB to be downloaded.
ajgnet commented 3 years ago

Hello, reporting the same problem on PfSense 22.01

[22.01-DEVELOPMENT][admin@gw01]/root: pkg update
Updating ntop repository catalogue...
Fetching meta.conf: 100%    163 B   0.2kB/s    00:01
pkg: https://packages.ntop.org/FreeBSD/FreeBSD:12:amd64/latest/packagesite.pkg: Not Found
Fetching packagesite.txz: 100%    2 KiB   2.2kB/s    00:01
Processing entries: 100%
ntop repository update completed. 6 packages processed.
Updating pfSense-core repository catalogue...
pfSense-core repository is up to date.
Updating pfSense repository catalogue...
pfSense repository is up to date.
All repositories are up to date.

This causes the web updater to fail until /usr/local/etc/pkg/repos/ntop.conf is removed

Screen Shot 2021-09-16 at 2 18 24 PM

It appears "packagesite.pkg" does not exist in the pkg repo: https://packages.ntop.org/FreeBSD/FreeBSD:12:amd64/latest/

BSDer commented 3 years ago

Hi, chances are you are stuck somewhere. Try pkg delete ntop and pkg add https://packages.ntop.org/FreeBSD/FreeBSD:12:amd64/latest/ntop-1.0.txz. Should this not work, please report the content of /usr/local/etc/pkg/repos/ntop.conf and ls -l /usr/local/etc/pkg/fingerprints/ntop/trusted/

ajgnet commented 3 years ago

Hi - That did not resolve the error for me.


[21.09-RC][admin@gw01]/root: pkg delete ntop
Updating database digests format: 100%
Checking integrity... done (0 conflicting)
Deinstallation has been requested for the following 1 packages (of 0 packages in the universe):

Installed packages to be REMOVED:
    ntop: 1.0

Number of packages to be removed: 1

Proceed with deinstalling packages? [y/N]: y
[1/1] Deinstalling ntop-1.0...
[1/1] Deleting files for ntop-1.0: 100%

[21.09-RC][admin@]/root: pkg add https://packages.ntop.org/FreeBSD/FreeBSD:12:amd64/latest/ntop-1.0.txz
Fetching ntop-1.0.txz: 100%    872 B   0.9kB/s    00:01
Installing ntop-1.0...
Extracting ntop-1.0: 100%

[21.09-RC][admin@gw01]/root: pkg update
Updating ntop repository catalogue...
pkg: https://packages.ntop.org/FreeBSD/FreeBSD:12:amd64/latest/packagesite.pkg: Not Found
ntop repository is up to date.
Updating pfSense-core repository catalogue...
pfSense-core repository is up to date.
Updating pfSense repository catalogue...
pfSense repository is up to date.
All repositories are up to date.

[21.09-RC][admin@gw01]/root: cat /usr/local/etc/pkg/repos/ntop.conf
ntop: {
  fingerprints: "/usr/local/etc/pkg/fingerprints/ntop",
  url: https://packages.ntop.org/FreeBSD/${ABI}/latest,
  signature_type: "fingerprints",
  priority: 100,
  enabled: yes
}

[21.09-RC][admin@gw01]/root: ls -l /usr/local/etc/pkg/fingerprints/ntop/trusted/
total 5
-rw-r--r--  1 root  wheel  99 Sep 17 01:40 packages.ntop.org.20210108```
BSDer commented 3 years ago

Your installation is correct. Ok, so this is a problem within NTOP repo which is out of sync with pkg from the ports. As a workaround you can downgrade your pkg to a pre 1.17.0 version (e.g. from here https://pkg.net.isc.upenn.edu/FreeBSD%3A12%3Aamd64%3Alab/All/pkg-1.16.3.txz) or wait for NTOP repo to be updated (this shall happen soon, stay tuned).

nobless commented 3 years ago

Thanks, waiting for the update!

lucaderi commented 3 years ago

@BSDer Can you tell me exactly what we're supposed to do to fix this problem? We're not super experts with BSD packaging as far as I can see

BSDer commented 3 years ago

Hi Luca,

for each i in ${VERSION:ABI:WHATEVER} do;
  cd /usr/local/poudriere/data/packages/${i}
  ln -s .latest/packagesite.txz packagesite.pkg
done

You might have poudriere files in another place, change accordingly. Run a test if you can, or ping me and I will test it. Regards,

lucaderi commented 3 years ago

Hi @BSDer Can you please update and report if it is working now? I basically made a symbolic link from what I have understood from your message

nobless commented 3 years ago

Seems problem is fixed, no errors now:

Updating ntop repository catalogue...
Fetching meta.conf: 100%    163 B   0.2kB/s    00:01
Fetching packagesite.pkg: 100%    2 KiB   2.2kB/s    00:01
Processing entries: 100%
ntop repository update completed. 6 packages processed.
Updating pfSense-core repository catalogue...
Fetching meta.conf: 100%    163 B   0.2kB/s    00:01
Fetching packagesite.pkg: 100%    2 KiB   1.8kB/s    00:01
Processing entries: 100%
pfSense-core repository update completed. 7 packages processed.
Updating pfSense repository catalogue...
Fetching meta.conf: 100%    163 B   0.2kB/s    00:01
Fetching packagesite.pkg: 100%  145 KiB 148.6kB/s    00:01
Processing entries: 100%
pfSense repository update completed. 510 packages processed.
All repositories are up to date.
[2.6.0-DEVELOPMENT][admin@firewall]/root:  pkg upgrade
Updating ntop repository catalogue...
Fetching meta.conf: 100%    163 B   0.2kB/s    00:01
Fetching packagesite.pkg: 100%    2 KiB   2.2kB/s    00:01
Processing entries: 100%
ntop repository update completed. 6 packages processed.
Updating pfSense-core repository catalogue...
pfSense-core repository is up to date.
Updating pfSense repository catalogue...
pfSense repository is up to date.
All repositories are up to date.
Updating database digests format: 100%
Checking for upgrades (8 candidates): 100%
Processing candidates (8 candidates):  87%
pkg: nprobe has a missing dependency: librdkafka
Processing candidates (8 candidates): 100%
The following 8 package(s) will be affected (of 0 checked):

Installed packages to be UPGRADED:
        n2disk: 3.7.210908 -> 3.7.210919 [ntop]
        nprobe: 9.7.210908 -> 9.7.210919 [ntop]
        ntopng: 5.1.210908 -> 5.1.210919 [ntop]
        pfSense: 2.6.0.a.20210908.0500 -> 2.6.0.a.20210919.0500 [pfSense]
        pfSense-base: 2.6.0.a.20210908.0100 -> 2.6.0.a.20210919.0100 [pfSense-core]
        pfSense-default-config: 2.6.0.a.20210908.0100 -> 2.6.0.a.20210919.0100 [pfSense-core]
        pfSense-kernel-pfSense: 2.6.0.a.20210908.0100 -> 2.6.0.a.20210919.0100 [pfSense-core]
        pfSense-rc: 2.6.0.a.20210908.0100 -> 2.6.0.a.20210919.0100 [pfSense-core]

Number of packages to be upgraded: 8

The process will require 2 MiB more space.
152 MiB to be downloaded.

Also sugestions, to maintain this package correctly:

rename repo package to ntopng-repo for example, as ntop-1.0 is static naming without version increment. Increase version number to 1.1, as it now we need to remove it completly to install new version.

Thanks for the fix BSDer and lucaderi!

UPDATE: Seems not complete solution, posted below

nobless commented 3 years ago

Update: notice that PFSense update in Web GUI is still broken.

Without ntop-1.0 package: image

With ntop-1.0 package: image

lucaderi commented 3 years ago

We have done some tests and found no problems whatsoever.

From pfsense we have done

openssl s_client -connect packages.ntop.org:443 
CONNECTED(00000003)
depth=2 C = US, O = Internet Security Research Group, CN = ISRG Root X1
verify return:1
depth=1 C = US, O = Let's Encrypt, CN = R3
verify return:1
depth=0 CN = packages.ntop.org
verify return:1
---
Certificate chain
 0 s:CN = packages.ntop.org
   i:C = US, O = Let's Encrypt, CN = R3
 1 s:C = US, O = Let's Encrypt, CN = R3
   i:C = US, O = Internet Security Research Group, CN = ISRG Root X1
 2 s:C = US, O = Internet Security Research Group, CN = ISRG Root X1
   i:O = Digital Signature Trust Co., CN = DST Root CA X3
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIFKDCCBBCgAwIBAgISA4XAVtBerZH/lwYRR4/I5/QzMA0GCSqGSIb3DQEBCwUA
MDIxCzAJBgNVBAYTAlVTMRYwFAYDVQQKEw1MZXQncyBFbmNyeXB0MQswCQYDVQQD
EwJSMzAeFw0yMTA4MjAyMzU1NTFaFw0yMTExMTgyMzU1NTBaMBwxGjAYBgNVBAMT
EXBhY2thZ2VzLm50b3Aub3JnMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKC
AQEA2ZoUJQAtRsRDyUaGmrlE+FK+YMRX/ligS3OHLO9/Kr1pe1f/+IuiNdg8I5z9
T3sspgSm6g3PTffrMR/EtOc5Y4JBWX+S1dc+9FsLYU23bOTZz6NltUeY1c5lNuSZ
UTlVVrCd0mfHer0B5epqrW7HyVHPP0I6b6BMft7njuUp41AxqXZu/YUamjXRL5sS
DZiBY0jYcv96M9auBB02ywM/n6GKXOCR8X8RbC0BLsSoB4fYj2hVlZBxD+QbkWxL
NZiEl6zKXLRN0hyMUcA3Vrw+H7/VlEy7eRM7lJguLQegHPQt4X4qgxQqelkvk8ty
jwYxjcwi/y0M7/egXgOsds42nQIDAQABo4ICTDCCAkgwDgYDVR0PAQH/BAQDAgWg
MB0GA1UdJQQWMBQGCCsGAQUFBwMBBggrBgEFBQcDAjAMBgNVHRMBAf8EAjAAMB0G
A1UdDgQWBBSWyBBtZcwc7gbIiQBm32ehDKw08jAfBgNVHSMEGDAWgBQULrMXt1hW
y65QCUDmH6+dixTCxjBVBggrBgEFBQcBAQRJMEcwIQYIKwYBBQUHMAGGFWh0dHA6
Ly9yMy5vLmxlbmNyLm9yZzAiBggrBgEFBQcwAoYWaHR0cDovL3IzLmkubGVuY3Iu
b3JnLzAcBgNVHREEFTATghFwYWNrYWdlcy5udG9wLm9yZzBMBgNVHSAERTBDMAgG
BmeBDAECATA3BgsrBgEEAYLfEwEBATAoMCYGCCsGAQUFBwIBFhpodHRwOi8vY3Bz
LmxldHNlbmNyeXB0Lm9yZzCCAQQGCisGAQQB1nkCBAIEgfUEgfIA8AB2AJQgvB6O
1Y1siHMfgosiLA3R2k1ebE+UPWHbTi9YTaLCAAABe2Y08scAAAQDAEcwRQIhAInl
yP9D53rPlxVcYkFAfFJWHq73JqA8+oWiR0v6HsRLAiAgirb44+fAwbtP8U6UFpZ+
4e6EvEtLaBM2k6+9Mej6fAB2APZclC/RdzAiFFQYCDCUVo7jTRMZM7/fDC8gC8xO
8WTjAAABe2Y08rsAAAQDAEcwRQIgIPBTEPjU9q0VN5BqsoIt//SXk5A9oKUeR2cE
Zj/bX/UCIQDNO8caNHdRHwmFIlKC0FxEOTUdKuP3XQeSuJvrLs+4JTANBgkqhkiG
9w0BAQsFAAOCAQEAik1wISKcbyUKAbCibTLXK9a2LA1oK+eNyE7H82MEjXaXgiAM
MdKL3NYoNpii2sweMo/xQg4zYJocG6wH+ejmWijVy2d7Erbk8BO9sVmzqw26nA3z
2+ZLPawP+4Ly1awRHsRQ83PVHmJGzk1C7WMkWhT3DF9o8O08edEasQCPfR74WsUV
vJiM143ni6gry5ZG3reepNDErueMcSzraZA2e5Ee63rSPhqEq6zJ1sBkbIl5JvN3
Xm+NSUyjdyYkZ5wookh+aOmnAd7zx+6ICtNWL760tX0GUX+RQ0I+5GlAHB9oOEvm
9tYQS/ckhrkf9z2qe7FFsyaZhdyXtDLWEiVJCA==
-----END CERTIFICATE-----

so it looks good. On the remote end (packages.ntop.org) we also see no errors during the upgrade

x.x.x.x - - [21/Sep/2021:10:54:40 +0200] "GET /FreeBSD/FreeBSD:12:amd64/latest/ntop-1.0.txz HTTP/1.1" 200 6303 "-" "pkg/1.16.3"

From shell if we do

# pkg upgrade

Everything works as expected.

We have the feeling that the pfSense GUI has some issues with third party repositories.

@BSDer What do you think?

nobless commented 3 years ago

yeah cli works, GUI is not working after installing ntopng repository

BSDer commented 3 years ago

Following up on this after a number of attempts to make pfsense cope with ntop, we have found that pfsense is not keen on working with packages from 3rd parties, for example check out https://forum.netgate.com/topic/97731/freebsd-packages-on-2-3rc/28 where people end up asking pfsense developer to add this or that package to their repository.

Whilst we perceive pfsense behavior as a bug, they see it more like a feature: if it's not in our repository, you shall not use it in a firewall.

As a workaround users interested in using ntop with pfsense (who do this at their own risk and assume the consequences) can follow these steps:

  1. from the CLI: mv /usr/local/etc/pkg/repos/ntop.conf /usr/local/etc/pkg/repos/ntop.conf.off
  2. update pfsense via web interface
  3. from the CLI: mv /usr/local/etc/pkg/repos/ntop.conf.off /usr/local/etc/pkg/repos/ntop.conf

Should the situation improve, we will follow up here.

@nobless I do not think that ntop package requires to be updated to 1.1 as no changes have been make to the package itself: @lucaderi updated the package server to work with latest versions of pkg (post 1.17.x), in fact other users are now be able to update ntop descendants without reinstalling ntop itself.

lucaderi commented 2 years ago

It has been reported that the current package works on pfSense so I close this ticket for the time being