search

FreshPorts / freshports

The website part of FreshPorts
http://www.freshports.org/
BSD 2-Clause "Simplified" License
68 stars 24 forks source link

Package message not presented for x11/nvidia-hybrid-graphics/files/pkg-message.in … #321

Closed grahamperrin closed 3 years ago

grahamperrin commented 3 years ago

https://www.freshports.org/x11/nvidia-hybrid-graphics/#message no message at the time of writing.

https://cgit.freebsd.org/ports/tree/x11/nvidia-hybrid-graphics/files/pkg-message.in exists …

245

dlangille commented 3 years ago 
% pwd
/var/db/freshports/ports-jail/var/db/repos/ports/x11/nvidia-hybrid-graphics
% make -V PKGMESSAGE
/var/db/freshports/ports-jail/var/db/repos/ports/x11/nvidia-hybrid-graphics/work-default/pkg-message
%

I think the issue is /work-default/ vs /work/

dlangille commented 3 years ago

Instead of make apply-slist, this approach is better:

[root@mydev:/usr/ports/x11/nvidia-hybrid-graphics] # make -V PKGMESSAGE
/var/ports/usr/ports/x11/nvidia-hybrid-graphics/work-default/pkg-message

But that file does not exist:

[root@mydev:/usr/ports/x11/nvidia-hybrid-graphics] # ls -l /var/ports/usr/ports/x11/nvidia-hybrid-graphics/work-default/pkg-message
ls: /var/ports/usr/ports/x11/nvidia-hybrid-graphics/work-default/pkg-message: No such file or directory

until after this:

[root@mydev:/usr/ports/x11/nvidia-hybrid-graphics] # make configure -DNO_DIALOG          
===> Skipping 'config' as NO_DIALOG is defined
===>  License BSD2CLAUSE accepted by the user
===>   nvidia-hybrid-graphics-0.5 depends on file: /usr/local/sbin/pkg - found
===> Fetching all distfiles required by nvidia-hybrid-graphics-0.5 for building
===>  Extracting for nvidia-hybrid-graphics-0.5
/bin/mkdir -p /var/ports/usr/ports/x11/nvidia-hybrid-graphics/work-default/.
/bin/mkdir -p /var/ports/usr/ports/x11/nvidia-hybrid-graphics/work-default/src/bin
/bin/mkdir -p /var/ports/usr/ports/x11/nvidia-hybrid-graphics/work-default/src/bin
/bin/mkdir -p /var/ports/usr/ports/x11/nvidia-hybrid-graphics/work-default/src/bin
/bin/mkdir -p /var/ports/usr/ports/x11/nvidia-hybrid-graphics/work-default/src/bin
/bin/mkdir -p /var/ports/usr/ports/x11/nvidia-hybrid-graphics/work-default/src/libexec/nvidia-headless-utils
/bin/mkdir -p /var/ports/usr/ports/x11/nvidia-hybrid-graphics/work-default/src/libexec
/bin/mkdir -p /var/ports/usr/ports/x11/nvidia-hybrid-graphics/work-default/src/etc/X11
/bin/mkdir -p /var/ports/usr/ports/x11/nvidia-hybrid-graphics/work-default/src/etc
/bin/mkdir -p /var/ports/usr/ports/x11/nvidia-hybrid-graphics/work-default/src/etc
===>  Patching for nvidia-hybrid-graphics-0.5
===>  Configuring for nvidia-hybrid-graphics-0.5

Then, it will be in this directory:

[root@mydev:/usr/ports/x11/nvidia-hybrid-graphics] # make -V WRKDIR
/var/ports/usr/ports/x11/nvidia-hybrid-graphics/work-default

What do we find in this:

[root@mydev:/usr/ports/x11/nvidia-hybrid-graphics] # ls -l /var/ports/usr/ports/x11/nvidia-hybrid-graphics/work-default/           
total 29
-rw-r--r--  1 root  wheel     0 Jul 27 17:43 .configure_done.nvidia-hybrid-graphics._usr_local
-rw-r--r--  1 root  wheel     0 Jul 27 17:43 .extract_done.nvidia-hybrid-graphics._usr_local
-rw-r--r--  1 root  wheel   188 Jul 27 17:43 .license-catalog.mk
-rw-r--r--  1 root  wheel    81 Jul 27 17:43 .license-report
-rw-r--r--  1 root  wheel     0 Jul 27 17:43 .license_done.nvidia-hybrid-graphics._usr_local
-rw-r--r--  1 root  wheel     0 Jul 27 17:43 .patch_done.nvidia-hybrid-graphics._usr_local
-rw-r--r--  1 root  wheel    97 Jul 27 17:43 BSD2CLAUSE
-rw-r--r--  1 root  wheel   808 Jul 27 17:43 nvidia_xorg
-rw-r--r--  1 root  wheel  1000 Jul 27 17:43 pkg-message
drwxr-xr-x  5 root  wheel     5 Jul 27 17:43 src

The file we want:

[root@mydev:/usr/ports/x11/nvidia-hybrid-graphics] # cat /var/ports/usr/ports/x11/nvidia-hybrid-graphics/work-default/pkg-message 
Running Xorg-nvidia-headless at startup is recommended especially on mobile
systems since leaving Nvidia GPU powered on with no Xorg may consume
significantly more power.

# sysrc nvidia_xorg_enable=YES
# service nvidia_xorg start

The Xorg instance uses separate configuration files from the defaults for Xorg:

    /usr/local/etc/X11/xorg-nvidia-headless.conf.d/

    /usr/local/etc/X11/xorg-nvidia-headless-template.conf
        Only used if xorg-nvidia-headless.conf does not exist.
    This file is passed through nvidia-xconfig, which does not preserve
    all options.

    /usr/local/etc/X11/xorg-nvidia-headless.conf
        Created by manually invoking /usr/local/bin/nvidia-headless-xconfig.
        Once created, this configuration is not passed through nvidia-xconfig.

Configuration is not normally needed for typical OpenGL acceleration usage.

A properly configured xorg-nvidia-headless.conf will allow Xorg-nvidia-headless
to be started by non-superuser if Xorg has been installed setuid.
[root@mydev:/usr/ports/x11/nvidia-hybrid-graphics] #
dlangille commented 3 years ago

Some history. The following represents decisions made when subversion was in use.

Why /var/db/freshports/ports-jail?

Some things are better done in a chroot.

Why /var/db/repos/ports within that chroot?

/usr/ports is traditional. However, when you have branches to deal with, where do you put that checked out copy of the latest quarterly branch? It can't be /usr/ports because that would conflict with the check out of head.

/var/db/repos was chosen.

That directory contained PORT-head, PORTS-2020Q4, etc.

Now, with git in use, only /var/db/repos/ports is used.

Now, I think we can move the repo back to /usr/ports.

Where we once used

/var/db/freshports/ports-jail/var/db/repos/ports

Now we can start using

/var/db/freshports/ports-jail/usr/ports - but that change is out of scope for this issue.

dlangille commented 3 years ago

https://dev.freshports.org/x11/nvidia-hybrid-graphics/#message seems fine now.

This will require running a fix-up script on all hosts. This will recollect all pkg-message data

grahamperrin commented 3 years ago

Thanks.

Nit: no anchor

image

dlangille commented 3 years ago

Except that this does not work in the chroot:

# make configure -DNO_DIALOG PORTSDIR=/var/db/repos/ports OPTIONSFILE=/nonexistentlocal -f /var/db/repos/ports/x11/nvidia-hybrid-graphics/Makefile
===> Skipping 'config' as NO_DIALOG is defined
===>  License BSD2CLAUSE accepted by the user
===>   nvidia-hybrid-graphics-0.5 depends on file: /usr/local/sbin/pkg - not found
===> Skipping 'config' as NO_DIALOG is defined
===>  Configuring for pkg-1.17.1
No installed jimsh or tclsh, building local bootstrap jimsh0
./autosetup/autosetup-find-tclsh: ./jimsh0: not found
No working C compiler found. Tried echo and gcc.
exec: -o jimsh0 ./autosetup/jimsh0.c
false: not found
===>  Script "configure" failed unexpectedly.
Please report the problem to pkg@FreeBSD.org [maintainer] and attach the
"/var/db/repos/ports/ports-mgmt/pkg/work/pkg-1.17.1/config.log" including the
output of the failure of your make command. Also, it might be a good idea to
provide an overview of all packages installed on your system (e.g. a
/var/db/repos/ports/ports-mgmt/pkg/work/pkg-1.17.1/src/pkg-static info -g
-Ea).
*** Error code 1

Stop.
make[4]: stopped in /var/db/repos/ports/ports-mgmt/pkg
*** Error code 1

Stop.
make[3]: stopped in /var/db/repos/ports/ports-mgmt/pkg
*** Error code 1

Stop.
make[2]: stopped in /var/db/repos/ports/ports-mgmt/pkg
*** Error code 1

Stop.
make[1]: stopped in /var/db/repos/ports/x11/nvidia-hybrid-graphics
*** Error code 1

Stop.
make: stopped in /var/db/repos/ports/x11/nvidia-hybrid-graphics
#
dlangille commented 3 years ago

Thanks.

Nit: no anchor

That should be fixed now.

dlangille commented 3 years ago

Except that this does not work in the chroot:

Everything done in the chroot could be done in a jail. A jail within a jail.

dlangille commented 3 years ago

that make just work..... I tested it. The only package required was pkg.

dlangille commented 3 years ago

Now that https://dev.freshports.org/ is using a FreeBSD jail, not a chroot, let's come back to this problem.

Work so far: https://gist.github.com/dlangille/3f919e36ec4f9af6ead2eaddeafd2869

dlangille commented 3 years ago

Success:

root@freshports:/usr/ports/x11/nvidia-hybrid-graphics # make -V PKGMESSAGE
/usr/ports/x11/nvidia-hybrid-graphics/work-default/pkg-message
root@freshports:/usr/ports/x11/nvidia-hybrid-graphics # ls -l /usr/ports/x11/nvidia-hybrid-graphics/work-default/pkg-message
ls: /usr/ports/x11/nvidia-hybrid-graphics/work-default/pkg-message: No such file or directory
root@freshports:/usr/ports/x11/nvidia-hybrid-graphics # make configure -DNO_DIALOG
===> Skipping 'config' as NO_DIALOG is defined
===>  License BSD2CLAUSE accepted by the user
===>   nvidia-hybrid-graphics-0.5 depends on file: /usr/local/sbin/pkg - found
===> Fetching all distfiles required by nvidia-hybrid-graphics-0.5 for building
===>  Extracting for nvidia-hybrid-graphics-0.5
/bin/mkdir -p /usr/ports/x11/nvidia-hybrid-graphics/work-default/.
/bin/mkdir -p /usr/ports/x11/nvidia-hybrid-graphics/work-default/src/bin
/bin/mkdir -p /usr/ports/x11/nvidia-hybrid-graphics/work-default/src/bin
/bin/mkdir -p /usr/ports/x11/nvidia-hybrid-graphics/work-default/src/bin
/bin/mkdir -p /usr/ports/x11/nvidia-hybrid-graphics/work-default/src/bin
/bin/mkdir -p /usr/ports/x11/nvidia-hybrid-graphics/work-default/src/libexec/nvidia-headless-utils
/bin/mkdir -p /usr/ports/x11/nvidia-hybrid-graphics/work-default/src/libexec
/bin/mkdir -p /usr/ports/x11/nvidia-hybrid-graphics/work-default/src/etc/X11
/bin/mkdir -p /usr/ports/x11/nvidia-hybrid-graphics/work-default/src/etc
/bin/mkdir -p /usr/ports/x11/nvidia-hybrid-graphics/work-default/src/etc
===>  Patching for nvidia-hybrid-graphics-0.5
===>  Configuring for nvidia-hybrid-graphics-0.5
root@freshports:/usr/ports/x11/nvidia-hybrid-graphics # ls -l /usr/ports/x11/nvidia-hybrid-graphics/work-default/pkg-message
-rw-r--r--  1 root  wheel  1000 Aug  9 22:46 /usr/ports/x11/nvidia-hybrid-graphics/work-default/pkg-message
root@freshports:/usr/ports/x11/nvidia-hybrid-graphics #
dlangille commented 3 years ago 
root@freshports:/usr/ports/x11/nvidia-hybrid-graphics # cat /usr/ports/x11/nvidia-hybrid-graphics/work-default/pkg-message
Running Xorg-nvidia-headless at startup is recommended especially on mobile
systems since leaving Nvidia GPU powered on with no Xorg may consume
significantly more power.

# sysrc nvidia_xorg_enable=YES
# service nvidia_xorg start

The Xorg instance uses separate configuration files from the defaults for Xorg:

    /usr/local/etc/X11/xorg-nvidia-headless.conf.d/

    /usr/local/etc/X11/xorg-nvidia-headless-template.conf
        Only used if xorg-nvidia-headless.conf does not exist.
    This file is passed through nvidia-xconfig, which does not preserve
    all options.

    /usr/local/etc/X11/xorg-nvidia-headless.conf
        Created by manually invoking /usr/local/bin/nvidia-headless-xconfig.
        Once created, this configuration is not passed through nvidia-xconfig.

Configuration is not normally needed for typical OpenGL acceleration usage.

A properly configured xorg-nvidia-headless.conf will allow Xorg-nvidia-headless
to be started by non-superuser if Xorg has been installed setuid.
root@freshports:/usr/ports/x11/nvidia-hybrid-graphics #
dlangille commented 3 years ago

However, this winds up fetching stuff. That's not a bad thing. It increases processing time. The host will need more disk space.

$ ps auwwx 
USER         PID %CPU %MEM   VSZ   RSS TT  STAT STARTED    TIME COMMAND
root       47230  3.2  0.0 16240  6344  -  RJ   23:03   0:00.28 /usr/bin/fetch -Fpr -S 36474676 https://jp.download.nvidia.com/XFree86/FreeBSD-x86_64/304.137/NVIDIA-FreeBSD-x86_64-304.137.tar.gz
root       47172  0.7  0.0 10356  1664  -  SJ   23:03   0:00.03 /usr/bin/make CONFIG_DONE_NVIDIA-DRIVER-304=1 /usr/ports/x11/nvidia-driver-304/work/.configure_done.nvidia-driver._usr_local
root       47205  0.7  0.0 11960  3020  -  SJ   23:03   0:00.01 /bin/sh /usr/ports/Mk/Scripts/do-fetch.sh NVIDIA-FreeBSD-x86_64-304.137.tar.gz
root       47153  0.3  0.0 10368  1648  -  SJ   23:03   0:00.03 /usr/bin/make configure -DNO_DIALOG PORTSDIR=/usr/ports OPTIONSFILE=/nonexistentlocal -f /usr/ports/x11/nvidia-driver-304/Makefile
root       47151  0.2  0.0 17344  7852  -  SJ   23:03   0:00.01 /usr/local/bin/sudo /usr/sbin/jexec freshports /make-pkg-message-in.sh /usr/ports x11/nvidia-driver-304 /usr/ports/x11/nvidia-driver-304/work/pkg-message
root       47152  0.2  0.0 11544  2920  -  SJ   23:03   0:00.00 /bin/sh /make-pkg-message-in.sh /usr/ports x11/nvidia-driver-304 /usr/ports/x11/nvidia-driver-304/work/pkg-message
freshports  5136  0.1  0.0 51204 40616  -  SJ   23:01   0:00.78 /usr/local/bin/perl /usr/local/libexec/freshports/load_xml_into_db.pl /var/db/ingress/message-queues/incoming/2021.06.15.19.30.22.000000.2ee6acf85a98de36269d3a727b4c45675b0eb9c3.xml
dlangille commented 3 years ago

@grahamperrin https://dev.freshports.org/x11/nvidia-hybrid-graphics/#message looks OK to me.

dlangille commented 3 years ago

Fixed the fetching by going back to make apply-list.

Details at https://lists.freebsd.org/archives/freebsd-ports/2021-August/000544.html

dlangille commented 3 years ago

This won't hit production quickly. Perhaps over the weekend.

dlangille commented 3 years ago

We should refresh all ports to get the proper pkg-message.

See ports_to_refresh in #328

grahamperrin commented 3 years ago

Thank you,

We should refresh all ports to get the proper pkg-message. …

It's unusual to not find the message at https://www.freshports.org/www/firefox/#message, will this reappear after the refresh?

dlangille commented 3 years ago

dev says no, but we shall see. I'll be working on the pkg-message refresh SQL soon. It was blocked by recently closed issues.

dlangille commented 3 years ago

This is interesting. There is a valid file:

[dan@dev-ingress01:~] $ time sudo jexec freshports 

root@freshports:/ # cd /usr/ports/www/firefox

root@freshports:/usr/ports/www/firefox # make -V PKGMESSAGE
/usr/ports/www/firefox/pkg-message
root@freshports:/usr/ports/www/firefox # ls -l /usr/ports/www/firefox/pkg-message
-rw-r--r--  1 root  wheel  1727 Aug  8 17:14 /usr/ports/www/firefox/pkg-message
root@freshports:/usr/ports/www/firefox # cat /usr/ports/www/firefox/pkg-message
[
{ type: install
  message: <<EOM
## Missing features

Some features found on Windows, macOS and Linux are not implemented:

- Encrypted Media Extensions (requires Widevine CDM binary)
- Process sandboxing (requires Capsicum backend)
- Reduced memory usage (requires mozjemalloc)
- Crash Reporter (requires Google Breakpad and reproducible builds)
- WebVR (requires open source runtime)
- TCP fast open
- `about:networking` (requires link state notification)

## Audio backend

Currently used audio backend can be inspected on `about:support` page.
Supported backends and default probing order is as follows:
- `pulse-rust` if `pulseaudio` package is installed (PULSEAUDIO option)
- `jack` if `jackit` package is installed (JACK option)
- `sndio` if `sndio` package is installed (SNDIO option)
- `alsa` if `alsa-lib` package is installed (ALSA option)
- `oss` (always available)
To force a specific backend open `about:config` page and create
`media.cubeb.backend` preference.

Microphone selection only works in `oss`, `pulse`, `pulse-rust` backends.
Other backends are limited to `default` which is usually `/dev/dsp`,
so use virtual_oss to reroute microphones from non-default devices.

## Gamepad API

Requires evdev(4) joystick support. On FreeBSD 13.0 enable hgame(4)
while older versions can use sysutils/iichid or multimedia/webcamd.

## smb:// issues
Network group, machine, and share browsing does not work correctly.

## sftp://
Only sftp access using public key authentication works. To easily
setup public key authentication to `remote_host`:

    $ ssh-keygen
    $ cat ~/.ssh/id_rsa.pub | ssh remote_host "cat >> .ssh/authorized_keys"

The SSH server on `remote_host` must allow pub key authentication.
EOM
}
]
root@freshports:/usr/ports/www/firefox #
dlangille commented 3 years ago

Fixed for https://dev.freshports.org/www/firefox/#message

The code was still prefixing:

[dan@dev-ingress01:~/scripts] $ svn di ~/modules/port.pm
Index: /usr/home/dan/modules/port.pm
===================================================================
--- /usr/home/dan/modules/port.pm   (revision 5778)
+++ /usr/home/dan/modules/port.pm   (working copy)
@@ -830,7 +830,7 @@
        print "\$pkgmessagepath='$pkgmessagepath'\n";
        print "\$FreshPorts::Config::JailBaseDir . \$pkgmessagepath='$FreshPorts::Config::JailBaseDir$pkgmessagepath'\n";
        # $pkgmessagepath is relative to the jail directory, so we need to prefix that here.
-       my $RealPKGMESSAGEPath = $this->_GetRealPath($FreshPorts::Config::JailBaseDir . $pkgmessagepath);
+       my $RealPKGMESSAGEPath = $this->_GetRealPath($pkgmessagepath);
        print "\$RealPKGMESSAGEPath='$RealPKGMESSAGEPath'\n";

        # if it's defined, and it exists....
[dan@dev-ingress01:~/scripts] $

That is not needed now with jexec and a real jail with a ports tree at /usr/ports/

dlangille commented 3 years ago

Problem with refresh script. It's expecting utf-8, but there's a better way:

Aug 29 19:54:36 dev-ingress01 FreshPorts[85438]: refresh-ports.pl found audio/etktab 
Aug 29 19:54:36 dev-ingress01 FreshPorts[85438]: Could not execute SQL     select ports.*,           categories.name as category,            element.name    as name,            element_pathname(element.id, FALSE) as element_pathname      from ports, categories, element      where ports.id          = 10292        and ports.category_id = categories.id        and ports.element_id  = element.id (/usr/local/libexec/freshports)
Aug 29 19:54:36 dev-ingress01 FreshPorts[85438]: from='FreshPorts Daemon <FreshPorts@FreshPorts.org>' to='dan@langille.org' subject='FreshPorts error on dev-ingress01.int.unixathome.org' (/usr/local/libexec/freshports) 
Aug 29 19:54:37 dev-ingress01 /usr/local/libexec/freshports/refresh-ports.sh[67319]: finished

Running that manually:

freshports.devgit=# select ports.*,           categories.name as category,            element.name    as name,            element_pathname(element.id, FALSE) as element_pathname      from ports, categories, element      where ports.id          = 10292        and ports.category_id = categories.id        and ports.element_id  = element.id;
ERROR:  invalid byte sequence for encoding "UTF8": 0xe4 0x20 0x3c
freshports.devgit=#
dlangille commented 3 years ago

Fixed that with: 

-$sql = "select PTR.port_id, categories.name as category, element.name as port
+$sql = "SET CLIENT_ENCODING TO 'ISO-8859-1'; select PTR.port_id, categories.name as category, element.name as port

It takes about 24 hours to update all the ports (about 30,000) of them. I might look at running several instances of this script and get that down to a few hours.

freshports.devgit=# begin;insert into ports_to_refresh(port_id) select id from ports_active where element_pathname(element_id) not like '/ports/branches/%';
BEGIN
INSERT 0 29718
freshports.devgit=#
dlangille commented 3 years ago

After several tests on dev, now running this on test:

https://twitter.com/FreshPorts/status/1433430944258531336

dlangille commented 3 years ago

Looks good on test:

dlangille commented 3 years ago

Another few runs today. Good. Next: staging.

dlangille commented 3 years ago

Looked good on stage yesteday:

dlangille commented 3 years ago

Fixed in prod.

grahamperrin commented 3 years ago

Thank you!