x/build/dashboard: update dragonfly to 6.4.0

tuxillo commented 10 months ago

It's long due but we should update the gobuilder from dfly 6.2.2 to 6.4.0

dmitshur commented 10 months ago

It seems the work of adding these builders to LUCI hasn't been done yet (#61092), so it might make sense to use 6.4.0 instead of 6.2.2 when doing that. (Since we're actively moving towards LUCI, there's less value in trying to do this update in the old infra at this point.)

@tuxillo, are you able to take a look at what adding this to LUCI would take? Also see #63471 and #60468.

CC @prattmic if you have a chance to look.

gopherbot commented 10 months ago

Change https://go.dev/cl/549035 mentions this issue: dashboard: update dragonfly to 6.4.0

tuxillo commented 10 months ago

@tuxillo, are you able to take a look at what adding this to LUCI would take? Also see #63471 and #60468.

After a quick look, I don't see any PR I could use to base the work on. Also, I'm not sure I can do this, there is step that says "Create the image and test on Luci" :)

tuxillo commented 7 months ago

has there been any progress? not sure what to do next.

prattmic commented 7 months ago

This will probably need to look something like https://go.dev/cl/526618, as that is another port that runs on GCE. We will also need to do some internal work to get LUCI to launch these VMs.

I can take a look at making adjustments to the image. Unfortunately, I tried to create the existing image, which failed in phase 2 trying to download packages:

+ pkg install -y bash curl git gdb
Feb 21 19:24:14  kernel: cd9660: RockRidge Extension
Updating Avalon repository catalogue...                              
Fetching meta.conf: 100%    163 B   0.2kB/s    00:01                                                                                         
Fetching packagesite.txz: 100%    6 MiB   1.1MB/s    00:06    
Processing entries: 100%
Avalon repository update completed. 31890 packages processed.
All repositories are up to date.    
New version of pkg detected; it needs to be installed first. 
The following 1 package(s) will be affected (of 0 checked):

Installed packages to be UPGRADED:
        pkg: 1.14.4 -> 1.20.9 [Avalon]

Number of packages to be upgraded: 1                                                                                                         

The process will require 2 MiB more space.                                                                                                   
3 MiB to be downloaded.                                                                                                                      
[1/1] Fetching pkg-1.20.9.pkg: 100%    3 MiB 697.4kB/s    00:05    
Checking integrity... done (0 conflicting)
[1/1] Upgrading pkg from 1.14.4 to 1.20.9...
[1/1] Extracting pkg-1.20.9: 100%               
pkg: Failed to execute lua script: [string "-- args: etc/pkg.conf.sample..."]:12: attempt to call a nil value (field 'stat')
pkg: lua script failed                   
pkg: No packages available to install matching 'bash' have been found in the repositories
pkg: No packages available to install matching 'git' have been found in the repositories
pkg: No packages available to install matching 'gdb' have been found in the repositories

Do you know what may be the problem here?

tuxillo commented 7 months ago

Do you know what may be the problem here?

Yeah, let me come up with a set of commands that can be actually put into a script. It's a problem with 6.4.0 alone that will be fixed in later releases (and it is fixed in latest).

prattmic commented 7 months ago

Ah, thanks for the quick response. I'll try 6.2.2 for now.

prattmic commented 7 months ago

6.2.2 gets further, but it is unfortunately broken as well:

+ pkg install -y bash curl git gdb
Feb 21 20:19:12  kernel: cd9660: RockRidge Extension
Updating Avalon repository catalogue...
Fetching meta.conf: 100%    163 B   0.2kB/s    00:01
Fetching packagesite.pkg: 100%    6 MiB   1.1MB/s    00:06
Processing entries: 100%
Avalon repository update completed. 31024 packages processed.
All repositories are up to date.
New version of pkg detected; it needs to be installed first.
The following 1 package(s) will be affected (of 0 checked):

Installed packages to be UPGRADED:
        pkg: 1.17.5_1 -> 1.18.4 [Avalon]

Number of packages to be upgraded: 1

3 MiB to be downloaded.
[1/1] Fetching pkg-1.18.4.pkg: 100%    3 MiB 811.3kB/s    00:04
Checking integrity... done (0 conflicting)
[1/1] Upgrading pkg from 1.17.5_1 to 1.18.4...
[1/1] Extracting pkg-1.18.4: 100%
Updating Avalon repository catalogue...
Avalon repository is up to date.
All repositories are up to date.
The following 35 package(s) will be affected (of 0 checked):

New packages to be INSTALLED:
        bash: 5.2_3 [Avalon]
        boost-libs: 1.80.0 [Avalon]
        gdb: 11.2 [Avalon]
        git: 2.38.1_1 [Avalon]
        gmp: 6.2.1 [Avalon]
        icu: 71.1,1 [Avalon]
        krb5: 1.20.1 [Avalon]
        libffi: 3.4.2 [Avalon]
        libiconv: 1.17 [Avalon]
        mpdecimal: 2.5.1 [Avalon]
        mpfr: 4.1.0_1 [Avalon]
        p5-Authen-SASL: 2.16_1 [Avalon]
        p5-CGI: 4.54 [Avalon]
        p5-Clone: 0.45 [Avalon]
        p5-Digest-HMAC: 1.04 [Avalon]
        p5-Encode-Locale: 1.05 [Avalon]
        p5-Error: 0.17029 [Avalon]
        p5-GSSAPI: 0.28_2 [Avalon]
        p5-HTML-Parser: 3.78 [Avalon]
        p5-HTML-Tagset: 3.20_1 [Avalon]
        p5-HTTP-Date: 6.05 [Avalon]
        p5-HTTP-Message: 6.37 [Avalon]
        p5-IO-HTML: 1.004 [Avalon]
        p5-IO-Socket-INET6: 2.72_1 [Avalon]
        p5-IO-Socket-SSL: 2.075 [Avalon]
        p5-LWP-MediaTypes: 6.04 [Avalon]
        p5-Mozilla-CA: 20211001 [Avalon]
        p5-Net-SSLeay: 1.92 [Avalon]
        p5-Socket6: 0.29 [Avalon]
        p5-TimeDate: 2.33,1 [Avalon]
        p5-URI: 5.12 [Avalon]
        perl5: 5.32.1_3 [Avalon]
        python39: 3.9.16 [Avalon]
        source-highlight: 3.1.9_3 [Avalon]

Installed packages to be UPGRADED:
        curl: 7.83.1 -> 7.86.0 [Avalon]

Number of packages to be installed: 34
Number of packages to be upgraded: 1

The process will require 497 MiB more space.
75 MiB to be downloaded.
[1/35] Fetching p5-Net-SSLeay-1.92.pkg: 100%  281 KiB 287.4kB/s    00:01
[2/35] Fetching p5-IO-Socket-SSL-2.075.pkg: 100%  175 KiB 179.5kB/s    00:01
[3/35] Fetching mpdecimal-2.5.1.pkg: 100%  341 KiB 349.1kB/s    00:01
[4/35] Fetching p5-URI-5.12.pkg: 100%   82 KiB  84.1kB/s    00:01
[5/35] Fetching boost-libs-1.80.0.pkg: 100%   13 MiB   1.6MB/s    00:09
[6/35] Fetching p5-CGI-4.54.pkg: 100%  155 KiB 158.6kB/s    00:01
[7/35] Fetching p5-TimeDate-2.33,1.pkg: 100%   33 KiB  33.5kB/s    00:01
[8/35] Fetching p5-Clone-0.45.pkg: 100%   10 KiB  10.5kB/s    00:01
[9/35] Fetching curl-7.86.0.pkg: 100%    1 MiB 501.2kB/s    00:03
[10/35] Fetching gmp-6.2.1.pkg: 100%  460 KiB 235.7kB/s    00:02
[11/35] Fetching mpfr-4.1.0_1.pkg: 100%  452 KiB 462.5kB/s    00:01
[12/35] Fetching libiconv-1.17.pkg: 100%  623 KiB 637.6kB/s    00:01
[13/35] Fetching p5-Error-0.17029.pkg: 100%   27 KiB  27.3kB/s    00:01
[14/35] Fetching p5-HTML-Tagset-3.20_1.pkg: 100%   12 KiB  12.0kB/s    00:01
[15/35] Fetching git-2.38.1_1.pkg: 100%    5 MiB 955.6kB/s    00:06
[16/35] Fetching p5-Authen-SASL-2.16_1.pkg: 100%   40 KiB  41.5kB/s    00:01
[17/35] Fetching krb5-1.20.1.pkg: 100%    1 MiB 394.1kB/s    00:03
[18/35] Fetching p5-Socket6-0.29.pkg: 100%   16 KiB  16.5kB/s    00:01
[19/35] Fetching p5-HTTP-Date-6.05.pkg: 100%   10 KiB  10.6kB/s    00:01
[20/35] Fetching p5-GSSAPI-0.28_2.pkg: 100%   39 KiB  40.3kB/s    00:01
[21/35] Fetching python39-3.9.16.pkg: 100%   17 MiB   1.7MB/s    00:11
[22/35] Fetching icu-71.1,1.pkg: 100%   11 MiB   1.4MB/s    00:08
[23/35] Fetching source-highlight-3.1.9_3.pkg: 100%    1 MiB 528.9kB/s    00:02
[24/35] Fetching p5-IO-Socket-INET6-2.72_1.pkg: 100%   13 KiB  13.1kB/s    00:01
[25/35] Fetching p5-Encode-Locale-1.05.pkg: 100%   12 KiB  11.9kB/s    00:01
[26/35] Fetching perl5-5.32.1_3.pkg: 100%   14 MiB   1.7MB/s    00:09
[27/35] Fetching p5-HTML-Parser-3.78.pkg: 100%   78 KiB  80.4kB/s    00:01
[28/35] Fetching p5-Mozilla-CA-20211001.pkg: 100%  118 KiB 121.2kB/s    00:01
[29/35] Fetching p5-LWP-MediaTypes-6.04.pkg: 100%   20 KiB  20.1kB/s    00:01
[30/35] Fetching libffi-3.4.2.pkg: 100%   31 KiB  31.9kB/s    00:01
[31/35] Fetching p5-IO-HTML-1.004.pkg: 100%   15 KiB  15.4kB/s    00:01
[32/35] Fetching p5-Digest-HMAC-1.04.pkg: 100%   14 KiB  14.1kB/s    00:01
[33/35] Fetching gdb-11.2.pkg: 100%    5 MiB 894.1kB/s    00:06
[34/35] Fetching bash-5.2_3.pkg: 100%    2 MiB 586.2kB/s    00:03
[35/35] Fetching p5-HTTP-Message-6.37.pkg: 100%   79 KiB  80.6kB/s    00:01
Checking integrity... done (1 conflicting)
  - git-2.38.1_1 [Avalon] conflicts with git-lite-2.35.2 [installed] on /usr/local/bin/git
Checking integrity... done (0 conflicting)
Conflicts with the existing packages have been found.
One more solver iteration is needed to resolve them.
The following 37 package(s) will be affected (of 0 checked):

New packages to be INSTALLED:
        bash: 5.2_3 [Avalon]
        boost-libs: 1.80.0 [Avalon]
        gdb: 11.2 [Avalon]
        git: 2.38.1_1 [Avalon]
        gmp: 6.2.1 [Avalon]
        icu: 71.1,1 [Avalon]
        krb5: 1.20.1 [Avalon]
        libffi: 3.4.2 [Avalon]
        libiconv: 1.17 [Avalon]
        mpdecimal: 2.5.1 [Avalon]
        mpfr: 4.1.0_1 [Avalon]
        p5-Authen-SASL: 2.16_1 [Avalon]
        p5-CGI: 4.54 [Avalon]
        p5-Clone: 0.45 [Avalon]
        p5-Digest-HMAC: 1.04 [Avalon]
        p5-Encode-Locale: 1.05 [Avalon]
        p5-Error: 0.17029 [Avalon]
        p5-GSSAPI: 0.28_2 [Avalon]
        p5-HTML-Parser: 3.78 [Avalon]
        p5-HTML-Tagset: 3.20_1 [Avalon]
        p5-HTTP-Date: 6.05 [Avalon]
        p5-HTTP-Message: 6.37 [Avalon]
        p5-IO-HTML: 1.004 [Avalon]
        p5-IO-Socket-INET6: 2.72_1 [Avalon]
        p5-IO-Socket-SSL: 2.075 [Avalon]
        p5-LWP-MediaTypes: 6.04 [Avalon]
        p5-Mozilla-CA: 20211001 [Avalon]
        p5-Net-SSLeay: 1.92 [Avalon]
        p5-Socket6: 0.29 [Avalon]
        p5-TimeDate: 2.33,1 [Avalon]
        p5-URI: 5.12 [Avalon]
        perl5: 5.32.1_3 [Avalon]
        python39: 3.9.16 [Avalon]
        source-highlight: 3.1.9_3 [Avalon]

Installed packages to be UPGRADED:
        curl: 7.83.1 -> 7.86.0 [Avalon]

Number of packages to be installed: 34
Number of packages to be upgraded: 1

The process will require 497 MiB more space.
Fetching git-lite-2.38.1_1.pkg: 100%    5 MiB 894.8kB/s    00:06
[1/37] Installing perl5-5.32.1_3...
[1/37] Extracting perl5-5.32.1_3: 100%
[2/37] Installing p5-TimeDate-2.33,1...
[2/37] Extracting p5-TimeDate-2.33,1: 100%
[3/37] Installing p5-URI-5.12...
[3/37] Extracting p5-URI-5.12: 100%
[4/37] Installing p5-Clone-0.45...
[4/37] Extracting p5-Clone-0.45: 100%
[5/37] Installing p5-HTTP-Date-6.05...
[5/37] Extracting p5-HTTP-Date-6.05: 100%
[6/37] Installing p5-Encode-Locale-1.05...
[6/37] Extracting p5-Encode-Locale-1.05: 100%
[7/37] Installing p5-LWP-MediaTypes-6.04...
[7/37] Extracting p5-LWP-MediaTypes-6.04: 100%
[8/37] Installing p5-IO-HTML-1.004...
[8/37] Extracting p5-IO-HTML-1.004: 100%
[9/37] Installing p5-HTML-Tagset-3.20_1...
[9/37] Extracting p5-HTML-Tagset-3.20_1: 100%
[10/37] Installing krb5-1.20.1...
[10/37] Extracting krb5-1.20.1: 100%
[11/37] Installing p5-Socket6-0.29...
[11/37] Extracting p5-Socket6-0.29: 100%
[12/37] Installing icu-71.1,1...
[12/37] Extracting icu-71.1,1: 100%
[13/37] Installing p5-HTTP-Message-6.37...
[13/37] Extracting p5-HTTP-Message-6.37: 100%
[14/37] Installing p5-Net-SSLeay-1.92...
[14/37] Extracting p5-Net-SSLeay-1.92: 100%
[15/37] Installing mpdecimal-2.5.1...
[15/37] Extracting mpdecimal-2.5.1: 100%
[16/37] Installing boost-libs-1.80.0...
[16/37] Extracting boost-libs-1.80.0: 100%
[17/37] Installing gmp-6.2.1...
[17/37] Extracting gmp-6.2.1: 100%
[18/37] Installing p5-GSSAPI-0.28_2...
[18/37] Extracting p5-GSSAPI-0.28_2: 100%
[19/37] Installing p5-IO-Socket-INET6-2.72_1...
[19/37] Extracting p5-IO-Socket-INET6-2.72_1: 100%
[20/37] Installing p5-HTML-Parser-3.78...
[20/37] Extracting p5-HTML-Parser-3.78: 100%
[21/37] Installing p5-Mozilla-CA-20211001...
[21/37] Extracting p5-Mozilla-CA-20211001: 100%
[22/37] Installing libffi-3.4.2...
[22/37] Extracting libffi-3.4.2: 100%
[23/37] Installing p5-Digest-HMAC-1.04...
[23/37] Extracting p5-Digest-HMAC-1.04: 100%
[24/37] Upgrading curl from 7.83.1 to 7.86.0...
[24/37] Extracting curl-7.86.0: 100%
[25/37] Deinstalling git-lite-2.35.2...
[25/37] Deleting files for git-lite-2.35.2: 100%
==> You should manually remove the "git_daemon" user.
==> You should manually remove the "git_daemon" group
[26/37] Installing p5-IO-Socket-SSL-2.075...
[26/37] Extracting p5-IO-Socket-SSL-2.075: 100%
[27/37] Installing p5-CGI-4.54...
[27/37] Extracting p5-CGI-4.54: 100%
[28/37] Installing mpfr-4.1.0_1...
[28/37] Extracting mpfr-4.1.0_1: 100%
[29/37] Installing libiconv-1.17...
[29/37] Extracting libiconv-1.17: 100%
[30/37] Installing p5-Error-0.17029...
[30/37] Extracting p5-Error-0.17029: 100%
[31/37] Installing p5-Authen-SASL-2.16_1...
[31/37] Extracting p5-Authen-SASL-2.16_1: 100%
[32/37] Installing python39-3.9.16...
[32/37] Extracting python39-3.9.16: 100%
[33/37] Installing source-highlight-3.1.9_3...
[33/37] Extracting source-highlight-3.1.9_3: 100%
[33/37] Installing git-lite-2.38.1_1...
===> Creating groups.
Using existing group 'git_daemon'.
===> Creating users
Using existing user 'git_daemon'.
[33/37] Extracting git-lite-2.38.1_1: 100%
[34/37] Installing git-2.38.1_1...
pkg: git-2.38.1_1 conflicts with git-lite-2.38.1_1 (installs files into the same place).  Problematic file: /usr/local/bin/git

tuxillo commented 7 months ago

Ah, thanks for the quick response. I'll try 6.2.2 for now.

For 6.4.0, can you please try the following? Make sure after the failure, you do the command below and then retry the pkg install ... command.

cp /usr/local/etc/pkg/repos/df-latest.conf.sample  /usr/local/etc/pkg/repos/df-latest.conf

Context: There was a bug that was triggered in the pkg 1.14 upgrade which ended up deleting the repos configuration.

prattmic commented 7 months ago

That did not work either, the second run failed with:

pkg: libressl-3.7.3_1 conflicts with openssl-1.1.1w,1 (installs files into the same place).  Problematic file: /usr/local/bin/openssl

You could be able to run the image creation process as well, by running make in env/dragonfly-amd64 (see README for dependencies). If you could get it working and send a CL, that would be ideal.

prattmic commented 7 months ago

After a quick look, I don't see any PR I could use to base the work on. Also, I'm not sure I can do this, there is step that says "Create the image and test on Luci" :)

To this original question, the best baseline for LUCI migration is the port of the openbsd-amd64 builder image in CL 526618, plus minor follow-ups in CL 528515 and CL 542064.

The key parts are:

Run x/build/cmd/bootstrapswarm as user swarming.
Pass the GCE hostname from the metadata server to bootstrapswarm --hostname.
Pass the swarming server from the metadata server to bootstrapswarm --swarming.
swarming should have passwordless sudo allowing reboot (edit: does dragonflybsd have sudo? If not, we'll need to figure out the right way to allow reboot).
The buildlet binary is no longer used.

You won't be able to fully test this until it is running on GCE with swarming access, but it should be testable that at least bootstrapswarm attempts to access the swarming server and gets an authentication error.

tuxillo commented 7 months ago

You could be able to run the image creation process as well, by running make in env/dragonfly-amd64 (see README for dependencies). If you could get it working and send a CL, that would be ideal.

Let me try myself and come back to you with something that actually works. Sorry for the hassle!

dmitshur commented 7 months ago

As a minor detail to append to https://github.com/golang/go/issues/64684#issuecomment-1959951186, the default value of the bootstrapswarm -swarming flag should already be the one you want to use, meaning you can leave that flag unset.

Edit: Whoops, what I said applies more in the case of reverse builders. In this case, using the value from the metadata server as suggested in that comment is indeed preferable.

tuxillo commented 7 months ago

@prattmic created a PR for the 6.4.0 build issue. I have tested it in Debian.

gopherbot commented 7 months ago

Change https://go.dev/cl/568315 mentions this issue: dashboard: fix env/dragonfly-amd64 build

Oguidan commented 3 months ago

Hi @tuxillo please can you explain what's the dragonfly

tuxillo commented 3 months ago

Hi @tuxillo please can you explain what's the dragonfly

It's a FreeBSD fork.

Oguidan commented 3 months ago

Hi @tuxillo please can you explain what's the dragonfly

It's a FreeBSD fork.

Ah ok, so if I understand this issue it's to make Go compatible with the Dragonfly 6.4.

golang / go

x/build/dashboard: update dragonfly to 6.4.0 #64684