bhearsum
commented
6 months ago I'm asking around other projects to see if they're seeing this as well.
Open eu9ene opened 6 months ago
bhearsum
commented
6 months ago I'm asking around other projects to see if they're seeing this as well.
bhearsum
commented
6 months ago Haven't seen reports of this elsewhere.
@eu9ene - have you seen this on GPU workers only? Or also on the CPU workers?
eu9ene
commented
6 months ago Haven't seen reports of this elsewhere.
@eu9ene - have you seen this on GPU workers only? Or also on the CPU workers?
I'm not sure but I feel like I've been seeing this in random places.
gregtatum
commented
6 months ago Here's an instance on a GPU worker: https://firefox-ci-tc.services.mozilla.com/tasks/XRYrb2BxTOyJa209jrLQ7A/runs/0/logs/public/logs/live.log
bhearsum
commented
6 months ago Thanks; so it seems very unlikely to be related to specific worker images.
@aerickson - I don't suppose you have any idea what's going on here?
aerickson
commented
6 months ago @bhearsum I'm not sure what's going on. Translations GPU workers on GCP should use GCP's DNS servers (provided by DHCP) and the Snakepit worker use our internal infoblox servers (configured in dnsmasq). It seems like a network blip or perhaps the DNS server could be overloaded for a second?
I haven't heard about any Github outages around DNS. I've never really heard of DNS outages (it's a pretty resilient service/protocol).
If we find a concentrated event or location let me know and I'll dig in some more.
bhearsum
commented
6 months ago We do see these fairly often - I would say maybe on 5-10% of the tasks run. I'll try to collect some data to help us analyze this better.
bhearsum
commented
6 months ago Here's failures by worker group:
defaultdict(<class 'int'>,
{'us-central1': 7,
'us-central1-a': 9,
'us-central1-b': 7,
'us-central1-c': 7,
'us-central1-f': 9,
'us-west1': 5,
'us-west1-a': 5,
'us-west1-b': 11})And here's timestamps when we hit the failures:
['2024-01-19T21:20:34.616Z',
'2024-02-02T20:02:49.997Z',
'2024-02-22T16:07:57.737Z',
'2024-02-26T23:13:59.353Z',
'2024-02-28T00:36:35.015Z',
'2024-02-28T17:48:06.906Z',
'2024-02-29T15:03:54.666Z',
'2024-03-06T14:32:56.066Z',
'2024-03-06T18:10:27.186Z',
'2024-03-21T15:03:54.430Z',
'2024-03-21T20:29:42.584Z',
'2024-03-26T13:29:20.434Z',
'2024-03-30T22:11:50.010Z',
'2024-04-01T14:10:30.644Z',
'2024-04-01T15:18:22.619Z',
'2024-04-09T10:47:13.166Z',
'2024-04-18T12:41:01.952Z',
'2024-04-22T19:58:27.752Z',
'2024-04-23T19:46:39.366Z',
'2024-04-24T17:57:31.559Z',
'2024-04-25T07:33:55.024Z',
'2024-04-29T17:55:15.937Z',
'2024-04-30T00:38:45.847Z',
'2024-04-30T01:08:46.769Z',
'2024-04-30T14:06:18.417Z',
'2024-04-30T18:56:31.177Z',
'2024-05-01T23:21:11.540Z',
'2024-05-02T00:22:43.542Z',
'2024-05-06T14:40:00.719Z',
'2024-05-06T16:41:19.051Z',
'2024-05-07T00:01:30.432Z',
'2024-05-07T21:23:47.145Z',
'2024-05-07T21:36:54.706Z',
'2024-05-09T19:01:17.356Z',
'2024-05-09T22:10:58.358Z',
'2024-05-09T22:11:47.850Z',
'2024-05-10T18:42:00.163Z',
'2024-05-10T22:14:23.709Z',
'2024-05-13T23:01:23.656Z',
'2024-05-13T23:24:46.266Z',
'2024-05-14T14:49:27.875Z',
'2024-05-14T15:08:43.843Z',
'2024-05-14T23:18:39.852Z',
'2024-05-14T23:43:02.154Z',
'2024-05-15T01:01:44.678Z',
'2024-05-15T14:30:49.732Z',
'2024-05-15T23:20:08.322Z',
'2024-05-16T21:56:58.285Z',
'2024-05-17T11:38:30.465Z',
'2024-05-17T14:02:32.250Z',
'2024-05-17T16:18:16.690Z',
'2024-05-17T17:51:07.523Z',
'2024-05-17T20:37:44.360Z',
'2024-05-17T20:49:01.813Z',
'2024-05-17T20:49:18.471Z',
'2024-05-17T21:36:37.947Z',
'2024-05-17T22:05:41.152Z',
'2024-05-17T22:06:42.275Z',
'2024-05-17T23:11:32.583Z',
'2024-05-20T15:29:29.312Z']And by worker image:
defaultdict(<class 'int'>, {'gpu': 58, 'cpu': 2})Clearly the most notable part here here is that we're seeing more issues on GPU images. And within that, we seem to have gotten more beginning in late April/early May. We added a test pool with a new image on April 22nd (that I was running a lot to test things) in https://phabricator.services.mozilla.com/D208202. That image went to production on May 8th in https://phabricator.services.mozilla.com/D209840.
https://github.com/mozilla-platform-ops/monopacker/pull/140 was the PR related to this image, but I don't know how deterministic the other parts are? Eg: could we have picked up a change to a system package that is now causing problems?
@aerickson - do you have any thoughts? If we still have the old image, maybe we could poke around and compare the new to old one? (I'd be happy to do this if you want.)
eu9ene
commented
6 months ago Ok, it happens every single pipeline run for me now and does not restart. Marking as blocker.
bhearsum
commented
6 months ago Poking at this a bit on an interactive instance, too. I sortof repro'ed (it seems to have retried with success though):
ubuntu@translations-1-b-linux-v100-gpu-ecnlewoqrtmrltrxrl8o3w:~/tasks/task_171642353933996$ host github.com
;; communications error to 127.0.0.53#53: timed out
github.com has address 140.82.116.3
github.com mail is handled by 5 alt2.aspmx.l.google.com.
github.com mail is handled by 1 aspmx.l.google.com.
github.com mail is handled by 5 alt1.aspmx.l.google.com.
github.com mail is handled by 10 alt3.aspmx.l.google.com.
github.com mail is handled by 10 alt4.aspmx.l.google.com.Looking at the machine configuration, I see that it uses the standard resolved that we expect on Ubuntu:
ubuntu@translations-1-b-linux-v100-gpu-ecnlewoqrtmrltrxrl8o3w:~/tasks/task_171642353933996$ cat /etc/resolv.conf
# This is /run/systemd/resolve/stub-resolv.conf managed by man:systemd-resolved(8).
# Do not edit.
#
# This file might be symlinked as /etc/resolv.conf. If you're looking at
# /etc/resolv.conf and seeing this text, you have followed the symlink.
#
# This is a dynamic resolv.conf file for connecting local clients to the
# internal DNS stub resolver of systemd-resolved. This file lists all
# configured search domains.
#
# Run "resolvectl status" to see details about the uplink DNS servers
# currently in use.
#
# Third party programs should typically not access this file directly, but only
# through the symlink at /etc/resolv.conf. To manage man:resolv.conf(5) in a
# different way, replace this symlink by a static file or a different symlink.
#
# See man:systemd-resolved.service(8) for details about the supported modes of
# operation for /etc/resolv.conf.
nameserver 127.0.0.53
options edns0 trust-ad
search c.fxci-production-level1-workers.internal google.internal
ubuntu@translations-1-b-linux-v100-gpu-ecnlewoqrtmrltrxrl8o3w:~/tasks/task_171642353933996$ resolvectl status
Global
Protocols: -LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
resolv.conf mode: stub
Link 2 (ens5)
Current Scopes: DNS
Protocols: +DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
Current DNS Server: 169.254.169.254
DNS Servers: 169.254.169.254
DNS Domain: c.fxci-production-level1-workers.internal google.internalIt claims to be timing out talking to 127.0.0.53 - but I don't know if that means it really couldn't talk to the local resolver, or if that's just the local resolver passing along a failure from the upstream. It seems more likely that it's the latter, but I can't say that with any certainty.
The upstream server is a reserved address, and I'm guessing it's something internal to GCP? I'm really not sure, to be honest - that's quite out of my depth.
I looked through syslogs and found nothing of note, just messages like this every time I perfomed a lookup:
May 23 00:42:26 translations-1-b-linux-v100-gpu-efqfgjymq4cpec8nldnk-q systemd-resolved[445]: Received dns UDP packet of size 28, ifindex=0, ttl=64, fragsize=0
May 23 00:42:26 translations-1-b-linux-v100-gpu-efqfgjymq4cpec8nldnk-q systemd-resolved[445]: Got DNS stub UDP query packet for id 63009
May 23 00:42:26 translations-1-b-linux-v100-gpu-efqfgjymq4cpec8nldnk-q systemd-resolved[445]: Looking up RR for github.com IN A.
May 23 00:42:26 translations-1-b-linux-v100-gpu-efqfgjymq4cpec8nldnk-q systemd-resolved[445]: Cache miss for github.com IN A
May 23 00:42:26 translations-1-b-linux-v100-gpu-efqfgjymq4cpec8nldnk-q systemd-resolved[445]: Firing regular transaction 18081 for <github.com IN A> scope dns on ens5/* (validate=yes).
q4cpec8nldnk-q systemd-resolved[445]: Received dns UDP packet of size 28, ifindex=0, ttl=64, fragsize=0
May 23 00:42:26 translations-1-b-linux-v100-gpu-efqfgjymq4cpec8nldnk-q systemd-resolved[445]: Got DNS stub UDP query packet for id 7454
May 23 00:42:26 translations-1-b-linux-v100-gpu-efqfgjymq4cpec8nldnk-q systemd-resolved[445]: Looking up RR for github.com IN MX.
May 23 00:42:26 translations-1-b-linux-v100-gpu-efqfgjymq4cpec8nldnk-q systemd-resolved[445]: Cache miss for github.com IN MX
May 23 00:42:26 translations-1-b-linux-v100-gpu-efqfgjymq4cpec8nldnk-q systemd-resolved[445]: Firing regular transaction 16268 for <github.com IN MX> scope dns on ens5/* (validate=yes).
May 23 00:42:26 translations-1-b-linux-v100-gpu-efqfgjymq4cpec8nldnk-q systemd-resolved[445]: Using feature level UDP+EDNS0 for transaction 16268.
May 23 00:42:26 translations-1-b-linux-v100-gpu-efqfgjymq4cpec8nldnk-q systemd-resolved[445]: Using DNS server 169.254.169.254 for transaction 16268.
May 23 00:42:26 translations-1-b-linux-v100-gpu-efqfgjymq4cpec8nldnk-q systemd-resolved[445]: Announcing packet size 1432 in egress EDNS(0) packet.
May 23 00:42:26 translations-1-b-linux-v100-gpu-efqfgjymq4cpec8nldnk-q systemd-resolved[445]: Emitting UDP, link MTU is 1460, socket MTU is 0, minimal MTU is 40
May 23 00:42:26 translations-1-b-linux-v100-gpu-efqfgjymq4cpec8nldnk-q systemd-resolved[445]: Sending query packet with id 16268 of size 39.
May 23 00:42:26 translations-1-b-linux-v100-gpu-efqfgjymq4cpec8nldnk-q systemd-resolved[445]: Processing query...
May 23 00:42:26 translations-1-b-linux-v100-gpu-efqfgjymq4cpec8nldnk-q systemd-resolved[445]: Received dns UDP packet of size 154, ifindex=2, ttl=0, fragsize=0
May 23 00:42:26 translations-1-b-linux-v100-gpu-efqfgjymq4cpec8nldnk-q systemd-resolved[445]: Processing incoming packet of size 154 on transaction 16268 (rcode=SUCCESS).
May 23 00:42:26 translations-1-b-linux-v100-gpu-efqfgjymq4cpec8nldnk-q systemd-resolved[445]: Added positive unauthenticated non-confidential cache entry for github.com IN MX 3598s on ens5/INET/169.254.169.254
May 23 00:42:26 translations-1-b-linux-v100-gpu-efqfgjymq4cpec8nldnk-q systemd-resolved[445]: message repeated 4 times: [ Added positive unauthenticated non-confidential cache entry for github.com IN MX 3598s on ens5/INET/169.254.169.254]
May 23 00:42:26 translations-1-b-linux-v100-gpu-efqfgjymq4cpec8nldnk-q systemd-resolved[445]: Regular transaction 16268 for <github.com IN MX> on scope dns on ens5/* now complete with <success> from network (unsigned; non-confidential).
May 23 00:42:26 translations-1-b-linux-v100-gpu-efqfgjymq4cpec8nldnk-q systemd-resolved[445]: Sending response packet with id 7454 on interface 1/AF_INET of size 143.
May 23 00:42:26 translations-1-b-linux-v100-gpu-efqfgjymq4cpec8nldnk-q systemd-resolved[445]: Freeing transaction 16268.I was looking through worker logs of a worker that had a dns issue in production and found other things of interest.
In the task we had:
[vcs 2024-05-23T19:33:53.574Z] executing ['git', 'clone', 'https://github.com/mozilla/firefox-translations-training', '/home/ubuntu/tasks/task_171649283082993/checkouts/vcs']
[vcs 2024-05-23T19:33:53.582Z] Cloning into '/home/ubuntu/tasks/task_171649283082993/checkouts/vcs'...
[vcs 2024-05-23T19:34:13.971Z] fatal: unable to access 'https://github.com/mozilla/firefox-translations-training/': Could not resolve host: github.comAnd in the syslogs I found:
May 23 19:34:32 translations-1-b-linux-v100-gpu-sqdnmhpqsrk--fyf5bflrw systemd-resolved[448]: Looking up RR for github.com IN A.
2024-05-23 20:34:41.006
May 23 19:34:32 translations-1-b-linux-v100-gpu-sqdnmhpqsrk--fyf5bflrw systemd-resolved[448]: Cache miss for github.com IN A
2024-05-23 20:34:41.006
May 23 19:34:32 translations-1-b-linux-v100-gpu-sqdnmhpqsrk--fyf5bflrw systemd-resolved[448]: Firing regular transaction 65143 for <github.com IN A> scope dns on ens5/* (validate=yes).
2024-05-23 20:34:41.006
May 23 19:34:32 translations-1-b-linux-v100-gpu-sqdnmhpqsrk--fyf5bflrw systemd-resolved[448]: Looking up RR for github.com IN AAAA.
2024-05-23 20:34:41.006
May 23 19:34:32 translations-1-b-linux-v100-gpu-sqdnmhpqsrk--fyf5bflrw systemd-resolved[448]: Cache miss for github.com IN AAAA
2024-05-23 20:34:41.006
May 23 19:34:32 translations-1-b-linux-v100-gpu-sqdnmhpqsrk--fyf5bflrw systemd-resolved[448]: Firing regular transaction 23655 for <github.com IN AAAA> scope dns on ens5/* (validate=yes).
2024-05-23 20:34:45.041
May 23 19:34:33 translations-1-b-linux-v100-gpu-sqdnmhpqsrk--fyf5bflrw systemd-resolved[448]: Added positive unauthenticated non-confidential cache entry for github.com IN A 60s on ens5/INET/169.254.169.254
2024-05-23 20:34:45.041
May 23 19:34:33 translations-1-b-linux-v100-gpu-sqdnmhpqsrk--fyf5bflrw systemd-resolved[448]: Regular transaction 65143 for <github.com IN A> on scope dns on ens5/* now complete with <success> from network (unsigned; non-confidential).
2024-05-23 20:34:45.041
May 23 19:34:33 translations-1-b-linux-v100-gpu-sqdnmhpqsrk--fyf5bflrw systemd-resolved[448]: Not caching negative entry for: github.com IN AAAA, cache mode set to no-negative
2024-05-23 20:34:45.041
May 23 19:34:33 translations-1-b-linux-v100-gpu-sqdnmhpqsrk--fyf5bflrw systemd-resolved[448]: Regular transaction 23655 for <github.com IN AAAA> on scope dns on ens5/* now complete with <success> from network (unsigned; non-confidential).
2024-05-23 20:34:45.041
May 23 19:34:33 translations-1-b-linux-v100-gpu-sqdnmhpqsrk--fyf5bflrw systemd-resolved[448]: Looking up RR for github.com IN A.
2024-05-23 20:34:45.041
May 23 19:34:33 translations-1-b-linux-v100-gpu-sqdnmhpqsrk--fyf5bflrw systemd-resolved[448]: Positive cache hit for github.com IN A
2024-05-23 20:34:45.041
May 23 19:34:33 translations-1-b-linux-v100-gpu-sqdnmhpqsrk--fyf5bflrw systemd-resolved[448]: Regular transaction 7141 for <github.com IN A> on scope dns on ens5/* now complete with <success> from cache (unsigned; non-confidential).
2024-05-23 20:34:45.041
May 23 19:34:33 translations-1-b-linux-v100-gpu-sqdnmhpqsrk--fyf5bflrw systemd-resolved[448]: Looking up RR for github.com IN AAAA.
2024-05-23 20:34:45.041
May 23 19:34:33 translations-1-b-linux-v100-gpu-sqdnmhpqsrk--fyf5bflrw systemd-resolved[448]: Cache miss for github.com IN AAAA
2024-05-23 20:34:45.041
May 23 19:34:33 translations-1-b-linux-v100-gpu-sqdnmhpqsrk--fyf5bflrw systemd-resolved[448]: Firing regular transaction 60167 for <github.com IN AAAA> scope dns on ens5/* (validate=yes).
2024-05-23 20:34:45.041
May 23 19:34:33 translations-1-b-linux-v100-gpu-sqdnmhpqsrk--fyf5bflrw systemd-resolved[448]: Not caching negative entry for: github.com IN AAAA, cache mode set to no-negative
2024-05-23 20:34:45.041
May 23 19:34:33 translations-1-b-linux-v100-gpu-sqdnmhpqsrk--fyf5bflrw systemd-resolved[448]: Regular transaction 60167 for <github.com IN AAAA> on scope dns on ens5/* now complete with <success> from network (unsigned; non-confidential).
2024-05-23 20:34:45.041
May 23 19:34:33 translations-1-b-linux-v100-gpu-sqdnmhpqsrk--fyf5bflrw systemd-resolved[448]: Looking up RR for github.com.c.fxci-production-level1-workers.internal IN A.
2024-05-23 20:34:45.041
May 23 19:34:33 translations-1-b-linux-v100-gpu-sqdnmhpqsrk--fyf5bflrw systemd-resolved[448]: Cache miss for github.com.c.fxci-production-level1-workers.internal IN A
2024-05-23 20:34:45.041
May 23 19:34:33 translations-1-b-linux-v100-gpu-sqdnmhpqsrk--fyf5bflrw systemd-resolved[448]: Firing regular transaction 48751 for <github.com.c.fxci-production-level1-workers.internal IN A> scope dns on ens5/* (validate=yes).
2024-05-23 20:34:45.041
May 23 19:34:33 translations-1-b-linux-v100-gpu-sqdnmhpqsrk--fyf5bflrw systemd-resolved[448]: Looking up RR for github.com.c.fxci-production-level1-workers.internal IN AAAA.
2024-05-23 20:34:45.041
May 23 19:34:33 translations-1-b-linux-v100-gpu-sqdnmhpqsrk--fyf5bflrw systemd-resolved[448]: Cache miss for github.com.c.fxci-production-level1-workers.internal IN AAAA
2024-05-23 20:34:45.041
May 23 19:34:33 translations-1-b-linux-v100-gpu-sqdnmhpqsrk--fyf5bflrw systemd-resolved[448]: Firing regular transaction 26112 for <github.com.c.fxci-production-level1-workers.internal IN AAAA> scope dns on ens5/* (validate=yes).
2024-05-23 20:34:48.739
May 23 19:34:33 translations-1-b-linux-v100-gpu-sqdnmhpqsrk--fyf5bflrw systemd-resolved[448]: Cache miss for github.com.c.fxci-production-level1-workers.internal IN A
2024-05-23 20:34:48.739
May 23 19:34:33 translations-1-b-linux-v100-gpu-sqdnmhpqsrk--fyf5bflrw systemd-resolved[448]: Firing regular transaction 48751 for <github.com.c.fxci-production-level1-workers.internal IN A> scope dns on ens5/* (validate=yes).
2024-05-23 20:34:48.739
May 23 19:34:33 translations-1-b-linux-v100-gpu-sqdnmhpqsrk--fyf5bflrw systemd-resolved[448]: Cache miss for github.com.c.fxci-production-level1-workers.internal IN AAAA
2024-05-23 20:34:48.739
May 23 19:34:33 translations-1-b-linux-v100-gpu-sqdnmhpqsrk--fyf5bflrw systemd-resolved[448]: Firing regular transaction 26112 for <github.com.c.fxci-production-level1-workers.internal IN AAAA> scope dns on ens5/* (validate=yes).
2024-05-23 20:34:48.739
May 23 19:34:33 translations-1-b-linux-v100-gpu-sqdnmhpqsrk--fyf5bflrw systemd-resolved[448]: Not caching negative entry for: github.com.c.fxci-production-level1-workers.internal IN A, cache mode set to no-negative
2024-05-23 20:34:48.739
May 23 19:34:33 translations-1-b-linux-v100-gpu-sqdnmhpqsrk--fyf5bflrw systemd-resolved[448]: Regular transaction 48751 for <github.com.c.fxci-production-level1-workers.internal IN A> on scope dns on ens5/* now complete with <rcode-failure> from network (unsigned; non-confidential).
2024-05-23 20:34:48.739
May 23 19:34:33 translations-1-b-linux-v100-gpu-sqdnmhpqsrk--fyf5bflrw systemd-resolved[448]: Not caching negative entry for: github.com.c.fxci-production-level1-workers.internal IN AAAA, cache mode set to no-negative
2024-05-23 20:34:48.739
May 23 19:34:33 translations-1-b-linux-v100-gpu-sqdnmhpqsrk--fyf5bflrw systemd-resolved[448]: Regular transaction 26112 for <github.com.c.fxci-production-level1-workers.internal IN AAAA> on scope dns on ens5/* now complete with <rcode-failure> from network (unsigned; non-confidential).
2024-05-23 20:34:52.098
May 23 19:34:33 translations-1-b-linux-v100-gpu-sqdnmhpqsrk--fyf5bflrw systemd-resolved[448]: Looking up RR for github.com.c.fxci-production-level1-workers.internal IN A.
2024-05-23 20:34:52.098
May 23 19:34:33 translations-1-b-linux-v100-gpu-sqdnmhpqsrk--fyf5bflrw systemd-resolved[448]: Cache miss for github.com.c.fxci-production-level1-workers.internal IN A
2024-05-23 20:34:52.098(For whatever reason, there seems to be a timestamp discrepancy between the task log and the system logs, for example we have messages like May 23 19:34:23 translations-1-b-linux-v100-gpu-sqdnmhpqsrk--fyf5bflrw start-worker[572]: 2024/05/23 19:33:51 Environment: []string that show a 32 second difference.)
Within a very short time we see:
Maybe that second lookup is expected, maybe it's not - I'm really not sure. The github.com one succeeds, the .internal one does not (of course). I searched through all the logs and there are no failures to resolve the real github.com - the only failures are for .internal domains.
I'm still not sure what to make of this, just dropping more info at the moment.
bhearsum
commented
6 months ago Another interesting thing is that we 12 attempts to look up the A record for github.com in the span of 12 seconds, while there were 3 attempts to clone over the span of ~40 seconds (the last one succeeding):
"May 23 19:34:32 translations-1-b-linux-v100-gpu-sqdnmhpqsrk--fyf5bflrw systemd-resolved[448]: Looking up RR for github.com IN A."
"May 23 19:34:33 translations-1-b-linux-v100-gpu-sqdnmhpqsrk--fyf5bflrw systemd-resolved[448]: Looking up RR for github.com IN A."
"May 23 19:34:33 translations-1-b-linux-v100-gpu-sqdnmhpqsrk--fyf5bflrw systemd-resolved[448]: Looking up RR for github.com IN A."
"May 23 19:34:33 translations-1-b-linux-v100-gpu-sqdnmhpqsrk--fyf5bflrw systemd-resolved[448]: Looking up RR for github.com IN A."
"May 23 19:34:37 translations-1-b-linux-v100-gpu-sqdnmhpqsrk--fyf5bflrw systemd-resolved[448]: Looking up RR for github.com IN A."
"May 23 19:34:38 translations-1-b-linux-v100-gpu-sqdnmhpqsrk--fyf5bflrw systemd-resolved[448]: Looking up RR for github.com IN A."
"May 23 19:34:39 translations-1-b-linux-v100-gpu-sqdnmhpqsrk--fyf5bflrw systemd-resolved[448]: Looking up RR for github.com IN A."
"May 23 19:34:41 translations-1-b-linux-v100-gpu-sqdnmhpqsrk--fyf5bflrw systemd-resolved[448]: Looking up RR for github.com IN A."
"May 23 19:34:41 translations-1-b-linux-v100-gpu-sqdnmhpqsrk--fyf5bflrw systemd-resolved[448]: Looking up RR for github.com IN A."
"May 23 19:34:41 translations-1-b-linux-v100-gpu-sqdnmhpqsrk--fyf5bflrw systemd-resolved[448]: Looking up RR for github.com IN A."
"May 23 19:34:42 translations-1-b-linux-v100-gpu-sqdnmhpqsrk--fyf5bflrw systemd-resolved[448]: Looking up RR for github.com IN A."
"May 23 19:34:44 translations-1-b-linux-v100-gpu-sqdnmhpqsrk--fyf5bflrw systemd-resolved[448]: Looking up RR for github.com IN A."[vcs 2024-05-23T19:33:53.574Z] executing ['git', 'clone', 'https://github.com/mozilla/firefox-translations-training', '/home/ubuntu/tasks/task_171649283082993/checkouts/vcs']
[vcs 2024-05-23T19:33:53.582Z] Cloning into '/home/ubuntu/tasks/task_171649283082993/checkouts/vcs'...
[vcs 2024-05-23T19:34:13.971Z] fatal: unable to access 'https://github.com/mozilla/firefox-translations-training/': Could not resolve host: github.com
[vcs 2024-05-23T19:34:15.976Z] executing ['git', 'clone', 'https://github.com/mozilla/firefox-translations-training', '/home/ubuntu/tasks/task_171649283082993/checkouts/vcs']
[vcs 2024-05-23T19:34:15.978Z] Cloning into '/home/ubuntu/tasks/task_171649283082993/checkouts/vcs'...
[vcs 2024-05-23T19:34:33.081Z] fatal: unable to access 'https://github.com/mozilla/firefox-translations-training/': Could not resolve host: github.com
[vcs 2024-05-23T19:34:37.087Z] executing ['git', 'clone', 'https://github.com/mozilla/firefox-translations-training', '/home/ubuntu/tasks/task_171649283082993/checkouts/vcs']
[vcs 2024-05-23T19:34:37.089Z] Cloning into '/home/ubuntu/tasks/task_171649283082993/checkouts/vcs'...Did this go away by itself?
bhearsum
commented
5 months ago I'm not aware of anything we did to fix it. So if it's not happening anymore...yes!
bhearsum
commented
2 months ago We spoke too soon clearly, given #855.
Some suggestions for further investigation from the Taskcluster weekly meeting:
aerickson
commented
2 months ago https://github.com/mozilla-platform-ops/monopacker/pull/140 was the PR related to this image, but I don't know how deterministic the other parts are? Eg: could we have picked up a change to a system package that is now causing problems?
@aerickson - do you have any thoughts? If we still have the old image, maybe we could poke around and compare the new to old one? (I'd be happy to do this if you want.)
It's possible. We don't pin the base image (we accept patch level updates like 22.04.x) and the scripts run apt update && apt upgrade, so it's possible. We do have the older images if you want to investigate. We're rolling out SBOMs too, so it will be easier to compare in the future (no booting an instance, you'll be able to diff a markdown file).
bhearsum
commented
1 month ago I had a chance to compare the most recent GPU image to the previous one today. As far as packages installed goes, there's a bunch of minor upgrades but nothing that really stands out. The closest things to a network related package that changed are bind9-host and a very minor kernel upgrade - neither of which seem terribly likely to cause intermittent DNS issues. The full list of packages changes is as follows:
@@ -7,26 +7,26 @@
ii adduser 3.118ubuntu5 all add and remove users and groups
ii adwaita-icon-theme 41.0-1ubuntu1 all default icon theme of GNOME (small subset)
ii alsa-topology-conf 1.2.5.1-2 all ALSA topology configuration files
-ii alsa-ucm-conf 1.2.6.3-1ubuntu1.10 all ALSA Use Case Manager configuration files
+ii alsa-ucm-conf 1.2.6.3-1ubuntu1.11 all ALSA Use Case Manager configuration files
ii apparmor 3.0.4-2ubuntu2.3 amd64 user-space parser utility for AppArmor
ii apport 2.20.11-0ubuntu82.5 all automatically generate crash reports for debugging
ii apport-symptoms 0.24 all symptom scripts for apport
ii apt 2.4.11 amd64 commandline package manager
-ii apt-transport-https 2.4.11 all transitional package for https support
+ii apt-transport-https 2.4.12 all transitional package for https support
ii apt-utils 2.4.11 amd64 package management related utility programs
ii at-spi2-core 2.44.0-3 amd64 Assistive Technology Service Provider Interface (dbus core)
-ii base-files 12ubuntu4.5 amd64 Debian base system miscellaneous files
+ii base-files 12ubuntu4.6 amd64 Debian base system miscellaneous files
ii base-passwd 3.5.52build1 amd64 Debian base system master password and group files
-ii bash 5.1-6ubuntu1 amd64 GNU Bourne Again SHell
+ii bash 5.1-6ubuntu1.1 amd64 GNU Bourne Again SHell
ii bash-completion 1:2.11-5ubuntu1 all programmable completion for the bash shell
ii bc 1.07.1-3build1 amd64 GNU bc arbitrary precision calculator language
ii bcache-tools 1.0.8-4ubuntu3 amd64 bcache userspace tools
-ii bind9-dnsutils 1:9.18.18-0ubuntu0.22.04.1 amd64 Clients provided with BIND 9
-ii bind9-host 1:9.18.18-0ubuntu0.22.04.1 amd64 DNS Lookup Utility
-ii bind9-libs:amd64 1:9.18.18-0ubuntu0.22.04.1 amd64 Shared Libraries used by BIND 9
-ii binutils 2.38-4ubuntu2.5 amd64 GNU assembler, linker and binary utilities
-ii binutils-common:amd64 2.38-4ubuntu2.5 amd64 Common files for the GNU assembler, linker and binary utilities
-ii binutils-x86-64-linux-gnu 2.38-4ubuntu2.5 amd64 GNU binary utilities, for x86-64-linux-gnu target
+ii bind9-dnsutils 1:9.18.18-0ubuntu0.22.04.2 amd64 Clients provided with BIND 9
+ii bind9-host 1:9.18.18-0ubuntu0.22.04.2 amd64 DNS Lookup Utility
+ii bind9-libs:amd64 1:9.18.18-0ubuntu0.22.04.2 amd64 Shared Libraries used by BIND 9
+ii binutils 2.38-4ubuntu2.6 amd64 GNU assembler, linker and binary utilities
+ii binutils-common:amd64 2.38-4ubuntu2.6 amd64 Common files for the GNU assembler, linker and binary utilities
+ii binutils-x86-64-linux-gnu 2.38-4ubuntu2.6 amd64 GNU binary utilities, for x86-64-linux-gnu target
ii bsdextrautils 2.37.2-4ubuntu3 amd64 extra utilities from 4.4BSD-Lite
ii bsdutils 1:2.37.2-4ubuntu3 amd64 basic utilities from 4.4BSD-Lite
ii btrfs-progs 5.16.2-1 amd64 Checksumming Copy on Write Filesystem utilities
@@ -39,13 +39,13 @@
ii ca-certificates-java 20190909ubuntu1.2 all Common CA certificates (JKS keystore)
ii chrony 4.2-2ubuntu2 amd64 Versatile implementation of the Network Time Protocol
ii cloud-guest-utils 0.32-22-g45fe84a5-0ubuntu1 all cloud guest utilities
-ii cloud-init 23.3.3-0ubuntu0~22.04.1 all initialization and customization tool for cloud instances
+ii cloud-init 23.4.4-0ubuntu0~22.04.1 all initialization and customization tool for cloud instances
ii cloud-initramfs-copymods 0.47ubuntu1 all copy initramfs modules into root filesystem for later use
ii cloud-initramfs-dyn-netconf 0.47ubuntu1 all write a network interface file in /run for BOOTIF
ii command-not-found 22.04.0 all Suggest installation of packages in interactive bash sessions
ii console-setup 1.205ubuntu3 all console font and keymap setup program
ii console-setup-linux 1.205ubuntu3 all Linux specific part of console-setup
-ii coreutils 8.32-4.1ubuntu1 amd64 GNU core utilities
+ii coreutils 8.32-4.1ubuntu1.1 amd64 GNU core utilities
ii cpio 2.13+dfsg-7 amd64 GNU cpio -- a program to manage archives of files
ii cpp 4:11.2.0-1ubuntu1 amd64 GNU C preprocessor (cpp)
ii cpp-11 11.4.0-1ubuntu1~22.04 amd64 GNU C preprocessor
@@ -54,52 +54,52 @@
ii cryptsetup 2:2.4.3-1ubuntu1.2 amd64 disk encryption support - startup scripts
ii cryptsetup-bin 2:2.4.3-1ubuntu1.2 amd64 disk encryption support - command line tools
ii cryptsetup-initramfs 2:2.4.3-1ubuntu1.2 all disk encryption support - initramfs integration
-ii cuda 12.3.2-1 amd64 CUDA meta-package
-ii cuda-12-3 12.3.2-1 amd64 CUDA 12.3 meta-package
-ii cuda-cccl-12-3 12.3.101-1 amd64 CUDA CCCL
-ii cuda-command-line-tools-12-3 12.3.2-1 amd64 CUDA command-line tools
-ii cuda-compiler-12-3 12.3.2-1 amd64 CUDA compiler
-ii cuda-crt-12-3 12.3.107-1 amd64 CUDA crt
-ii cuda-cudart-12-3 12.3.101-1 amd64 CUDA Runtime native Libraries
-ii cuda-cudart-dev-12-3 12.3.101-1 amd64 CUDA Runtime native dev links, headers
-ii cuda-cuobjdump-12-3 12.3.101-1 amd64 CUDA cuobjdump
-ii cuda-cupti-12-3 12.3.101-1 amd64 CUDA profiling tools runtime libs.
-ii cuda-cupti-dev-12-3 12.3.101-1 amd64 CUDA profiling tools interface.
-ii cuda-cuxxfilt-12-3 12.3.101-1 amd64 CUDA cuxxfilt
-ii cuda-demo-suite-12-3 12.3.101-1 amd64 Demo suite for CUDA
-ii cuda-documentation-12-3 12.3.101-1 amd64 CUDA documentation
-ii cuda-driver-dev-12-3 12.3.101-1 amd64 CUDA Driver native dev stub library
-ii cuda-drivers 545.23.08-1 amd64 CUDA Driver meta-package, branch-agnostic
-ii cuda-drivers-545 545.23.08-1 amd64 CUDA Driver meta-package, branch-specific
-ii cuda-gdb-12-3 12.3.101-1 amd64 CUDA-GDB
+ii cuda 12.4.1-1 amd64 CUDA meta-package
+ii cuda-12-4 12.4.1-1 amd64 CUDA 12.4 meta-package
+ii cuda-cccl-12-4 12.4.127-1 amd64 CUDA CCCL
+ii cuda-command-line-tools-12-4 12.4.1-1 amd64 CUDA command-line tools
+ii cuda-compiler-12-4 12.4.1-1 amd64 CUDA compiler
+ii cuda-crt-12-4 12.4.131-1 amd64 CUDA crt
+ii cuda-cudart-12-4 12.4.127-1 amd64 CUDA Runtime native Libraries
+ii cuda-cudart-dev-12-4 12.4.127-1 amd64 CUDA Runtime native dev links, headers
+ii cuda-cuobjdump-12-4 12.4.127-1 amd64 CUDA cuobjdump
+ii cuda-cupti-12-4 12.4.127-1 amd64 CUDA profiling tools runtime libs.
+ii cuda-cupti-dev-12-4 12.4.127-1 amd64 CUDA profiling tools interface.
+ii cuda-cuxxfilt-12-4 12.4.127-1 amd64 CUDA cuxxfilt
+ii cuda-demo-suite-12-4 12.4.127-1 amd64 Demo suite for CUDA
+ii cuda-documentation-12-4 12.4.127-1 amd64 CUDA documentation
+ii cuda-driver-dev-12-4 12.4.127-1 amd64 CUDA Driver native dev stub library
+ii cuda-drivers 550.54.15-1 amd64 CUDA Driver meta-package, branch-agnostic
+ii cuda-drivers-550 550.54.15-1 amd64 CUDA Driver meta-package, branch-specific
+ii cuda-gdb-12-4 12.4.127-1 amd64 CUDA-GDB
ii cuda-keyring 1.0-1 all GPG keyring for the CUDA repository
-ii cuda-libraries-12-3 12.3.2-1 amd64 CUDA Libraries 12.3 meta-package
-ii cuda-libraries-dev-12-3 12.3.2-1 amd64 CUDA Libraries 12.3 development meta-package
-ii cuda-nsight-12-3 12.3.101-1 amd64 CUDA nsight
-ii cuda-nsight-compute-12-3 12.3.2-1 amd64 NVIDIA Nsight Compute
-ii cuda-nsight-systems-12-3 12.3.2-1 amd64 NVIDIA Nsight Systems
-ii cuda-nvcc-12-3 12.3.107-1 amd64 CUDA nvcc
-ii cuda-nvdisasm-12-3 12.3.101-1 amd64 CUDA disassembler
-ii cuda-nvml-dev-12-3 12.3.101-1 amd64 NVML native dev links, headers
-ii cuda-nvprof-12-3 12.3.101-1 amd64 CUDA Profiler tools
-ii cuda-nvprune-12-3 12.3.101-1 amd64 CUDA nvprune
-ii cuda-nvrtc-12-3 12.3.107-1 amd64 NVRTC native runtime libraries
-ii cuda-nvrtc-dev-12-3 12.3.107-1 amd64 NVRTC native dev links, headers
-ii cuda-nvtx-12-3 12.3.101-1 amd64 NVIDIA Tools Extension
-ii cuda-nvvm-12-3 12.3.107-1 amd64 CUDA nvvm
-ii cuda-nvvp-12-3 12.3.101-1 amd64 CUDA Profiler tools
-ii cuda-opencl-12-3 12.3.101-1 amd64 CUDA OpenCL native Libraries
-ii cuda-opencl-dev-12-3 12.3.101-1 amd64 CUDA OpenCL native dev links, headers
-ii cuda-profiler-api-12-3 12.3.101-1 amd64 CUDA Profiler API
-ii cuda-runtime-12-3 12.3.2-1 amd64 CUDA Runtime 12.3 meta-package
-ii cuda-sanitizer-12-3 12.3.101-1 amd64 CUDA Sanitizer
-ii cuda-toolkit-12-3 12.3.2-1 amd64 CUDA Toolkit 12.3 meta-package
-ii cuda-toolkit-12-3-config-common 12.3.101-1 all Common config package for CUDA Toolkit 12.3.
-ii cuda-toolkit-12-config-common 12.3.101-1 all Common config package for CUDA Toolkit 12.
-ii cuda-toolkit-config-common 12.3.101-1 all Common config package for CUDA Toolkit.
-ii cuda-tools-12-3 12.3.2-1 amd64 CUDA Tools meta-package
-ii cuda-visual-tools-12-3 12.3.2-1 amd64 CUDA visual tools
-ii curl 7.81.0-1ubuntu1.15 amd64 command line tool for transferring data with URL syntax
+ii cuda-libraries-12-4 12.4.1-1 amd64 CUDA Libraries 12.4 meta-package
+ii cuda-libraries-dev-12-4 12.4.1-1 amd64 CUDA Libraries 12.4 development meta-package
+ii cuda-nsight-12-4 12.4.127-1 amd64 CUDA nsight
+ii cuda-nsight-compute-12-4 12.4.1-1 amd64 NVIDIA Nsight Compute
+ii cuda-nsight-systems-12-4 12.4.1-1 amd64 NVIDIA Nsight Systems
+ii cuda-nvcc-12-4 12.4.131-1 amd64 CUDA nvcc
+ii cuda-nvdisasm-12-4 12.4.127-1 amd64 CUDA disassembler
+ii cuda-nvml-dev-12-4 12.4.127-1 amd64 NVML native dev links, headers
+ii cuda-nvprof-12-4 12.4.127-1 amd64 CUDA Profiler tools
+ii cuda-nvprune-12-4 12.4.127-1 amd64 CUDA nvprune
+ii cuda-nvrtc-12-4 12.4.127-1 amd64 NVRTC native runtime libraries
+ii cuda-nvrtc-dev-12-4 12.4.127-1 amd64 NVRTC native dev links, headers
+ii cuda-nvtx-12-4 12.4.127-1 amd64 NVIDIA Tools Extension
+ii cuda-nvvm-12-4 12.4.131-1 amd64 CUDA nvvm
+ii cuda-nvvp-12-4 12.4.127-1 amd64 CUDA Profiler tools
+ii cuda-opencl-12-4 12.4.127-1 amd64 CUDA OpenCL native Libraries
+ii cuda-opencl-dev-12-4 12.4.127-1 amd64 CUDA OpenCL native dev links, headers
+ii cuda-profiler-api-12-4 12.4.127-1 amd64 CUDA Profiler API
+ii cuda-runtime-12-4 12.4.1-1 amd64 CUDA Runtime 12.4 meta-package
+ii cuda-sanitizer-12-4 12.4.127-1 amd64 CUDA Sanitizer
+ii cuda-toolkit-12-4 12.4.1-1 amd64 CUDA Toolkit 12.4 meta-package
+ii cuda-toolkit-12-4-config-common 12.4.127-1 all Common config package for CUDA Toolkit 12.4.
+ii cuda-toolkit-12-config-common 12.4.127-1 all Common config package for CUDA Toolkit 12.
+ii cuda-toolkit-config-common 12.4.127-1 all Common config package for CUDA Toolkit.
+ii cuda-tools-12-4 12.4.1-1 amd64 CUDA Tools meta-package
+ii cuda-visual-tools-12-4 12.4.1-1 amd64 CUDA visual tools
+ii curl 7.81.0-1ubuntu1.16 amd64 command line tool for transferring data with URL syntax
ii dash 0.5.11+git20210903+057cd650a4ed-3build1 amd64 POSIX-compliant shell
ii dbus 1.12.20-2ubuntu4.1 amd64 simple interprocess messaging system (daemon and utilities)
ii dbus-user-session 1.12.20-2ubuntu4.1 amd64 simple interprocess messaging system (systemd --user integration)
@@ -121,8 +121,8 @@
ii dmidecode 3.3-3ubuntu0.1 amd64 SMBIOS/DMI table decoder
ii dmsetup 2:1.02.175-2.1ubuntu4 amd64 Linux Kernel Device Mapper userspace library
ii dosfstools 4.2-1build3 amd64 utilities for making and checking MS-DOS FAT filesystems
-ii dpkg 1.21.1ubuntu2.2 amd64 Debian package management system
-ii dpkg-dev 1.21.1ubuntu2.2 all Debian package development tools
+ii dpkg 1.21.1ubuntu2.3 amd64 Debian package management system
+ii dpkg-dev 1.21.1ubuntu2.3 all Debian package development tools
ii e2fsprogs 1.46.5-2ubuntu1.1 amd64 ext2/ext3/ext4 file system utilities
ii eatmydata 130-2build1 all Library and utilities designed to disable fsync and friends
ii ed 1.18-1 amd64 classic UNIX line editor
@@ -152,7 +152,7 @@
ii gcc-12 12.3.0-1ubuntu1~22.04 amd64 GNU C compiler
ii gcc-12-base:amd64 12.3.0-1ubuntu1~22.04 amd64 GCC, the GNU Compiler Collection (base package)
ii gdisk 1.0.8-4build1 amd64 GPT fdisk text-mode partitioning tool
-ii gds-tools-12-3 1.8.1.2-1 amd64 Tools for GPU Direct Storage
+ii gds-tools-12-4 1.9.1.3-1 amd64 Tools for GPU Direct Storage
ii gettext-base 0.21-4ubuntu4 amd64 GNU Internationalization utilities for the base system
ii gir1.2-glib-2.0:amd64 1.72.0-1 amd64 Introspection data for GLib, GObject, Gio and GModule
ii gir1.2-packagekitglib-1.0 1.2.5-2ubuntu2 amd64 GObject introspection data for the PackageKit GLib library
@@ -199,7 +199,7 @@
ii initramfs-tools-core 0.140ubuntu13.4 all generic modular initramfs generator (core tools)
ii install-info 6.8-4build1 amd64 Manage installed documentation in info format
ii iproute2 5.15.0-1ubuntu2 amd64 networking and traffic control tools
-ii iptables 1.8.7-1ubuntu5.1 amd64 administration tools for packet filtering and NAT
+ii iptables 1.8.7-1ubuntu5.2 amd64 administration tools for packet filtering and NAT
ii iptables-persistent 1.0.16 all boot-time loader for netfilter rules, iptables plugin
ii iputils-ping 3:20211215-1 amd64 Tools to test the reachability of network hosts
ii iputils-tracepath 3:20211215-1 amd64 Tools to trace the network path to a remote host
@@ -214,7 +214,7 @@
ii kmod 29-1ubuntu1 amd64 tools for managing Linux kernel modules
ii kpartx 0.8.8-1ubuntu1.22.04.4 amd64 create device mappings for partitions
ii landscape-common 19.12-0ubuntu13 amd64 Landscape administration system client - Common files
-ii less 590-1ubuntu0.22.04.1 amd64 pager program similar to more
+ii less 590-1ubuntu0.22.04.2 amd64 pager program similar to more
ii libacl1:amd64 2.3.1-1 amd64 access control list - shared library
ii libaio1:amd64 0.3.112-13build1 amd64 Linux kernel AIO access library - shared library
ii libalgorithm-diff-perl 1.201-1 all module to find differences between files
@@ -244,9 +244,9 @@
ii libavahi-client3:amd64 0.8-5ubuntu5.2 amd64 Avahi client library
ii libavahi-common-data:amd64 0.8-5ubuntu5.2 amd64 Avahi common data files
ii libavahi-common3:amd64 0.8-5ubuntu5.2 amd64 Avahi common library
-ii libbinutils:amd64 2.38-4ubuntu2.5 amd64 GNU binary utilities (private shared library)
-ii libblkid-dev:amd64 2.37.2-4ubuntu3 amd64 block device ID library - headers
-ii libblkid1:amd64 2.37.2-4ubuntu3 amd64 block device ID library
+ii libbinutils:amd64 2.38-4ubuntu2.6 amd64 GNU binary utilities (private shared library)
+ii libblkid-dev:amd64 2.37.2-4ubuntu3.4 amd64 block device ID library - headers
+ii libblkid1:amd64 2.37.2-4ubuntu3.4 amd64 block device ID library
ii libblockdev-fs2:amd64 2.26-1 amd64 file system plugin for libblockdev
ii libblockdev-loop2:amd64 2.26-1 amd64 Loop device plugin for libblockdev
ii libblockdev-part-err2:amd64 2.26-1 amd64 Partition error utility functions for libblockdev
@@ -259,10 +259,10 @@
ii libbsd0:amd64 0.11.5-1 amd64 utility functions from BSD systems - shared library
ii libbz2-1.0:amd64 1.0.8-5build1 amd64 high-quality block-sorting file compressor library - runtime
ii libc-bin 2.35-0ubuntu3.6 amd64 GNU C Library: Binaries
-ii libc-dev-bin 2.35-0ubuntu3.6 amd64 GNU C Library: Development binaries
-ii libc-devtools 2.35-0ubuntu3.6 amd64 GNU C Library: Development tools
-ii libc6:amd64 2.35-0ubuntu3.6 amd64 GNU C Library: Shared libraries
-ii libc6-dev:amd64 2.35-0ubuntu3.6 amd64 GNU C Library: Development Libraries and Header Files
+ii libc-dev-bin 2.35-0ubuntu3.7 amd64 GNU C Library: Development binaries
+ii libc-devtools 2.35-0ubuntu3.7 amd64 GNU C Library: Development tools
+ii libc6:amd64 2.35-0ubuntu3.7 amd64 GNU C Library: Shared libraries
+ii libc6-dev:amd64 2.35-0ubuntu3.7 amd64 GNU C Library: Development Libraries and Header Files
ii libcairo-gobject2:amd64 1.16.0-5ubuntu2 amd64 Cairo 2D vector graphics library (GObject library)
ii libcairo2:amd64 1.16.0-5ubuntu2 amd64 Cairo 2D vector graphics library
ii libcap-ng0:amd64 0.7.9-2.2build3 amd64 An alternate POSIX capabilities library
@@ -275,25 +275,25 @@
ii libcrypt-dev:amd64 1:4.4.27-1 amd64 libcrypt development files
ii libcrypt1:amd64 1:4.4.27-1 amd64 libcrypt shared library
ii libcryptsetup12:amd64 2:2.4.3-1ubuntu1.2 amd64 disk encryption support - shared library
-ii libctf-nobfd0:amd64 2.38-4ubuntu2.5 amd64 Compact C Type Format library (runtime, no BFD dependency)
-ii libctf0:amd64 2.38-4ubuntu2.5 amd64 Compact C Type Format library (runtime, BFD dependency)
-ii libcublas-12-3 12.3.4.1-1 amd64 CUBLAS native runtime libraries
-ii libcublas-dev-12-3 12.3.4.1-1 amd64 CUBLAS native dev links, headers
+ii libctf-nobfd0:amd64 2.38-4ubuntu2.6 amd64 Compact C Type Format library (runtime, no BFD dependency)
+ii libctf0:amd64 2.38-4ubuntu2.6 amd64 Compact C Type Format library (runtime, BFD dependency)
+ii libcublas-12-4 12.4.5.8-1 amd64 CUBLAS native runtime libraries
+ii libcublas-dev-12-4 12.4.5.8-1 amd64 CUBLAS native dev links, headers
ii libcudnn8 8.8.1.3-1+cuda12.0 amd64 cuDNN runtime libraries
ii libcudnn8-dev 8.8.1.3-1+cuda12.0 amd64 cuDNN development libraries and headers
-ii libcufft-12-3 11.0.12.1-1 amd64 CUFFT native runtime libraries
-ii libcufft-dev-12-3 11.0.12.1-1 amd64 CUFFT native dev links, headers
-ii libcufile-12-3 1.8.1.2-1 amd64 Library for GPU Direct Storage with CUDA 12.3
-ii libcufile-dev-12-3 1.8.1.2-1 amd64 Library for GPU Direct Storage native dev links, headers
-ii libcups2:amd64 2.4.1op1-1ubuntu4.7 amd64 Common UNIX Printing System(tm) - Core library
-ii libcurand-12-3 10.3.4.107-1 amd64 CURAND native runtime libraries
-ii libcurand-dev-12-3 10.3.4.107-1 amd64 CURAND native dev links, headers
+ii libcufft-12-4 11.2.1.3-1 amd64 CUFFT native runtime libraries
+ii libcufft-dev-12-4 11.2.1.3-1 amd64 CUFFT native dev links, headers
+ii libcufile-12-4 1.9.1.3-1 amd64 Library for GPU Direct Storage with CUDA 12.4
+ii libcufile-dev-12-4 1.9.1.3-1 amd64 Library for GPU Direct Storage native dev links, headers
+ii libcups2:amd64 2.4.1op1-1ubuntu4.8 amd64 Common UNIX Printing System(tm) - Core library
+ii libcurand-12-4 10.3.5.147-1 amd64 CURAND native runtime libraries
+ii libcurand-dev-12-4 10.3.5.147-1 amd64 CURAND native dev links, headers
ii libcurl3-gnutls:amd64 7.81.0-1ubuntu1.15 amd64 easy-to-use client-side URL transfer library (GnuTLS flavour)
-ii libcurl4:amd64 7.81.0-1ubuntu1.15 amd64 easy-to-use client-side URL transfer library (OpenSSL flavour)
-ii libcusolver-12-3 11.5.4.101-1 amd64 CUDA solver native runtime libraries
-ii libcusolver-dev-12-3 11.5.4.101-1 amd64 CUDA solver native dev links, headers
-ii libcusparse-12-3 12.2.0.103-1 amd64 CUSPARSE native runtime libraries
-ii libcusparse-dev-12-3 12.2.0.103-1 amd64 CUSPARSE native dev links, headers
+ii libcurl4:amd64 7.81.0-1ubuntu1.16 amd64 easy-to-use client-side URL transfer library (OpenSSL flavour)
+ii libcusolver-12-4 11.6.1.9-1 amd64 CUDA solver native runtime libraries
+ii libcusolver-dev-12-4 11.6.1.9-1 amd64 CUDA solver native dev links, headers
+ii libcusparse-12-4 12.3.1.170-1 amd64 CUSPARSE native runtime libraries
+ii libcusparse-dev-12-4 12.3.1.170-1 amd64 CUSPARSE native dev links, headers
ii libdatrie1:amd64 0.2.13-2 amd64 Double-array trie library
ii libdb5.3:amd64 5.3.28+dfsg1-0.8ubuntu3 amd64 Berkeley v5.3 Database Libraries [runtime]
ii libdbus-1-3:amd64 1.12.20-2ubuntu4.1 amd64 simple interprocess messaging system (library)
@@ -303,7 +303,7 @@
ii libdevmapper-event1.02.1:amd64 2:1.02.175-2.1ubuntu4 amd64 Linux Kernel Device Mapper event support library
ii libdevmapper1.02.1:amd64 2:1.02.175-2.1ubuntu4 amd64 Linux Kernel Device Mapper userspace library
ii libdns-export1110 1:9.11.19+dfsg-2.1ubuntu3 amd64 Exported DNS Shared Library
-ii libdpkg-perl 1.21.1ubuntu2.2 all Dpkg perl modules
+ii libdpkg-perl 1.21.1ubuntu2.3 all Dpkg perl modules
ii libdrm-amdgpu1:amd64 2.4.113-2~ubuntu0.22.04.1 amd64 Userspace interface to amdgpu-specific kernel DRM services -- runtime
ii libdrm-common 2.4.113-2~ubuntu0.22.04.1 all Userspace interface to kernel DRM services -- common files
ii libdrm-intel1:amd64 2.4.113-2~ubuntu0.22.04.1 amd64 Userspace interface to intel-specific kernel DRM services -- runtime
@@ -322,8 +322,8 @@
ii liberror-perl 0.17029-1 all Perl module for error/exception handling in an OO-ish way
ii libestr0:amd64 0.1.10-2.1build3 amd64 Helper functions for handling strings (lib)
ii libevent-core-2.1-7:amd64 2.1.12-stable-1build3 amd64 Asynchronous event notification library (core)
-ii libexpat1:amd64 2.4.7-1ubuntu0.2 amd64 XML parsing C library - runtime library
-ii libexpat1-dev:amd64 2.4.7-1ubuntu0.2 amd64 XML parsing C library - development kit
+ii libexpat1:amd64 2.4.7-1ubuntu0.3 amd64 XML parsing C library - runtime library
+ii libexpat1-dev:amd64 2.4.7-1ubuntu0.3 amd64 XML parsing C library - development kit
ii libext2fs2:amd64 1.46.5-2ubuntu1.1 amd64 ext2/ext3/ext4 file system libraries
ii libfakeroot:amd64 1.28-1ubuntu1 amd64 tool for simulating superuser privileges - shared libraries
ii libfastjson4:amd64 0.99.9-1build2 amd64 fast json library for C
@@ -356,7 +356,6 @@
ii libgl1-amber-dri:amd64 21.3.9-0ubuntu1~22.04.1 amd64 free implementation of the OpenGL API -- Amber DRI modules
ii libgl1-mesa-dri:amd64 23.2.1-1ubuntu3.1~22.04.2 amd64 free implementation of the OpenGL API -- DRI modules
ii libglapi-mesa:amd64 23.2.1-1ubuntu3.1~22.04.2 amd64 free implementation of the GL API -- shared library
-ii libgles2:amd64 1.4.0-1 amd64 Vendor neutral GL dispatch library -- GLESv2 support
ii libglib2.0-0:amd64 2.72.4-0ubuntu2.2 amd64 GLib library of C routines
ii libglib2.0-bin 2.72.4-0ubuntu2.2 amd64 Programs for the GLib library
ii libglib2.0-data 2.72.4-0ubuntu2.2 all Common files for GLib library
@@ -386,8 +385,8 @@
ii libinih1:amd64 53-1ubuntu3 amd64 simple .INI file parser
ii libintl-perl 1.26-3build2 all Uniforum message translations system compatible i18n library
ii libintl-xs-perl 1.26-3build2 amd64 XS Uniforum message translations system compatible i18n library
-ii libip4tc2:amd64 1.8.7-1ubuntu5.1 amd64 netfilter libip4tc library
-ii libip6tc2:amd64 1.8.7-1ubuntu5.1 amd64 netfilter libip6tc library
+ii libip4tc2:amd64 1.8.7-1ubuntu5.2 amd64 netfilter libip4tc library
+ii libip6tc2:amd64 1.8.7-1ubuntu5.2 amd64 netfilter libip6tc library
ii libisc-export1105:amd64 1:9.11.19+dfsg-2.1ubuntu3 amd64 Exported ISC Shared Library
ii libisl23:amd64 0.24-2build1 amd64 manipulating sets and relations of integer points bounded by linear constraints
ii libisns0:amd64 0.101-0ubuntu2 amd64 Internet Storage Name Service - shared libraries
@@ -408,8 +407,8 @@
ii libkrb5support0:amd64 1.19.2-2ubuntu0.3 amd64 MIT Kerberos runtime libraries - Support library
ii libksba8:amd64 1.6.0-2ubuntu0.2 amd64 X.509 and CMS support library
ii liblcms2-2:amd64 2.12~rc1-2build2 amd64 Little CMS 2 color management library
-ii libldap-2.5-0:amd64 2.5.16+dfsg-0ubuntu0.22.04.2 amd64 OpenLDAP libraries
-ii libldap-common 2.5.16+dfsg-0ubuntu0.22.04.2 all OpenLDAP common files for libraries
+ii libldap-2.5-0:amd64 2.5.17+dfsg-0ubuntu0.22.04.1 amd64 OpenLDAP libraries
+ii libldap-common 2.5.17+dfsg-0ubuntu0.22.04.1 all OpenLDAP common files for libraries
ii libllvm15:amd64 1:15.0.7-0ubuntu0.22.04.3 amd64 Modular compiler and toolchain technologies, runtime library
ii liblmdb0:amd64 0.9.24-1build2 amd64 Lightning Memory-Mapped Database shared library
ii liblocale-gettext-perl 1.07-4build3 amd64 module using libc functions for internationalization in Perl
@@ -429,8 +428,8 @@
ii libmnl0:amd64 1.0.4-3build2 amd64 minimalistic Netlink communication library
ii libmodule-find-perl 0.15-1 all module to find and use installed Perl modules
ii libmodule-scandeps-perl 1.31-1 all module to recursively scan Perl code for dependencies
-ii libmount-dev:amd64 2.37.2-4ubuntu3 amd64 device mounting library - headers
-ii libmount1:amd64 2.37.2-4ubuntu3 amd64 device mounting library
+ii libmount-dev:amd64 2.37.2-4ubuntu3.4 amd64 device mounting library - headers
+ii libmount1:amd64 2.37.2-4ubuntu3.4 amd64 device mounting library
ii libmpc3:amd64 1.2.1-2build1 amd64 multiple precision complex floating-point library
ii libmpdec3:amd64 2.5.1-2build2 amd64 library for decimal floating point arithmetic (runtime library)
ii libmpfr6:amd64 4.1.0-3build3 amd64 multiple precision floating-point computation
@@ -447,8 +446,8 @@
ii libnghttp2-14:amd64 1.43.0-1ubuntu0.1 amd64 library implementing HTTP/2 protocol (shared library)
ii libnl-3-200:amd64 3.5.0-0.1 amd64 library for dealing with netlink sockets
ii libnl-genl-3-200:amd64 3.5.0-0.1 amd64 library for dealing with netlink sockets - generic netlink
-ii libnpp-12-3 12.2.3.2-1 amd64 NPP native runtime libraries
-ii libnpp-dev-12-3 12.2.3.2-1 amd64 NPP native dev links, headers
+ii libnpp-12-4 12.2.5.30-1 amd64 NPP native runtime libraries
+ii libnpp-dev-12-4 12.2.5.30-1 amd64 NPP native dev links, headers
ii libnpth0:amd64 1.6-3build2 amd64 replacement for GNU Pth using system threads
ii libnsl-dev:amd64 1.3.0-2build2 amd64 libnsl development files
ii libnsl2:amd64 1.3.0-2build2 amd64 Public client interface for NIS(YP) and NIS+
@@ -457,18 +456,20 @@
ii libnss3:amd64 2:3.68.2-0ubuntu1.2 amd64 Network Security Service libraries
ii libntfs-3g89 1:2021.8.22-3ubuntu1.2 amd64 read/write NTFS driver for FUSE (runtime library)
ii libnuma1:amd64 2.0.14-3ubuntu2 amd64 Libraries for controlling NUMA policy
-ii libnvidia-cfg1-545:amd64 545.23.08-0ubuntu1 amd64 NVIDIA binary OpenGL/GLX configuration library
-ii libnvidia-common-545 545.23.08-0ubuntu1 all Shared files used by the NVIDIA libraries
-ii libnvidia-compute-545:amd64 545.23.08-0ubuntu1 amd64 NVIDIA libcompute package
-ii libnvidia-decode-545:amd64 545.23.08-0ubuntu1 amd64 NVIDIA Video Decoding runtime libraries
-ii libnvidia-encode-545:amd64 545.23.08-0ubuntu1 amd64 NVENC Video Encoding runtime library
-ii libnvidia-extra-545:amd64 545.23.08-0ubuntu1 amd64 Extra libraries for the NVIDIA driver
-ii libnvidia-fbc1-545:amd64 545.23.08-0ubuntu1 amd64 NVIDIA OpenGL-based Framebuffer Capture runtime library
-ii libnvidia-gl-545:amd64 545.23.08-0ubuntu1 amd64 NVIDIA OpenGL/GLX/EGL/GLES GLVND libraries and Vulkan ICD
-ii libnvjitlink-12-3 12.3.101-1 amd64 NVJIT link native runtime libraries
-ii libnvjitlink-dev-12-3 12.3.101-1 amd64 NVJIT native dev links, headers
-ii libnvjpeg-12-3 12.3.0.81-1 amd64 NVJPEG native runtime libraries
-ii libnvjpeg-dev-12-3 12.3.0.81-1 amd64 NVJPEG native dev links, headers
+ii libnvfatbin-12-4 12.4.127-1 amd64 nvfatbin link native runtime libraries
+ii libnvfatbin-dev-12-4 12.4.127-1 amd64 nvfatbin native dev links, headers
+ii libnvidia-cfg1-550:amd64 550.54.15-0ubuntu1 amd64 NVIDIA binary OpenGL/GLX configuration library
+ii libnvidia-common-550 550.54.15-0ubuntu1 all Shared files used by the NVIDIA libraries
+ii libnvidia-compute-550:amd64 550.54.15-0ubuntu1 amd64 NVIDIA libcompute package
+ii libnvidia-decode-550:amd64 550.54.15-0ubuntu1 amd64 NVIDIA Video Decoding runtime libraries
+ii libnvidia-encode-550:amd64 550.54.15-0ubuntu1 amd64 NVENC Video Encoding runtime library
+ii libnvidia-extra-550:amd64 550.54.15-0ubuntu1 amd64 Extra libraries for the NVIDIA driver
+ii libnvidia-fbc1-550:amd64 550.54.15-0ubuntu1 amd64 NVIDIA OpenGL-based Framebuffer Capture runtime library
+ii libnvidia-gl-550:amd64 550.54.15-0ubuntu1 amd64 NVIDIA OpenGL/GLX/EGL/GLES GLVND libraries and Vulkan ICD
+ii libnvjitlink-12-4 12.4.127-1 amd64 NVJIT link native runtime libraries
+ii libnvjitlink-dev-12-4 12.4.127-1 amd64 NVJIT native dev links, headers
+ii libnvjpeg-12-4 12.3.1.117-1 amd64 NVJPEG native runtime libraries
+ii libnvjpeg-dev-12-4 12.3.1.117-1 amd64 NVJPEG native dev links, headers
ii libopengl0:amd64 1.4.0-1 amd64 Vendor neutral GL dispatch library -- OpenGL support
ii libopeniscsiusr 2.1.5-1ubuntu1 amd64 iSCSI userspace library
ii libp11-kit0:amd64 0.24.0-6build1 amd64 library for loading and coordinating access to PKCS#11 modules - runtime
@@ -545,7 +546,7 @@
ii libsqlite3-0:amd64 3.37.2-2ubuntu0.3 amd64 SQLite 3 shared library
ii libss2:amd64 1.46.5-2ubuntu1.1 amd64 command-line interface parsing library
ii libssh-4:amd64 0.9.6-2ubuntu0.22.04.3 amd64 tiny C SSH library (OpenSSL flavor)
-ii libssl3:amd64 3.0.2-0ubuntu1.14 amd64 Secure Sockets Layer toolkit - shared libraries
+ii libssl3:amd64 3.0.2-0ubuntu1.15 amd64 Secure Sockets Layer toolkit - shared libraries
ii libstdc++-11-dev:amd64 11.4.0-1ubuntu1~22.04 amd64 GNU Standard C++ Library v3 (development files)
ii libstdc++6:amd64 12.3.0-1ubuntu1~22.04 amd64 GNU Standard C++ Library v3
ii libstemmer0d:amd64 2.2.0-1build1 amd64 Snowball stemming algorithms for use in Information Retrieval
@@ -558,7 +559,7 @@
ii libtext-wrapi18n-perl 0.06-9 all internationalized substitute of Text::Wrap
ii libthai-data 0.1.29-1build1 all Data files for Thai language support library
ii libthai0:amd64 0.1.29-1build1 amd64 Thai language support library
-ii libtiff5:amd64 4.3.0-6ubuntu0.7 amd64 Tag Image File Format (TIFF) library
+ii libtiff5:amd64 4.3.0-6ubuntu0.8 amd64 Tag Image File Format (TIFF) library
ii libtinfo5:amd64 6.3-2ubuntu0.1 amd64 shared low-level terminfo library (legacy version)
ii libtinfo6:amd64 6.3-2ubuntu0.1 amd64 shared low-level terminfo library for terminal handling
ii libtirpc-common 1.3.2-2ubuntu0.1 all transport-independent RPC library - common files
@@ -575,8 +576,8 @@
ii liburcu8:amd64 0.13.1-1 amd64 userspace RCU (read-copy-update) library
ii libusb-1.0-0:amd64 2:1.0.25-1ubuntu2 amd64 userspace USB programming library
ii libutempter0:amd64 1.2.1-2build2 amd64 privileged helper for utmp/wtmp updates (runtime)
-ii libuuid1:amd64 2.37.2-4ubuntu3 amd64 Universally Unique ID library
-ii libuv1:amd64 1.43.0-1 amd64 asynchronous event notification library - runtime library
+ii libuuid1:amd64 2.37.2-4ubuntu3.4 amd64 Universally Unique ID library
+ii libuv1:amd64 1.43.0-1ubuntu0.1 amd64 asynchronous event notification library - runtime library
ii libvdpau1:amd64 1.4-3build2 amd64 Video Decode and Presentation API for Unix (libraries)
ii libwayland-client0:amd64 1.20.0-1ubuntu0.1 amd64 wayland compositor infrastructure - client library
ii libwayland-cursor0:amd64 1.20.0-1ubuntu0.1 amd64 wayland compositor infrastructure - cursor library
@@ -622,20 +623,20 @@
ii libxkbcommon-x11-0:amd64 1.4.0-1 amd64 library to create keymaps with the XKB X11 protocol
ii libxkbcommon0:amd64 1.4.0-1 amd64 library interface to the XKB compiler - shared library
ii libxkbfile1:amd64 1:1.1.0-1build3 amd64 X11 keyboard file manipulation library
-ii libxml2:amd64 2.9.13+dfsg-1ubuntu0.3 amd64 GNOME XML library
+ii libxml2:amd64 2.9.13+dfsg-1ubuntu0.4 amd64 GNOME XML library
ii libxmlb2:amd64 0.3.6-2build1 amd64 Binary XML library
ii libxmlsec1:amd64 1.2.33-1build2 amd64 XML security library
ii libxmlsec1-openssl:amd64 1.2.33-1build2 amd64 Openssl engine for the XML security library
ii libxmu6:amd64 2:1.1.3-3 amd64 X11 miscellaneous utility library
ii libxmuu1:amd64 2:1.1.3-3 amd64 X11 miscellaneous micro-utility library
-ii libxnvctrl0:amd64 545.23.08-0ubuntu1 amd64 NV-CONTROL X extension (runtime library)
+ii libxnvctrl0:amd64 550.54.15-0ubuntu1 amd64 NV-CONTROL X extension (runtime library)
ii libxpm4:amd64 1:3.5.12-1ubuntu0.22.04.2 amd64 X11 pixmap library
ii libxrandr2:amd64 2:1.5.2-1build1 amd64 X11 RandR extension library
ii libxrender1:amd64 1:0.9.10-1build4 amd64 X Rendering Extension client library
ii libxshmfence1:amd64 1.3-1build4 amd64 X shared memory fences - shared library
ii libxslt1.1:amd64 1.1.34-4ubuntu0.22.04.1 amd64 XSLT 1.0 processing library - runtime library
ii libxt6:amd64 1:1.2.1-1 amd64 X11 toolkit intrinsics library
-ii libxtables12:amd64 1.8.7-1ubuntu5.1 amd64 netfilter xtables library
+ii libxtables12:amd64 1.8.7-1ubuntu5.2 amd64 netfilter xtables library
ii libxtst6:amd64 2:1.2.3-1build4 amd64 X11 Testing -- Record extension library
ii libxv1:amd64 2:1.0.11-1build2 amd64 X11 Video extension library
ii libxxf86dga1:amd64 2:1.1.5-0ubuntu3 amd64 X11 Direct Graphics Access extension library
@@ -644,16 +645,20 @@
ii libyaml-0-2:amd64 0.2.2-1build2 amd64 Fast YAML 1.1 parser and emitter library
ii libzstd1:amd64 1.4.8+dfsg-3build1 amd64 fast lossless compression algorithm
ii linux-base 4.5ubuntu9 all Linux image base package
-ii linux-gcp 6.5.0.1013.13~22.04.1 amd64 Complete Google Cloud Platform (GCP) Linux kernel and headers
-ii linux-gcp-6.5-headers-6.5.0-1013 6.5.0-1013.13~22.04.1 amd64 Header files related to Linux kernel version 6.5.0
-ii linux-headers-6.5.0-1013-gcp 6.5.0-1013.13~22.04.1 amd64 Linux kernel headers for version 6.5.0 on 64 bit x86 SMP
-ii linux-headers-gcp 6.5.0.1013.13~22.04.1 amd64 Google Cloud Platform (GCP) Linux kernel headers
-ii linux-image-6.5.0-1013-gcp 6.5.0-1013.13~22.04.1 amd64 Signed kernel image gcp
-ii linux-image-gcp 6.5.0.1013.13~22.04.1 amd64 Google Cloud Platform (GCP) Linux kernel image
-ii linux-libc-dev:amd64 5.15.0-94.104 amd64 Linux Kernel Headers for development
-ii linux-modules-6.5.0-1013-gcp 6.5.0-1013.13~22.04.1 amd64 Linux kernel extra modules for version 6.5.0 on 64 bit x86 SMP
+ii linux-gcp 6.5.0.1018.18~22.04.1 amd64 Complete Google Cloud Platform (GCP) Linux kernel and headers
+ii linux-gcp-6.5-headers-6.5.0-1016 6.5.0-1016.16~22.04.1 amd64 Header files related to Linux kernel version 6.5.0
+ii linux-gcp-6.5-headers-6.5.0-1018 6.5.0-1018.18~22.04.1 amd64 Header files related to Linux kernel version 6.5.0
+ii linux-headers-6.5.0-1016-gcp 6.5.0-1016.16~22.04.1 amd64 Linux kernel headers for version 6.5.0 on 64 bit x86 SMP
+ii linux-headers-6.5.0-1018-gcp 6.5.0-1018.18~22.04.1 amd64 Linux kernel headers for version 6.5.0 on 64 bit x86 SMP
+ii linux-headers-gcp 6.5.0.1018.18~22.04.1 amd64 Google Cloud Platform (GCP) Linux kernel headers
+ii linux-image-6.5.0-1016-gcp 6.5.0-1016.16~22.04.1 amd64 Signed kernel image gcp
+ii linux-image-6.5.0-1018-gcp 6.5.0-1018.18~22.04.1 amd64 Signed kernel image gcp
+ii linux-image-gcp 6.5.0.1018.18~22.04.1 amd64 Google Cloud Platform (GCP) Linux kernel image
+ii linux-libc-dev:amd64 5.15.0-105.115 amd64 Linux Kernel Headers for development
+ii linux-modules-6.5.0-1016-gcp 6.5.0-1016.16~22.04.1 amd64 Linux kernel extra modules for version 6.5.0 on 64 bit x86 SMP
+ii linux-modules-6.5.0-1018-gcp 6.5.0-1018.18~22.04.1 amd64 Linux kernel extra modules for version 6.5.0 on 64 bit x86 SMP
ii locales 2.35-0ubuntu3.6 all GNU C Library: National Language (locale) data [support]
-ii login 1:4.8.1-2ubuntu2.1 amd64 system login tools
+ii login 1:4.8.1-2ubuntu2.2 amd64 system login tools
ii logrotate 3.19.0-1ubuntu1.1 amd64 Log rotation utility
ii logsave 1.46.5-2ubuntu1.1 amd64 save the output of a command in a log file
ii lsb-base 11.1.0ubuntu4 all Linux Standard Base init script functionality
@@ -674,7 +679,7 @@
ii mercurial-common 6.1.1-1ubuntu1 all easy-to-use, scalable distributed version control system (common files)
ii mesa-vdpau-drivers:amd64 23.2.1-1ubuntu3.1~22.04.2 amd64 Mesa VDPAU video acceleration drivers
ii mokutil 0.6.0-2~22.04.2 amd64 tools for manipulating machine owner keys
-ii motd-news-config 12ubuntu4.5 all Configuration for motd-news shipped in base-files
+ii motd-news-config 12ubuntu4.6 all Configuration for motd-news shipped in base-files
ii mount 2.37.2-4ubuntu3 amd64 tools for mounting and manipulating filesystems
ii mtr-tiny 0.95-1 amd64 Full screen ncurses traceroute tool
ii multipath-tools 0.8.8-1ubuntu1.22.04.4 amd64 maintain multipath block device access
@@ -690,34 +695,34 @@
ii netplan.io 0.106.1-7ubuntu0.22.04.2 amd64 YAML network configuration abstraction for various backends
ii networkd-dispatcher 2.1-2ubuntu0.22.04.2 all Dispatcher service for systemd-networkd connection status changes
ii nftables 1.0.2-1ubuntu3 amd64 Program to control packet filtering rules by Netfilter project
-ii nsight-compute-2023.3.1 2023.3.1.1-1 amd64 NVIDIA Nsight Compute
-ii nsight-systems-2023.3.3 2023.3.3.42-233333266658v0 amd64 Nsight Systems is a statistical sampling profiler with tracing features.
+ii nsight-compute-2024.1.1 2024.1.1.4-1 amd64 NVIDIA Nsight Compute
+ii nsight-systems-2023.4.4 2023.4.4.54-234433681190v0 amd64 Nsight Systems is a statistical sampling profiler with tracing features.
ii ntfs-3g 1:2021.8.22-3ubuntu1.2 amd64 read/write NTFS driver for FUSE
-ii nvidia-compute-utils-545 545.23.08-0ubuntu1 amd64 NVIDIA compute utilities
-ii nvidia-dkms-545 545.23.08-0ubuntu1 amd64 NVIDIA DKMS package
-ii nvidia-driver-545 545.23.08-0ubuntu1 amd64 NVIDIA driver metapackage
-ii nvidia-kernel-common-545 545.23.08-0ubuntu1 amd64 Shared files used with the kernel module
-ii nvidia-kernel-source-545 545.23.08-0ubuntu1 amd64 NVIDIA kernel source package
-ii nvidia-modprobe 545.23.08-0ubuntu1 amd64 Load the NVIDIA kernel driver and create device files
+ii nvidia-compute-utils-550 550.54.15-0ubuntu1 amd64 NVIDIA compute utilities
+ii nvidia-dkms-550 550.54.15-0ubuntu1 amd64 NVIDIA DKMS package
+ii nvidia-driver-550 550.54.15-0ubuntu1 amd64 NVIDIA driver metapackage
+ii nvidia-firmware-550-550.54.15 550.54.15-0ubuntu1 amd64 Firmware files used by the kernel module
+ii nvidia-kernel-common-550 550.54.15-0ubuntu1 amd64 Shared files used with the kernel module
+ii nvidia-kernel-source-550 550.54.15-0ubuntu1 amd64 NVIDIA kernel source package
ii nvidia-prime 0.8.17.1 all Tools to enable NVIDIA's Prime
-ii nvidia-settings 545.23.08-0ubuntu1 amd64 Tool for configuring the NVIDIA graphics driver
-ii nvidia-utils-545 545.23.08-0ubuntu1 amd64 NVIDIA driver support binaries
+ii nvidia-settings 550.54.15-0ubuntu1 amd64 Tool for configuring the NVIDIA graphics driver
+ii nvidia-utils-550 550.54.15-0ubuntu1 amd64 NVIDIA driver support binaries
ii nvme-cli 1.16-3ubuntu0.1 amd64 NVMe management tool
ii open-iscsi 2.1.5-1ubuntu1 amd64 iSCSI initiator tools
-ii open-vm-tools 2:12.1.5-3~ubuntu0.22.04.4 amd64 Open VMware Tools for virtual machines hosted on VMware (CLI)
-ii openjdk-11-jre:amd64 11.0.21+9-0ubuntu1~22.04 amd64 OpenJDK Java runtime, using Hotspot JIT
-ii openjdk-11-jre-headless:amd64 11.0.21+9-0ubuntu1~22.04 amd64 OpenJDK Java runtime, using Hotspot JIT (headless)
+ii open-vm-tools 2:12.3.5-3~ubuntu0.22.04.1 amd64 Open VMware Tools for virtual machines hosted on VMware (CLI)
+ii openjdk-11-jre:amd64 11.0.22+7-0ubuntu2~22.04.1 amd64 OpenJDK Java runtime, using Hotspot JIT
+ii openjdk-11-jre-headless:amd64 11.0.22+7-0ubuntu2~22.04.1 amd64 OpenJDK Java runtime, using Hotspot JIT (headless)
ii openssh-client 1:8.9p1-3ubuntu0.6 amd64 secure shell (SSH) client, for secure access to remote machines
ii openssh-server 1:8.9p1-3ubuntu0.6 amd64 secure shell (SSH) server, for secure access from remote machines
ii openssh-sftp-server 1:8.9p1-3ubuntu0.6 amd64 secure shell (SSH) sftp server module, for SFTP access from remote machines
-ii openssl 3.0.2-0ubuntu1.14 amd64 Secure Sockets Layer toolkit - cryptographic utility
+ii openssl 3.0.2-0ubuntu1.15 amd64 Secure Sockets Layer toolkit - cryptographic utility
ii os-prober 1.79ubuntu2 amd64 utility to detect other OSes on a set of drives
ii overlayroot 0.47ubuntu1 all use an overlayfs on top of a read-only root filesystem
ii packagekit 1.2.5-2ubuntu2 amd64 Provides a package management service
ii packagekit-tools 1.2.5-2ubuntu2 amd64 Provides PackageKit command-line tools
ii parallel 20210822+ds-2 all build and execute command lines from standard input in parallel
ii parted 3.4-2build1 amd64 disk partition manipulator
-ii passwd 1:4.8.1-2ubuntu2.1 amd64 change and administer password and group data
+ii passwd 1:4.8.1-2ubuntu2.2 amd64 change and administer password and group data
ii pastebinit 1.5.1-1ubuntu1 all command-line pastebin client
ii patch 2.7.6-7build2 amd64 Apply a diff file to an original
ii pci.ids 0.0~2022.01.22-1 all PCI ID Repository
@@ -737,11 +742,11 @@
ii procps 2:3.3.17-6ubuntu2.1 amd64 /proc file system utilities
ii psmisc 23.4-2build3 amd64 utilities that use the proc file system
ii publicsuffix 20211207.1025-1 all accurate, machine-readable list of domain name suffixes
-ii python-apt-common 2.4.0ubuntu2 all Python interface to libapt-pkg (locales)
+ii python-apt-common 2.4.0ubuntu3 all Python interface to libapt-pkg (locales)
ii python-babel-localedata 2.8.0+dfsg.1-7 all tools for internationalizing Python applications - locale data files
ii python3 3.10.6-1~22.04 amd64 interactive high-level object-oriented language (default python3 version)
ii python3-apport 2.20.11-0ubuntu82.5 all Python 3 library for Apport crash report handling
-ii python3-apt 2.4.0ubuntu2 amd64 Python 3 interface to libapt-pkg
+ii python3-apt 2.4.0ubuntu3 amd64 Python 3 interface to libapt-pkg
ii python3-attr 21.2.0-1 all Attributes without boilerplate (Python 3)
ii python3-automat 20.2.0-1 all Self-service finite-state machines for the programmer on the go
ii python3-babel 2.8.0+dfsg.1-7 all tools for internationalizing Python applications - Python 3.x
@@ -755,14 +760,14 @@
ii python3-commandnotfound 22.04.0 all Python 3 bindings for command-not-found.
ii python3-configobj 5.0.6-5 all simple but powerful config file reader and writer for Python 3
ii python3-constantly 15.1.0-2 all Symbolic constants in Python
-ii python3-cryptography 3.4.8-1ubuntu2.1 amd64 Python library exposing cryptographic recipes and primitives (Python 3)
+ii python3-cryptography 3.4.8-1ubuntu2.2 amd64 Python library exposing cryptographic recipes and primitives (Python 3)
ii python3-dbus 1.2.18-3build1 amd64 simple interprocess messaging system (Python 3 interface)
ii python3-debconf 1.5.79ubuntu1 all interact with debconf from Python 3
ii python3-debian 0.1.43ubuntu1.1 all Python 3 modules to work with Debian-related data formats
ii python3-dev 3.10.6-1~22.04 amd64 header files and a static library for Python (default)
ii python3-distro 1.7.0-1 all Linux OS platform information API
ii python3-distro-info 1.1ubuntu0.2 all information about distributions' releases (Python 3 module)
-ii python3-distupgrade 1:22.04.17 all manage release upgrades
+ii python3-distupgrade 1:22.04.19 all manage release upgrades
ii python3-distutils 3.10.8-1~22.04 all distutils package for Python 3.x
ii python3-gdbm:amd64 3.10.8-1~22.04 amd64 GNU dbm database support for Python 3.x
ii python3-gi 3.42.1-0ubuntu1 amd64 Python 3 bindings for gobject-introspection libraries
@@ -811,7 +816,7 @@
ii python3-systemd 234-3ubuntu2 amd64 Python 3 bindings for systemd
ii python3-twisted 22.1.0-2ubuntu2.4 all Event-based framework for internet applications
ii python3-tz 2022.1-1ubuntu0.22.04.1 all Python3 version of the Olson timezone database
-ii python3-update-manager 1:22.04.18 all python 3.x module for update-manager
+ii python3-update-manager 1:22.04.19 all python 3.x module for update-manager
ii python3-urllib3 1.26.5-1~exp1ubuntu0.1 all HTTP library with thread-safe connection pooling for Python3
ii python3-wadllib 1.3.6-1 all Python 3 library for navigating WADL files
ii python3-wheel 0.37.1-2ubuntu0.22.04.1 all built-package format for Python
@@ -857,19 +862,20 @@
ii tar 1.34+dfsg-1ubuntu0.1.22.04.2 amd64 GNU version of the tar archiving utility
ii tcl 8.6.11+1build2 amd64 Tool Command Language (default version) - shell
ii tcl8.6 8.6.12+dfsg-1build1 amd64 Tcl (the Tool Command Language) v8.6 - shell
-ii tcpdump 4.99.1-3ubuntu0.1 amd64 command-line network traffic analyzer
+ii tcpdump 4.99.1-3ubuntu0.2 amd64 command-line network traffic analyzer
ii telnet 0.17-44build1 amd64 basic telnet client
ii thin-provisioning-tools 0.9.0-2ubuntu1 amd64 Tools for handling thinly provisioned device-mapper meta-data
ii time 1.9-0.1build2 amd64 GNU time program for measuring CPU resource usage
ii tmux 3.2a-4ubuntu0.2 amd64 terminal multiplexer
ii tnftp 20210827-4build1 amd64 enhanced ftp client
-ii tzdata 2023d-0ubuntu0.22.04 all time zone and daylight-saving time data
-ii ubuntu-advantage-tools 30~22.04 amd64 management tools for Ubuntu Pro
+ii tzdata 2024a-0ubuntu0.22.04 all time zone and daylight-saving time data
+ii ubuntu-advantage-tools 31.2~22.04 all transitional dummy package for ubuntu-pro-client
ii ubuntu-keyring 2021.03.26 all GnuPG keys of the Ubuntu archive
ii ubuntu-minimal 1.481.1 amd64 Minimal core of Ubuntu
ii ubuntu-mono 20.10-0ubuntu2 all Ubuntu Mono Icon theme
-ii ubuntu-pro-client-l10n 30~22.04 amd64 Translations for Ubuntu Pro Client
-ii ubuntu-release-upgrader-core 1:22.04.17 all manage release upgrades
+ii ubuntu-pro-client 31.2~22.04 amd64 Management tools for Ubuntu Pro
+ii ubuntu-pro-client-l10n 31.2~22.04 amd64 Translations for Ubuntu Pro Client
+ii ubuntu-release-upgrader-core 1:22.04.19 all manage release upgrades
ii ubuntu-server 1.481.1 amd64 The Ubuntu Server system
ii ubuntu-standard 1.481.1 amd64 The Ubuntu standard system
ii ucf 3.0043 all Update Configuration File(s): preserve user changes to config files
@@ -877,7 +883,7 @@
ii ufw 0.36.1-4ubuntu0.1 all program for managing a Netfilter firewall
ii uidmap 1:4.8.1-2ubuntu2.2 amd64 programs to help use subuids
ii unattended-upgrades 2.8ubuntu1 all automatic installation of security upgrades
-ii update-manager-core 1:22.04.18 all manage release upgrades
+ii update-manager-core 1:22.04.19 all manage release upgrades
ii update-notifier-common 3.192.54.6 all Files shared between update-notifier and other packages
ii usb-modeswitch 2.6.1-3ubuntu2 amd64 mode switching tool for controlling "flip flop" USB devices
ii usb-modeswitch-data 20191128-4 all mode switching data for usb-modeswitch
@@ -885,13 +891,13 @@
ii usbutils 1:014-1build1 amd64 Linux USB utilities
ii usrmerge 25ubuntu2 all Convert the system to the merged /usr directories scheme
ii util-linux 2.37.2-4ubuntu3 amd64 miscellaneous system utilities
-ii uuid-dev:amd64 2.37.2-4ubuntu3 amd64 Universally Unique ID library - headers and static libraries
+ii uuid-dev:amd64 2.37.2-4ubuntu3.4 amd64 Universally Unique ID library - headers and static libraries
ii uuid-runtime 2.37.2-4ubuntu3 amd64 runtime components for the Universally Unique ID library
ii vdpau-driver-all:amd64 1.4-3build2 amd64 Video Decode and Presentation API for Unix (driver metapackage)
-ii vim 2:8.2.3995-1ubuntu2.15 amd64 Vi IMproved - enhanced vi editor
-ii vim-common 2:8.2.3995-1ubuntu2.15 all Vi IMproved - Common files
-ii vim-runtime 2:8.2.3995-1ubuntu2.15 all Vi IMproved - Runtime files
-ii vim-tiny 2:8.2.3995-1ubuntu2.15 amd64 Vi IMproved - enhanced vi editor - compact version
+ii vim 2:8.2.3995-1ubuntu2.16 amd64 Vi IMproved - enhanced vi editor
+ii vim-common 2:8.2.3995-1ubuntu2.16 all Vi IMproved - Common files
+ii vim-runtime 2:8.2.3995-1ubuntu2.16 all Vi IMproved - Runtime files
+ii vim-tiny 2:8.2.3995-1ubuntu2.16 amd64 Vi IMproved - enhanced vi editor - compact version
ii wget 1.21.2-2ubuntu1 amd64 retrieves files from the web
ii whiptail 0.52.21-5ubuntu2 amd64 Displays user-friendly dialog boxes from shell scripts
ii x11-common 1:7.7+23ubuntu2 all X Window System (X.Org) infrastructure
@@ -905,10 +911,10 @@
ii xfonts-utils 1:7.7+6build2 amd64 X Window System font utility programs
ii xfsprogs 5.13.0-1ubuntu2 amd64 Utilities for managing the XFS filesystem
ii xkb-data 2.33-1 all X Keyboard Extension (XKB) configuration data
-ii xserver-common 2:21.1.4-2ubuntu1.7~22.04.8 all common files used by various X servers
-ii xserver-xorg-core 2:21.1.4-2ubuntu1.7~22.04.8 amd64 Xorg X server - core server
-ii xserver-xorg-video-nvidia-545 545.23.08-0ubuntu1 amd64 NVIDIA binary Xorg driver
-ii xxd 2:8.2.3995-1ubuntu2.15 amd64 tool to make (or reverse) a hex dump
+ii xserver-common 2:21.1.4-2ubuntu1.7~22.04.10 all common files used by various X servers
+ii xserver-xorg-core 2:21.1.4-2ubuntu1.7~22.04.10 amd64 Xorg X server - core server
+ii xserver-xorg-video-nvidia-550 550.54.15-0ubuntu1 amd64 NVIDIA binary Xorg driver
+ii xxd 2:8.2.3995-1ubuntu2.16 amd64 tool to make (or reverse) a hex dump
ii xz-utils 5.2.5-2ubuntu1 amd64 XZ-format compression utilities
ii zerofree 1.1.1-1build3 amd64 zero free blocks from ext2, ext3 and ext4 file-systems
ii zlib1g:amd64 1:1.2.11.dfsg-2ubuntu9.2 amd64 compression library - runtimeDoing some more research on this. https://github.com/systemd/systemd/issues/21123 is one of the first things that comes up. The solution that most people use is to switch to dnsmasq. @aerickson - is that something we can try here? (I can work up a monopacker PR for it - just want to check that it's a viable thing to try before doing so.)
aerickson
commented
1 month ago Nice find. Yeah, definitely.
If it's expected that it can fail we should add retries to all our steps:
Here is an example task: https://firefox-ci-tc.services.mozilla.com/tasks/JdDA-zYDQnG4166Zbsqq6w