search

rickysarraf / apt-offline

Offline APT Package Manager
http://rickysarraf.github.io/apt-offline/
GNU General Public License v3.0
215 stars 37 forks source link

`apt-offline set` sometimes fails to parse command-line when using `--` #204

Closed mtalexan closed 1 year ago

mtalexan commented 1 year ago

Running apt-offline set --install-packages {package names} -- wanted.asc silently fails to include some packages in the output, and fails with an explict error only when those packages that were silently skipped are the only packages listed, despite all the packages being available in the apt database.

Problem 1: A package that's available in the apt database isn't working with apt-offline
Problem 2: Packages that fail in this way fail silently when included in a list of packages

Steps to Reproduce

Ubuntu 22.04.2, with apt-offline version 1.8.4 installed from the repos. For reproduction purposes, a single online machine can be used to try to generate the signature.

The following packages have been found to fail (so far): emacs, emacs-gtk

  1. sudo apt-get update
  2. Confirm all database updates succeed
  3. sudo apt-cache policy emacs
  4. Confirm a version is listed as a Candidate
  5. sudo apt-cache show emacs
  6. Confirm a version is listed and a Filename is included.
  7. sudo apt-get install --dry-run parallel | grep "^Inst " to see what would be needed by an install
  8. sudo apt-offline set --apt-backend apt-get --install-packages emacs -- wanted.asc
  9. Notice Error
Step 1 Output: sudo apt-get update

``` Hit:1 http://us.archive.ubuntu.com/ubuntu jammy InRelease Hit:2 http://us.archive.ubuntu.com/ubuntu jammy-updates InRelease Hit:3 http://us.archive.ubuntu.com/ubuntu jammy-backports InRelease Hit:4 http://packages.microsoft.com/repos/code stable InRelease Hit:5 http://security.ubuntu.com/ubuntu jammy-security InRelease Reading package lists... Done ```

Step 3 Output: sudo apt-cache policy emacs

``` emacs: Installed: (none) Candidate: 1:27.1+1-3ubuntu5 Version table: 1:27.1+1-3ubuntu5 500 500 http://us.archive.ubuntu.com/ubuntu jammy/universe amd64 Packages 500 http://us.archive.ubuntu.com/ubuntu jammy/universe i386 Packages ```

Step 5 Output: sudo apt-cache show emacs

``` Package: emacs Architecture: all Version: 1:27.1+1-3ubuntu5 Priority: optional Section: universe/editors Origin: Ubuntu Maintainer: Ubuntu Developers Original-Maintainer: Rob Browning Bugs: https://bugs.launchpad.net/ubuntu/+filebug Installed-Size: 77 Depends: emacs-gtk (>= 1:27.1) | emacs-lucid (>= 1:27.1) | emacs-nox (>= 1:27.1) Filename: pool/universe/e/emacs/emacs_27.1+1-3ubuntu5_all.deb Size: 13284 MD5sum: 72cdaf6570f1d8a0a2973eeb9f517872 SHA1: 507b22697d96cbfc001020e085f0c4f804a6dcf1 SHA256: 1e186c5436d9d83de73d6deabc4f63229f373ccaa30fa10a22b1e39f1f7036c6 SHA512: 415d6bf24bc031f3fc7b2d6dd1009bfb7959ce92eef843ecbbf0b4a1cdd44a3d8678a5902323798982b3b2fe30aaf8d66cf8ef556ee31082a2fc58d0459d18d1 Homepage: https://www.gnu.org/software/emacs/ Description-en: GNU Emacs editor (metapackage) GNU Emacs is the extensible self-documenting text editor. This is a metapackage that will always depend on the latest recommended Emacs variant (currently emacs-gtk). Description-md5: f991b22b712b84422dc583d29a7b2171 ```

Step 7 Output: sudo apt-get install --dry-run emacs | grep "^Inst "

``` Inst emacs-common (1:27.1+1-3ubuntu5 Ubuntu:22.04/jammy [all]) Inst emacs-bin-common (1:27.1+1-3ubuntu5 Ubuntu:22.04/jammy [amd64]) Inst m17n-db (1.8.0-3 Ubuntu:22.04/jammy [all]) Inst libotf1 (0.9.16-3build1 Ubuntu:22.04/jammy [amd64]) Inst libm17n-0 (1.8.0-4 Ubuntu:22.04/jammy [amd64]) Inst emacs-gtk (1:27.1+1-3ubuntu5 Ubuntu:22.04/jammy [amd64]) Inst emacs (1:27.1+1-3ubuntu5 Ubuntu:22.04/jammy [all]) Inst emacs-el (1:27.1+1-3ubuntu5 Ubuntu:22.04/jammy [all]) ```

Step 8 Output: sudo apt-offline set --apt-backend apt-get --install-packages emacs -- wanted.asc

``` Gathering installation details for package ['emacs'] ERROR: Generated signature file wanted.asc is of 0 bytes ERROR: This is usually the case when the underneath apt system has no payload to download ```

Steps to Reproduce Hidden Error

  1. sudo apt-get update
  2. Confirm all database updates succeed
  3. sudo apt-cache policy parallel
  4. Confirm a version is listed as a Candidate
  5. sudo apt-cache policy emacs
  6. Confirm a version is listed as a Candidate
  7. sudo apt-cache show parallel
  8. Confirm a version is listed and a Filename is included.
  9. sudo apt-cache show emacs
  10. Confirm a version is listed and a Filename is included.
  11. sudo apt-get install --dry-run parallel | grep "^Inst " to see what would be needed by an install
  12. sudo apt-offline set --apt-backend apt-get --install-packages parallel -- wanted.asc
  13. Confirm this step succeeds
  14. cat wanted.asc confirm the contents include everything the dry-run would have installed
  15. sudo apt-get install --dry-run emacs | grep "^Inst " to see what would be needed by an install
  16. sudo apt-offline set --apt-backend apt-get --install-packages emacs -- wanted.asc
  17. Notice this step fails
  18. sudo apt-offline set --apt-backend apt-get --install-packages emacs parallel -- wanted.asc
  19. Notice this step succeeds
  20. cat wanted.asc see that only parallel and its dependencies are included
Step 3 Output: sudo apt-cache policy parallel

``` parallel: Installed: (none) Candidate: 20210822+ds-2 Version table: 20210822+ds-2 500 500 http://us.archive.ubuntu.com/ubuntu jammy/universe amd64 Packages 500 http://us.archive.ubuntu.com/ubuntu jammy/universe i386 Packages ```

Step 7 Output: sudo apt-cache show parallel

``` Package: parallel Architecture: all Version: 20210822+ds-2 Priority: extra Section: universe/utils Origin: Ubuntu Maintainer: Ubuntu Developers Original-Maintainer: Debian Med Packaging Team Bugs: https://bugs.launchpad.net/ubuntu/+filebug Installed-Size: 2943 Depends: procps, sysstat, perl:any Suggests: ash, csh, fish, ksh, tcsh, zsh Filename: pool/universe/p/parallel/parallel_20210822+ds-2_all.deb Size: 1946526 MD5sum: 1670f08bf20379ba4d00f61614e7a40e SHA1: 7ffd10d06d2f32c1f6e340797d0627169d733b87 SHA256: 97cfb71545c60e510d3b8b854cbe7e2b905bd1bf3656b79ea260038237bbcbb1 SHA512: f272766263d8f28914bf858e5edd5dab3cc4d8bc0ea5b58d5ef3ab66310585f2455d85691ceaa8299b50b12afc824df6570ab0f22b48b544f16dea66bf9cd0ea Homepage: https://www.gnu.org/software/parallel/ Description-en: build and execute command lines from standard input in parallel GNU Parallel is a shell tool for executing jobs in parallel using one or more machines. A job is typically a single command or a small script that has to be run for each of the lines in the input. The typical input is a list of files, a list of hosts, a list of users, or a list of tables. . If you use xargs today you will find GNU Parallel very easy to use. If you write loops in shell, you will find GNU Parallel may be able to replace most of the loops and make them run faster by running jobs in parallel. If you use ppss or pexec you will find GNU Parallel will often make the command easier to read. . GNU Parallel also makes sure output from the commands is the same output as you would get had you run the commands sequentially. This makes it possible to use output from GNU Parallel as input for other programs. Description-md5: 8a851a81a172dd16a4bcbeb1f8f3c4ab ```

Step 11 Output: sudo apt-get install --dry-run parallel | grep "^Inst "

``` Inst sysstat (12.5.2-2ubuntu0.1 Ubuntu:22.04/jammy-updates, Ubuntu:22.04/jammy-security [amd64]) Inst parallel (20210822+ds-2 Ubuntu:22.04/jammy [all]) ```

Step 12 Output: sudo apt-offline set --apt-backend apt-get --install-packages parallel -- wanted.asc

``` Gathering installation details for package ['parallel'] ```

Step 14 Output: cat wanted.asc

``` 'http://us.archive.ubuntu.com/ubuntu/pool/main/s/sysstat/sysstat_12.5.2-2ubuntu0.1_amd64.deb' sysstat_12.5.2-2ubuntu0.1_amd64.deb 487044 MD5Sum:68f0f873cb63af7c4a1b63cb80027dd1 'http://us.archive.ubuntu.com/ubuntu/pool/universe/p/parallel/parallel_20210822%2bds-2_all.deb' parallel_20210822+ds-2_all.deb 1946526 MD5Sum:1670f08bf20379ba4d00f61614e7a40e ```

Step 18 Output: sudo apt-offline set --apt-backend apt-get --install-packages emacs parallel -- wanted.asc

``` Gathering installation details for package ['emacs', 'parallel'] ```

Step 20 Output: cat wanted.asc

``` 'http://us.archive.ubuntu.com/ubuntu/pool/main/s/sysstat/sysstat_12.5.2-2ubuntu0.1_amd64.deb' sysstat_12.5.2-2ubuntu0.1_amd64.deb 487044 MD5Sum:68f0f873cb63af7c4a1b63cb80027dd1 'http://us.archive.ubuntu.com/ubuntu/pool/universe/p/parallel/parallel_20210822%2bds-2_all.deb' parallel_20210822+ds-2_all.deb 1946526 MD5Sum:1670f08bf20379ba4d00f61614e7a40e ```

Other Notes

The emacs package is a group-package that has options of emacs-gtk, emacs-nox, or emacs-lucid. However the emacs-gtk package, which is the default selection for the group, produces the same errors/results even though it's a regular package. Neither the emacs-nox nor the emacs-lucid have this problem though.

rickysarraf commented 1 year ago

What exactly is a group package ? Do you mean a meta package ?

I tried with your example above and I have:

rrs@priyasi:~/.../apt-offline (master)$ sudo apt-offline set /tmp/emacs.uris --install-packages emacs
[sudo] password for rrs: 
Gathering installation details for package ['emacs']
20:03 ♒ ॐ ♅ ♄ ⛢     ☺ 😄    

rrs@priyasi:~/.../apt-offline (master)$ cat /tmp/emacs.uris 
'http://deb.debian.org/debian/pool/main/e/emacs/emacs-el_28.2%2b1-10_all.deb' emacs-el_1%3a28.2+1-10_all.deb 16872928 MD5Sum:4220a0f76d24cb81971d03fd6df4d965
'http://deb.debian.org/debian/pool/main/e/emacs/emacs-common_28.2%2b1-10_all.deb' emacs-common_1%3a28.2+1-10_all.deb 14004332 MD5Sum:374a384eec3c9eabb1b97ef2bae84d88
'http://deb.debian.org/debian/pool/main/e/emacs/emacs-bin-common_28.2%2b1-10_amd64.deb' emacs-bin-common_1%3a28.2+1-10_amd64.deb 101852 MD5Sum:fb3254fac1732c713addd782a408663c
'http://deb.debian.org/debian/pool/main/g/gcc-12/libgccjit0_12.2.0-14_amd64.deb' libgccjit0_12.2.0-14_amd64.deb 8782528 MD5Sum:0af91771f4dcf247ddffd33c0bcde993
'http://deb.debian.org/debian/pool/main/m/m17n-db/m17n-db_1.8.0-5_all.deb' m17n-db_1.8.0-5_all.deb 1297224 MD5Sum:d727a0795d335d2cc76f8ebe074998aa
'http://deb.debian.org/debian/pool/main/libo/libotf/libotf1_0.9.16-4_amd64.deb' libotf1_0.9.16-4_amd64.deb 55484 MD5Sum:a5be4717ef0201ff1f9760ed74181fbc
'http://deb.debian.org/debian/pool/main/m/m17n-lib/libm17n-0_1.8.0-6_amd64.deb' libm17n-0_1.8.0-6_amd64.deb 253528 MD5Sum:67320e7a44df16e3f0622a8d2814deb6
'http://deb.debian.org/debian/pool/main/e/emacs/emacs-gtk_28.2%2b1-10_amd64.deb' emacs-gtk_1%3a28.2+1-10_amd64.deb 5997860 MD5Sum:d35f587db91d0b946835d24c6e4b778b
'http://deb.debian.org/debian/pool/main/e/emacs/emacs_28.2%2b1-10_all.deb' emacs_1%3a28.2+1-10_all.deb 15780 MD5Sum:c14b4d686bf10dd393503cee6c08d27b
20:04 ♒ ॐ ♅ ♄ ⛢     ☺ 😄
rickysarraf commented 1 year ago

Same with 2 packages

rrs@priyasi:~/.../apt-offline (master)$ sudo apt-offline set /tmp/emacs.uris --install-packages emacs parallel
Gathering installation details for package ['emacs', 'parallel']
20:06 ♒ ॐ ♅ ♄ ⛢     ☺ 😄    

rrs@priyasi:~/.../apt-offline (master)$ cat /tmp/emacs.uris 
'http://deb.debian.org/debian/pool/main/e/emacs/emacs-el_28.2%2b1-10_all.deb' emacs-el_1%3a28.2+1-10_all.deb 16872928 MD5Sum:4220a0f76d24cb81971d03fd6df4d965
'http://deb.debian.org/debian/pool/main/e/emacs/emacs-common_28.2%2b1-10_all.deb' emacs-common_1%3a28.2+1-10_all.deb 14004332 MD5Sum:374a384eec3c9eabb1b97ef2bae84d88
'http://deb.debian.org/debian/pool/main/e/emacs/emacs-bin-common_28.2%2b1-10_amd64.deb' emacs-bin-common_1%3a28.2+1-10_amd64.deb 101852 MD5Sum:fb3254fac1732c713addd782a408663c
'http://deb.debian.org/debian/pool/main/g/gcc-12/libgccjit0_12.2.0-14_amd64.deb' libgccjit0_12.2.0-14_amd64.deb 8782528 MD5Sum:0af91771f4dcf247ddffd33c0bcde993
'http://deb.debian.org/debian/pool/main/m/m17n-db/m17n-db_1.8.0-5_all.deb' m17n-db_1.8.0-5_all.deb 1297224 MD5Sum:d727a0795d335d2cc76f8ebe074998aa
'http://deb.debian.org/debian/pool/main/libo/libotf/libotf1_0.9.16-4_amd64.deb' libotf1_0.9.16-4_amd64.deb 55484 MD5Sum:a5be4717ef0201ff1f9760ed74181fbc
'http://deb.debian.org/debian/pool/main/m/m17n-lib/libm17n-0_1.8.0-6_amd64.deb' libm17n-0_1.8.0-6_amd64.deb 253528 MD5Sum:67320e7a44df16e3f0622a8d2814deb6
'http://deb.debian.org/debian/pool/main/e/emacs/emacs-gtk_28.2%2b1-10_amd64.deb' emacs-gtk_1%3a28.2+1-10_amd64.deb 5997860 MD5Sum:d35f587db91d0b946835d24c6e4b778b
'http://deb.debian.org/debian/pool/main/e/emacs/emacs_28.2%2b1-10_all.deb' emacs_1%3a28.2+1-10_all.deb 15780 MD5Sum:c14b4d686bf10dd393503cee6c08d27b
'http://deb.debian.org/debian/pool/main/p/parallel/parallel_20221122%2bds-2_all.deb' parallel_20221122+ds-2_all.deb 1873196 MD5Sum:904fb973c23ca9d92e413a975e1ed667
20:06 ♒ ॐ ♅ ♄ ⛢     ☺ 😄
mtalexan commented 1 year ago

What exactly is a group package ? Do you mean a meta package ?

Yes, sorry that's exactly what I meant.

I tried with your example above and I have: ...snip...

Ah ha, it appears to be an issue with the command-line parser (I think?).

On a completely fresh Ubuntu 22.04 desktop minimal install (new VM), this:

sudo apt-get update
sudo apt-get install apt-offline
sudo apt-offline set --install-packages emacs -- emacs.uris

produces an error. However on a separate completely fresh Ubuntu 22.04 desktop minimal install (new VM), this:

sudo apt-get update
sudo apt-get install apt-offline
sudo apt-offline set emacs.uris --install-packages emacs

works as expected.

In both cases the initial output printed is:

Gathering installation details for package ['emacs']

But in the case with an error it immediately prints the error I noted above.

It looks like there's an error with the command-line parsing mismatching the documented support. From the man page:

apt-offline relies on argparse for argument/option parsing. To explicitly instruct apt-offline about an argument, you can pass it with the -- delimiter.

Ex. apt-offline set --update --upgrade --install-packages wm2 -- foo.sig

By specifying the -- delimiter, we instruct apt-offline that foo.sig is an argument to the apt-offline command and not to the --install-packages option.

Otherwise, you could also use it positionally next to the set command

Ex. apt-offline set foo.sig --update --upgrade --install-packages wm2

But when I try to use the format from the first example for the emacs package, or a list that includes the emacs package, it fails, while the format of the second example works. Strangely though, both work with the parallel package by itself.

rickysarraf commented 1 year ago

produces an error. However on a separate completely fresh Ubuntu 22.04 desktop minimal install (new VM), this:

Given that the issue is not reproducible on 2 different machine with the same version of the OS, it should be fair to say that the issue is with the machine in particular. Possibly, you may have other local python modules installed ?

On the problematic machine, could you run the same command with --verbose switch ? Maybe that'd shed some more light.

mtalexan commented 1 year ago

Given that the issue is not reproducible on 2 different machine with the same version of the OS, it should be fair to say that the issue is with the machine in particular. Possibly, you may have other local python modules installed ?

I'm actually saying the problem is easily reproducible on any machine, but it's a problem only when certain command-line syntax is used.

Putting the file name at the end like in the manpage example fails:

apt-offline set --install-packages emacs -- emacs.uris

But putting it earlier like the alternative in the manpage works:

apt-online set emacs.uris --install-packages emacs
rickysarraf commented 1 year ago

@mtalexan

I'm actually saying the problem is easily reproducible on any machine, but it's a problem only when certain command-line syntax is used.

It seems to be working fine for me here.

rrs@priyasi:~$ sudo apt-offline set --install-packages emacs parallel -- /tmp/emacs.uris
[sudo] password for rrs: 
Gathering installation details for package ['emacs', 'parallel']
13:56 ♒ ॐ ♅ ♄ ⛢     ☺ 😄    

rrs@priyasi:~$ cat /tmp/emacs.uris 
'http://deb.debian.org/debian/pool/main/e/emacs/emacs-el_28.2%2b1-10_all.deb' emacs-el_1%3a28.2+1-10_all.deb 16872928 MD5Sum:4220a0f76d24cb81971d03fd6df4d965
'http://deb.debian.org/debian/pool/main/e/emacs/emacs-common_28.2%2b1-10_all.deb' emacs-common_1%3a28.2+1-10_all.deb 14004332 MD5Sum:374a384eec3c9eabb1b97ef2bae84d88
'http://deb.debian.org/debian/pool/main/e/emacs/emacs-bin-common_28.2%2b1-10_amd64.deb' emacs-bin-common_1%3a28.2+1-10_amd64.deb 101852 MD5Sum:fb3254fac1732c713addd782a408663c
'http://deb.debian.org/debian/pool/main/g/gcc-12/libgccjit0_12.2.0-14_amd64.deb' libgccjit0_12.2.0-14_amd64.deb 8782528 MD5Sum:0af91771f4dcf247ddffd33c0bcde993
'http://deb.debian.org/debian/pool/main/m/m17n-db/m17n-db_1.8.0-5_all.deb' m17n-db_1.8.0-5_all.deb 1297224 MD5Sum:d727a0795d335d2cc76f8ebe074998aa
'http://deb.debian.org/debian/pool/main/libo/libotf/libotf1_0.9.16-4_amd64.deb' libotf1_0.9.16-4_amd64.deb 55484 MD5Sum:a5be4717ef0201ff1f9760ed74181fbc
'http://deb.debian.org/debian/pool/main/m/m17n-lib/libm17n-0_1.8.0-6_amd64.deb' libm17n-0_1.8.0-6_amd64.deb 253528 MD5Sum:67320e7a44df16e3f0622a8d2814deb6
'http://deb.debian.org/debian/pool/main/e/emacs/emacs-gtk_28.2%2b1-10_amd64.deb' emacs-gtk_1%3a28.2+1-10_amd64.deb 5997860 MD5Sum:d35f587db91d0b946835d24c6e4b778b
'http://deb.debian.org/debian/pool/main/e/emacs/emacs_28.2%2b1-10_all.deb' emacs_1%3a28.2+1-10_all.deb 15780 MD5Sum:c14b4d686bf10dd393503cee6c08d27b
'http://deb.debian.org/debian/pool/main/p/parallel/parallel_20221122%2bds-2_all.deb' parallel_20221122+ds-2_all.deb 1873196 MD5Sum:904fb973c23ca9d92e413a975e1ed667
13:56 ♒ ॐ ♅ ♄ ⛢     ☺ 😄
mtalexan commented 1 year ago

I was just trying to reproduce the issue with the --verbose flag, and now it's working with both syntaxes. I created a fresh VM again for this test since it was reproducible with multiple fresh VMs last week, so I don't have the old one to reexamine. Weird.