Avoid rehashing

[runejuhl]

Fixes #93

[runejuhl]

I've played around with tests, and I've hit an issue -- maybe someone here can help?

I get some errors when running some of the bats tests due to a missing GPG pubkey.

Tests are running inside a clean docker Ubuntu cosmic with gpg, git and make installed and the private keys are generated just fine. It seems that the issue is that apt is missing a GPG key so it fails on the apt-get fetches package list test.

It generates the key and I can see a key in ./test/tmp/freight/var/cache, but apparently it's the wrong one?

This is the output from running the test suite:

I have no name!@f53873d350e1:/freight$ make check
ln -sf $(which dash) test/tmp/bin/sh
PATH=test/tmp/bin/:$PATH test/tmp/bats/bin/bats test/
 ✓ freight-add adds package to distro main component
 ✓ freight-add adds package to a component
 ✓ freight-add adds package and hard link to multiple components
 ✓ freight-add detects duplicate package
 ✓ freight-add adds source .dsc files
 ✓ freight-add adds source .tar.gz files
 ✓ freight-add adds source .orig.tar.gz files
 ✓ freight-add handles VARLIB being a symlink
 ✓ freight-cache builds distro Release/InRelease file
 ✓ freight-cache builds per-component Release file
 ✓ freight-cache builds pool
 ✓ freight-cache generates valid Release/InRelease signatures
 ✓ freight-cache signs Release/InRelease with two keys
 ✓ freight-cache works without tty
 ✗ apt-get fetches package list
   (in test file test/apt_cache.bats, line 61)
     `apt-get -c ${FIXTURES}/apt.conf update' failed with status 100
   # [freight] added /freight/test/fixtures/test_1.0_all.deb to apt/example
   # [freight] added /freight/test/fixtures/test_1.0_all.deb to apt/example/comp
   apt-get is /usr/bin/apt-get
   *Ver: Standard .deb
   # [freight] adding test_1.0_all.deb to pool
   # [freight] adding test_1.0_all.deb to pool
   Get:1 file:/freight/test/tmp/freight/var/cache example InRelease [6215 B]
   Get:1 file:/freight/test/tmp/freight/var/cache example InRelease [6215 B]
   Err:1 file:/freight/test/tmp/freight/var/cache example InRelease
     The following signatures couldn't be verified because the public key is not available: NO_PUBKEY EAA3B9A06E420F30
   rm: cannot remove '/var/cache/apt/archives/partial/*.deb': Permission denied
   Reading package lists...
   W: GPG error: file:/freight/test/tmp/freight/var/cache example InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY EAA3B9A06E420F30
   E: The repository 'file:/freight/test/tmp/freight/var/cache example InRelease' is not signed.
   E: flAbsPath on ./test/tmp/apt/var/lib/dpkg/status failed - realpath (2: No such file or directory)
   E: Could not open file  - open (2: No such file or directory)
   E: Problem opening 
   E: The package lists or status file could not be parsed or opened.
 ✓ freight-cache removes deleted packages from pool
 ✓ freight-cache --keep retains deleted packages in pool
 ✓ freight-cache handles VARLIB being a symlink
 ✓ freight-cache skips partial source packages
 ✓ freight-cache builds source-only archive
 ✗ apt-get fetches source package list
   (in test file test/apt_cache_source.bats, line 38)
     `apt-get -c ${FIXTURES}/apt.conf update' failed with status 100
   apt-get is /usr/bin/apt-get
   *Ver: Standard .deb
   # [freight] added /freight/test/fixtures/source_1.0-1.dsc to apt/example
   # [freight] added /freight/test/fixtures/source_1.0-1.tar.gz to apt/example
   # [freight] added /freight/test/fixtures/source_1.0.orig.tar.gz to apt/example
   # [freight] adding source_1.0-1.dsc to pool
   # [freight] adding source_1.0.orig.tar.gz to pool
   # [freight] adding source_1.0-1.tar.gz to pool
   Get:1 file:/freight/test/tmp/freight/var/cache example InRelease [3010 B]
   Get:1 file:/freight/test/tmp/freight/var/cache example InRelease [3010 B]
   Err:1 file:/freight/test/tmp/freight/var/cache example InRelease
     The following signatures couldn't be verified because the public key is not available: NO_PUBKEY EAA3B9A06E420F30
   rm: cannot remove '/var/cache/apt/archives/partial/*.deb': Permission denied
   Reading package lists...
   W: GPG error: file:/freight/test/tmp/freight/var/cache example InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY EAA3B9A06E420F30
   E: The repository 'file:/freight/test/tmp/freight/var/cache example InRelease' is not signed.
   E: flAbsPath on ./test/tmp/apt/var/lib/dpkg/status failed - realpath (2: No such file or directory)
   E: Could not open file  - open (2: No such file or directory)
   E: Problem opening 
   E: The package lists or status file could not be parsed or opened.

21 tests, 2 failures
make: *** [Makefile:97: check] Error 1

[mmoll]

@runejuhl it seems to work fine in the travis container with Ubuntu/xenial: https://travis-ci.org/mmoll/freight/jobs/508562196

[runejuhl]

Thank you for testing it @mmol, I'll just ignore it for now then.

[KlavsKlavsen]

It seems only a few shellcheck fixes is needed for freight to be more "shell safe" and for this then to be merged, as all functional tests succeed just fine ?

[runejuhl]

Hi @KlavsKlavsen

In theory it should work, but I'm not sure if I ever tried it out for real. I do recall running the test suite in Docker, but as far as I remember there was something that stopped the regular tests from using my changes. I think it was due to test data per test run being generated in a tempdir, so no two tests would ever try hashing the same file twice, but I'm not certain.

[yegorich]

@runejuhl I have the same test failures when I run the tests locally. Have you managed to fix them?

[runejuhl]

Hi @yegorich! No, sorry to disappoint, but it fell off the radar and I never got any further with it. We're still using Freight though, so if you can figure out the issue with the tests I'd love to have this PR finally merged.

[yegorich]

If I understand it correctly, the key must be added to apt's key-ring. This part is missing. I still have no idea why Travis was working. Did you have to provide a password for the newly generated key? @mmoll any idea?

I'll try to look at your PR. What's the purpose of this file .dir-locals.el in this PR?

[runejuhl]

I'll try to look at your PR. What's the purpose of this file .dir-locals.el in this PR?

.dir-locals.el is a file read by emacs that can set emacs variables or run arbitrary code (after user confirmation) in order to configure emacs to adhere to project defaults. Since a common indentation level is 2 spaces and Freight uses 4 spaces I added the file to avoid messing up indentation.

[runejuhl]

I've rebased onto @yegorich's fixes at https://github.com/freight-team/freight/pull/110 and all tests now pass:

$ docker run -ti --rm -v $(pwd):/freight -u $(id -u) -w /freight $(docker build -q .) make check
PATH=test/tmp/bin/:$PATH test/tmp/bats/bin/bats test/
 ✓ freight-add adds package to distro main component
 ✓ freight-add adds package to a component
 ✓ freight-add adds package and hard link to multiple components
 ✓ freight-add detects duplicate package
 ✓ freight-add adds source .dsc files
 ✓ freight-add adds source .tar.gz files
 ✓ freight-add adds source .orig.tar.gz files
 ✓ freight-add handles VARLIB being a symlink
 ✓ freight-cache builds distro Release/InRelease file
 ✓ freight-cache builds per-component Release file
 ✓ freight-cache builds pool
 ✓ freight-cache generates valid Release/InRelease signatures
 ✓ freight-cache signs Release/InRelease with two keys
 ✓ freight-cache works without tty
 ✓ apt-get fetches package list
 ✓ freight-cache removes deleted packages from pool
 ✓ freight-cache --keep retains deleted packages in pool
 ✓ freight-cache handles VARLIB being a symlink
 ✓ freight-cache skips partial source packages
 ✓ freight-cache builds source-only archive
 ✓ apt-get fetches source package list

21 tests, 0 failures

The Dockerfile used is very simple:

FROM ubuntu:focal

RUN apt update && apt install -y git make gnupg

Let me know if you want me to add the Dockerfile to this PR too.

[yegorich]

Dockerfile is a good idea but I would put it into its own PR.

[yegorich]

Have you tried to invoke docker when you remove test/tmp folder? This is what I get:

git clone --depth 1 https://github.com/sstephenson/bats.git test/tmp/bats
Cloning into 'test/tmp/bats'...
git clone --depth 1 https://github.com/jasonkarns/bats-assert.git test/tmp/bats-assert
Cloning into 'test/tmp/bats-assert'...
mkdir -p test/tmp/bin
ln -sf $(which dash) test/tmp/bin/sh
PATH=test/tmp/bin/:$PATH test/tmp/bats/bin/bats test/
1..21
not ok 1 freight-add adds package to distro main component
# (from function `gpg_init' in file test/freight_helpers.bash, line 45,
#  from function `freight_init' in file test/freight_helpers.bash, line 17,
#  from function `setup' in test file test/apt_add.bats, line 6)
#   `freight_init' failed with status 2
# gpg: keybox '/usr/local/src/test/tmp/gpg/pubring.kbx' created
# gpg: agent_genkey failed: Inappropriate ioctl for device
# gpg: key generation failed: Inappropriate ioctl for device

Here at least export GPG_TTY=$(tty) is missing.

But even if I perform the test locally so that all keys are created, the test in docker fails.

[yegorich]

@runejuhl could you rebase, remove Docker patch, and push again. I'd like to see whether GitHub Actions would work for you as you don't have the secrets in your repository. Thanks

[yegorich]

@runejuhl I mean put .dir-locals.el into a separate PR.

[runejuhl]

@yegorich sure!