search

aptly-dev / aptly

aptly - Debian repository management tool
https://www.aptly.info/
MIT License
2.58k stars 376 forks source link

panic: unable to load files #350

Open Gregy opened 8 years ago

Gregy commented 8 years ago

Hello, after about a week running automated repository (new packages are loaded through ssh and by gitlab-ci) I started getting error messages. Since the first error appeared I cannot publish any new packages no matter the endpoint/snapshot. I started the server with aptly 0.8 from debian repos and after the error I upgraded to 0.9.6 but it did not help.

Aptly shows the following error on command "aptly publish switch -component=api development api-all-development-1455552836":

on 0.8-3 (version from debian jessie official repository)

Loading packages...
Generating metadata files and linking package files...
Finalizing metadata files...
Signing file 'Release' with gpg, please enter your passphrase when prompted:
Clearsigning file 'Release' with gpg, please enter your passphrase when prompted:
Cleaning up prefix "." components api...
panic: unable to load files [recovered]
    panic: unable to load files

goroutine 1 [running]:
runtime.panic(0x820a00, 0xc21152edf0)
    /usr/lib/go/src/pkg/runtime/panic.c:266 +0xb6
github.com/smira/aptly/cmd.func·027()
    /home/seb/work/debian/build-area/aptly-0.8/src/github.com/smira/aptly/cmd/run.go:14 +0x9d
runtime.panic(0x820a00, 0xc21152edf0)
    /usr/lib/go/src/pkg/runtime/panic.c:248 +0x106
github.com/smira/aptly/deb.(*PackageCollection).loadFiles(0xc210d72a20, 0xc211b19960, 0x1)
    /home/seb/work/debian/build-area/aptly-0.8/src/github.com/smira/aptly/deb/package_collection.go:150 +0xe7
github.com/smira/aptly/deb.(*Package).Files(0xc211b19960, 0xc2116921a0, 0x15, 0x0)
    /home/seb/work/debian/build-area/aptly-0.8/src/github.com/smira/aptly/deb/package.go:385 +0x94
github.com/smira/aptly/deb.func·010(0xc211b19960, 0xc21044c4e0, 0x2b)
    /home/seb/work/debian/build-area/aptly-0.8/src/github.com/smira/aptly/deb/publish.go:906 +0x9a
github.com/smira/aptly/deb.(*PackageList).ForEach(0xc211e017e0, 0x7f08fda519e8, 0x7f08fc305138, 0xc211017a20)
    /home/seb/work/debian/build-area/aptly-0.8/src/github.com/smira/aptly/deb/list.go:118 +0x8b
github.com/smira/aptly/deb.(*PublishedRepoCollection).CleanupPrefixComponentFiles(0xc2109a7d20, 0xc210d17d10, 0x1, 0xc2100d9290, 0x1, ...)
    /home/seb/work/debian/build-area/aptly-0.8/src/github.com/smira/aptly/deb/publish.go:911 +0xa4f
github.com/smira/aptly/cmd.aptlyPublishSwitch(0xc2100d8ea0, 0xc21006f620, 0x2, 0x2, 0x7f08fdbf1a68, ...)
    /home/seb/work/debian/build-area/aptly-0.8/src/github.com/smira/aptly/cmd/publish_switch.go:99 +0x1034
github.com/smira/commander.(*Command).Dispatch(0xc2100d8ea0, 0xc21006f620, 0x2, 0x2, 0x1, ...)
    /home/seb/work/debian/build-area/aptly-0.8/src/github.com/smira/aptly/_vendor/src/github.com/smira/commander/commands.go:305 +0x4fd
github.com/smira/commander.(*Command).Dispatch(0xc2100da120, 0xc21006f610, 0x3, 0x3, 0x820a01, ...)
    /home/seb/work/debian/build-area/aptly-0.8/src/github.com/smira/aptly/_vendor/src/github.com/smira/commander/commands.go:283 +0x1a9
github.com/smira/commander.(*Command).Dispatch(0xc2100da6c0, 0xc21006f600, 0x4, 0x4, 0xc21004ebd0, ...)
    /home/seb/work/debian/build-area/aptly-0.8/src/github.com/smira/aptly/_vendor/src/github.com/smira/commander/commands.go:283 +0x1a9
github.com/smira/aptly/cmd.Run(0xc2100da6c0, 0xc21000a010, 0x5, 0x5, 0x1, ...)
    /home/seb/work/debian/build-area/aptly-0.8/src/github.com/smira/aptly/cmd/run.go:38 +0x178
main.main()
    /home/seb/work/debian/build-area/aptly-0.8/src/github.com/smira/aptly/main.go:9 +0x77

goroutine 3 [syscall]:
os/signal.loop()
    /usr/lib/go/src/pkg/os/signal/signal_unix.go:21 +0x1e
created by os/signal.init·1
    /usr/lib/go/src/pkg/os/signal/signal_unix.go:27 +0x31

goroutine 14 [finalizer wait]:
runtime.park(0x40b000, 0xfa4208, 0xfa1028)
    /usr/lib/go/src/pkg/runtime/proc.c:1342 +0x66
runfinq()
    /usr/lib/go/src/pkg/runtime/mgc0.c:2279 +0x84
runtime.goexit()
    /usr/lib/go/src/pkg/runtime/proc.c:1394

on 0.9.6:

Loading packages...
Generating metadata files and linking package files...
panic: unable to load files [recovered]
    panic: unable to load files

goroutine 1 [running]:
github.com/smira/aptly/cmd.Run.func1(0xc8210bbf18)
    /Users/smira/Documents/go/src/github.com/smira/aptly/cmd/run.go:16 +0x7f
github.com/smira/aptly/deb.(*PackageCollection).loadFiles(0xc8203eeec0, 0xc820165740, 0x57b)
    /Users/smira/Documents/go/src/github.com/smira/aptly/deb/package_collection.go:150 +0x14a
github.com/smira/aptly/deb.(*Package).Files(0xc820165740, 0x0, 0x0, 0x0)
    /Users/smira/Documents/go/src/github.com/smira/aptly/deb/package.go:396 +0xb6
github.com/smira/aptly/deb.(*Package).LinkFromPool(0xc820165740, 0x7f6bca9ec390, 0xc8204c4f70, 0x7f6bca9ec2b0, 0xc8203eed40, 0xc820155340, 0x1, 0xc820155400, 0x3, 0x0, ...)
    /Users/smira/Documents/go/src/github.com/smira/aptly/deb/package.go:535 +0xb7
github.com/smira/aptly/deb.(*PublishedRepo).Publish.func1(0xc820165740, 0x0, 0x0)
    /Users/smira/Documents/go/src/github.com/smira/aptly/deb/publish.go:547 +0x39a
github.com/smira/aptly/deb.(*PackageList).ForEachIndexed(0xc820086cc0, 0xc8210bb640, 0x0, 0x0)
    /Users/smira/Documents/go/src/github.com/smira/aptly/deb/list.go:139 +0xed
github.com/smira/aptly/deb.(*PublishedRepo).Publish(0xc82013d2b0, 0x7f6bca9ec2b0, 0xc8203eed40, 0x7f6bca9ec368, 0xc8200cfce0, 0xc820167100, 0x7f6bca9ec258, 0xc8203ea120, 0x7f6bca9ec2f8, 0xc820086a80, ...)
    /Users/smira/Documents/go/src/github.com/smira/aptly/deb/publish.go:592 +0x1135
github.com/smira/aptly/cmd.aptlyPublishSwitch(0xc8201527e0, 0xc82014fba0, 0x2, 0x2, 0x0, 0x0)
    /Users/smira/Documents/go/src/github.com/smira/aptly/cmd/publish_switch.go:97 +0x12ba
github.com/smira/commander.(*Command).Dispatch(0xc8201527e0, 0xc82014fba0, 0x2, 0x2, 0x0, 0x0)
    /Users/smira/Documents/go/src/github.com/smira/aptly/_vendor/src/github.com/smira/commander/commands.go:305 +0x647
github.com/smira/commander.(*Command).Dispatch(0xc820152a20, 0xc82014fb90, 0x3, 0x3, 0x0, 0x0)
    /Users/smira/Documents/go/src/github.com/smira/aptly/_vendor/src/github.com/smira/commander/commands.go:283 +0x1f0
github.com/smira/commander.(*Command).Dispatch(0xc820153200, 0xc82014fb80, 0x4, 0x4, 0x0, 0x0)
    /Users/smira/Documents/go/src/github.com/smira/aptly/_vendor/src/github.com/smira/commander/commands.go:283 +0x1f0
github.com/smira/aptly/cmd.Run(0xc820153200, 0xc82000a130, 0x5, 0x5, 0xc3c601, 0x0)
    /Users/smira/Documents/go/src/github.com/smira/aptly/cmd/run.go:40 +0x19a
main.main()
    /Users/smira/Documents/go/src/github.com/smira/aptly/main.go:9 +0x73

goroutine 6 [syscall, 1 minutes]:
os/signal.loop()
    /usr/local/Cellar/go/1.5.3/libexec/src/os/signal/signal_unix.go:22 +0x18
created by os/signal.init.1
    /usr/local/Cellar/go/1.5.3/libexec/src/os/signal/signal_unix.go:28 +0x37

goroutine 7 [runnable]:
github.com/syndtr/goleveldb/leveldb/util.(*BufferPool).drain(0xc8201441c0)
    /Users/smira/Documents/go/src/github.com/smira/aptly/_vendor/src/github.com/syndtr/goleveldb/leveldb/util/buffer_pool.go:206 +0x29d
created by github.com/syndtr/goleveldb/leveldb/util.NewBufferPool
    /Users/smira/Documents/go/src/github.com/smira/aptly/_vendor/src/github.com/syndtr/goleveldb/leveldb/util/buffer_pool.go:237 +0x26b

goroutine 12 [runnable]:
github.com/syndtr/goleveldb/leveldb.(*DB).mpoolDrain(0xc82006cdc0)
    /Users/smira/Documents/go/src/github.com/smira/aptly/_vendor/src/github.com/syndtr/goleveldb/leveldb/db_state.go:82 +0x14b
created by github.com/syndtr/goleveldb/leveldb.openDB
    /Users/smira/Documents/go/src/github.com/smira/aptly/_vendor/src/github.com/syndtr/goleveldb/leveldb/db.go:140 +0x7a1

goroutine 21 [sleep]:
time.Sleep(0xbebc200)
    /usr/local/Cellar/go/1.5.3/libexec/src/runtime/time.go:59 +0xf9
github.com/cheggaaa/pb.(*ProgressBar).writer(0xc820b99950)
    /Users/smira/Documents/go/src/github.com/smira/aptly/_vendor/src/github.com/cheggaaa/pb/pb.go:332 +0x5b
created by github.com/cheggaaa/pb.(*ProgressBar).Start
    /Users/smira/Documents/go/src/github.com/smira/aptly/_vendor/src/github.com/cheggaaa/pb/pb.go:101 +0x9c

aptly db recover runs fine but does not help. If I try to run aptly db cleanup I get this:

aptly@repo:~$ aptly db -verbose=true cleanup
Loading mirrors, local repos, snapshots and published repos...
Loading mirrors:
- zfs
- saltstack-2015-05
- salt-latest
- mongodb
- zfs-full
- node4
- node
Loading local repos:
- api-orig-staging
- gui-orig-production
- api-orig-production
- gui-orig-development
- api-orig-development
- gui-orig-staging
Loading snapshots:
- gui-all-development-1455570324
- zfs-prod-1
- gui-orig-staging-1455292106
- mongodb-prod
- hv-all-prod-1
- api-orig-production-1454916051
- gui-orig-development-1455570467
- gui-all-development-1455550033
- api-all-staging-1454675931
- mongo-031215-v3.0.7
- hv-all-development-120116
- gui-orig-staging-1455573869
- node-031215
- hv-270116
- api-all-development-1455552836
- salt-120116
- api-orig-staging-1454675931
- gui-all-production-1455295270
- zfs-031215-v0.6.5.2
- node-prod
- api-all-development-1455210365
- gui-all-staging-1455292106
- gui-all-development-1455570467
- api-all-production-1454916051
- salt-031215
- gui-orig-development-1455570324
- gui-orig-production-1455295270
- saltstack-prod
- gui-orig-development-1455550033
- saltstack-stage
- api-orig-development-1455210365
- api-orig-development-1455552836
- hv-all-development-031215
- gui-all-staging-1455573869
Loading published repositories:
- :./development{|}
- :./production{|}
- :./staging{|}
Loading list of all packages...
Deleting unreferenced packages (0)...
Building list of files referenced by packages...
panic: unable to load files [recovered]
    panic: unable to load files

goroutine 1 [running]:
github.com/smira/aptly/cmd.Run.func1(0xc820221ef8)
    /Users/smira/Documents/go/src/github.com/smira/aptly/cmd/run.go:16 +0x7f
github.com/smira/aptly/deb.(*PackageCollection).loadFiles(0xc820b8cae0, 0xc820164780, 0x0)
    /Users/smira/Documents/go/src/github.com/smira/aptly/deb/package_collection.go:150 +0x14a
github.com/smira/aptly/deb.(*Package).Files(0xc820164780, 0x0, 0x0, 0x0)
    /Users/smira/Documents/go/src/github.com/smira/aptly/deb/package.go:396 +0xb6
github.com/smira/aptly/deb.(*Package).FilepathList(0xc820164780, 0x7f4657a9c780, 0xc820b8cee0, 0x0, 0x0, 0x0, 0x0, 0x0)
    /Users/smira/Documents/go/src/github.com/smira/aptly/deb/package.go:631 +0x6b
github.com/smira/aptly/cmd.aptlyDbCleanup.func7(0xc820b4a030, 0x2d, 0x2d, 0x0, 0x0)
    /Users/smira/Documents/go/src/github.com/smira/aptly/cmd/db_cleanup.go:213 +0x855
github.com/smira/aptly/deb.(*PackageRefList).ForEach(0xc820b8c240, 0xc820221b30, 0x0, 0x0)
    /Users/smira/Documents/go/src/github.com/smira/aptly/deb/reflist.go:81 +0xe8
github.com/smira/aptly/cmd.aptlyDbCleanup(0xc82013cb40, 0xc8200117d0, 0x0, 0x0, 0x0, 0x0)
    /Users/smira/Documents/go/src/github.com/smira/aptly/cmd/db_cleanup.go:221 +0xcd9
github.com/smira/commander.(*Command).Dispatch(0xc82013cb40, 0xc8200117d0, 0x0, 0x0, 0x0, 0x0)
    /Users/smira/Documents/go/src/github.com/smira/aptly/_vendor/src/github.com/smira/commander/commands.go:305 +0x647
github.com/smira/commander.(*Command).Dispatch(0xc82013cd80, 0xc8200117d0, 0x1, 0x1, 0x0, 0x0)
    /Users/smira/Documents/go/src/github.com/smira/aptly/_vendor/src/github.com/smira/commander/commands.go:283 +0x1f0
github.com/smira/commander.(*Command).Dispatch(0xc820154a20, 0xc8200117c0, 0x2, 0x2, 0x0, 0x0)
    /Users/smira/Documents/go/src/github.com/smira/aptly/_vendor/src/github.com/smira/commander/commands.go:283 +0x1f0
github.com/smira/aptly/cmd.Run(0xc820154a20, 0xc82000a210, 0x3, 0x3, 0xc3c601, 0x0)
    /Users/smira/Documents/go/src/github.com/smira/aptly/cmd/run.go:40 +0x19a
main.main()
    /Users/smira/Documents/go/src/github.com/smira/aptly/main.go:9 +0x73

goroutine 6 [syscall]:
os/signal.loop()
    /usr/local/Cellar/go/1.5.3/libexec/src/os/signal/signal_unix.go:22 +0x18
created by os/signal.init.1
    /usr/local/Cellar/go/1.5.3/libexec/src/os/signal/signal_unix.go:28 +0x37

goroutine 8 [runnable]:
github.com/syndtr/goleveldb/leveldb/util.(*BufferPool).drain(0xc8201441c0)
    /Users/smira/Documents/go/src/github.com/smira/aptly/_vendor/src/github.com/syndtr/goleveldb/leveldb/util/buffer_pool.go:206 +0x29d
created by github.com/syndtr/goleveldb/leveldb/util.NewBufferPool
    /Users/smira/Documents/go/src/github.com/smira/aptly/_vendor/src/github.com/syndtr/goleveldb/leveldb/util/buffer_pool.go:237 +0x26b

goroutine 17 [sleep]:
time.Sleep(0xbebc200)
    /usr/local/Cellar/go/1.5.3/libexec/src/runtime/time.go:59 +0xf9
github.com/cheggaaa/pb.(*ProgressBar).writer(0xc8201602d0)
    /Users/smira/Documents/go/src/github.com/smira/aptly/_vendor/src/github.com/cheggaaa/pb/pb.go:332 +0x5b
created by github.com/cheggaaa/pb.(*ProgressBar).Start
    /Users/smira/Documents/go/src/github.com/smira/aptly/_vendor/src/github.com/cheggaaa/pb/pb.go:101 +0x9c

If I try to drop a publish I get this:

aptly publish drop development
Removing /var/aptly/.aptly/public/dists/development...
Cleaning up prefix "." components api, gui, hv...
panic: unable to load files [recovered]
    panic: unable to load files

goroutine 1 [running]:
github.com/smira/aptly/cmd.Run.func1(0xc8200f7f18)
    /Users/smira/Documents/go/src/github.com/smira/aptly/cmd/run.go:16 +0x7f
github.com/smira/aptly/deb.(*PackageCollection).loadFiles(0xc820010fa0, 0xc82026a000, 0x1f)
    /Users/smira/Documents/go/src/github.com/smira/aptly/deb/package_collection.go:150 +0x14a
github.com/smira/aptly/deb.(*Package).Files(0xc82026a000, 0x0, 0x0, 0x0)
    /Users/smira/Documents/go/src/github.com/smira/aptly/deb/package.go:396 +0xb6
github.com/smira/aptly/deb.(*PublishedRepoCollection).CleanupPrefixComponentFiles.func1(0xc82026a000, 0x0, 0x0)
    /Users/smira/Documents/go/src/github.com/smira/aptly/deb/publish.go:1003 +0xd3
github.com/smira/aptly/deb.(*PackageList).ForEach(0xc820209ec0, 0xc8200f7670, 0x0, 0x0)
    /Users/smira/Documents/go/src/github.com/smira/aptly/deb/list.go:123 +0xa3
github.com/smira/aptly/deb.(*PublishedRepoCollection).CleanupPrefixComponentFiles(0xc82016b080, 0xc820157610, 0x1, 0xc82016eac0, 0x3, 0x4, 0x7f4db277b068, 0xc820157b30, 0xc82000a2c0, 0x7f4db277af98, ...)
    /Users/smira/Documents/go/src/github.com/smira/aptly/deb/publish.go:1008 +0xca7
github.com/smira/aptly/deb.(*PublishedRepoCollection).Remove(0xc82016b080, 0x7f4db277b008, 0xc8200cd760, 0x0, 0x0, 0xb9de40, 0x1, 0x7ffc255518f2, 0xb, 0xc82000a2c0, ...)
    /Users/smira/Documents/go/src/github.com/smira/aptly/deb/publish.go:1080 +0x4c9
github.com/smira/aptly/cmd.aptlyPublishDrop(0xc820151b00, 0xc82014fc20, 0x1, 0x2, 0x0, 0x0)
    /Users/smira/Documents/go/src/github.com/smira/aptly/cmd/publish_drop.go:26 +0x31e
github.com/smira/commander.(*Command).Dispatch(0xc820151b00, 0xc82014fc20, 0x1, 0x2, 0x0, 0x0)
    /Users/smira/Documents/go/src/github.com/smira/aptly/_vendor/src/github.com/smira/commander/commands.go:305 +0x647
github.com/smira/commander.(*Command).Dispatch(0xc820154240, 0xc82014fc10, 0x2, 0x3, 0x0, 0x0)
    /Users/smira/Documents/go/src/github.com/smira/aptly/_vendor/src/github.com/smira/commander/commands.go:283 +0x1f0
github.com/smira/commander.(*Command).Dispatch(0xc820154a20, 0xc82014fc00, 0x3, 0x4, 0x0, 0x0)
    /Users/smira/Documents/go/src/github.com/smira/aptly/_vendor/src/github.com/smira/commander/commands.go:283 +0x1f0
github.com/smira/aptly/cmd.Run(0xc820154a20, 0xc82000a210, 0x3, 0x3, 0xc3c601, 0x0)
    /Users/smira/Documents/go/src/github.com/smira/aptly/cmd/run.go:40 +0x19a
main.main()
    /Users/smira/Documents/go/src/github.com/smira/aptly/main.go:9 +0x73

goroutine 6 [syscall]:
os/signal.loop()
    /usr/local/Cellar/go/1.5.3/libexec/src/os/signal/signal_unix.go:22 +0x18
created by os/signal.init.1
    /usr/local/Cellar/go/1.5.3/libexec/src/os/signal/signal_unix.go:28 +0x37

My guess is some file got corrupted somehow. But I have no idea how to locate it and fix it (delete it?).

For now I will recover from backup but I will keep the corrupted data around for further testing.

smira commented 8 years ago

This sounds like goleveldb corruption, it happened with 0.8.3?

Gregy commented 8 years ago

It happened with aptly version 0.8 (from "aptly version" output). I think the -3 in 0.8-3 is the version of debian package itself.