Open kurt-mueller-osumc opened 3 years ago
Can you try a more recent release? dnanexus/dxda:sha-c5d258d
should contain the latest release v0.5.7.
No joy :(
docker-compose run --rm dxda download -max_threads=24 manifest.json.bz2
Pulling dxda (dnanexus/dxda:sha-c5d258d)...
sha-c5d258d: Pulling from dnanexus/dxda
61e03ba1d414: Pull complete
4afb39f216bd: Pull complete
e489abdc9f90: Pull complete
999fff7bcc24: Pull complete
a61d29dbfa18: Pull complete
3a9b3d1a0262: Pull complete
6d2539d5d99a: Pull complete
Digest: sha256:9a7f1b94a536f3647650b862f2a1c144aa4f1b501ccdb616574ca6b578ea9be3
Status: Downloaded newer image for dnanexus/dxda:sha-c5d258d
Creating aostor-dxda_dxda_run ... done
Logging detailed output to: manifest.json.bz2.download.log
Obtained token using environment
Downloading files using 24 threads
maximal memory chunk size: 16 MiB
Creating manifest database manifest.json.bz2.stats.db
panic: database is locked
goroutine 1 [running]:
github.com/dnanexus/dxda.check(...)
/go/src/github.com/dnanexus/dxda/util.go:149
github.com/dnanexus/dxda.(*State).CreateManifestDB(0xc000106100, 0xc002470000, 0x19d9, 0x1e00, 0x7fffae6a8f08, 0x1c)
/go/src/github.com/dnanexus/dxda/dxda.go:362 +0x62a
main.(*downloadCmd).Execute(0xc00001a300, 0x92f640, 0xc00001a038, 0xc0000143c0, 0x0, 0x0, 0x0, 0x0)
/go/src/github.com/dnanexus/dxda/cmd/dx-download-agent/dx-download-agent.go:85 +0x63b
github.com/google/subcommands.(*Commander).Execute(0xc000106000, 0x92f640, 0xc00001a038, 0x0, 0x0, 0x0, 0xc00008df50)
/go/src/github.com/google/subcommands/subcommands.go:209 +0x2f9
github.com/google/subcommands.Execute(...)
/go/src/github.com/google/subcommands/subcommands.go:492
main.main()
/go/src/github.com/dnanexus/dxda/cmd/dx-download-agent/dx-download-agent.go:214 +0x21e
ERROR: 2
When I rerun the command, dxda will start downloading the files.... however, the log / database doesn't update.
I also receive the locked database panic error. Looks like the it creates the manifest/stats DB file, but no tables are created.
Rerunning the command:
Logging detailed output to: manifest.json.bz2.download.log
Obtained token using environment
Downloading files using 20 threads
maximal memory chunk size: 16 MiB
All files have parts, assuming they are not archived or open
Ensuring files are created for existing manifest
panic: no such table: manifest_regular_stats
goroutine 1 [running]:
github.com/dnanexus/dxda.check(...)
/go/src/github.com/dnanexus/dxda/util.go:149
github.com/dnanexus/dxda.(*State).queryDBIntegerResult(0xc0000d0100, 0x8bbf89, 0x48, 0x0)
/go/src/github.com/dnanexus/dxda/dxda.go:220 +0x221
github.com/dnanexus/dxda.(*State).CheckDiskSpace(0xc0000d0100, 0xc00007c210, 0x1)
/go/src/github.com/dnanexus/dxda/dxda.go:262 +0x55
main.(*downloadCmd).Execute(0xc0000862b0, 0x92f640, 0xc000086000, 0xc00009a2a0, 0x0, 0x0, 0x0, 0x0)
/go/src/github.com/dnanexus/dxda/cmd/dx-download-agent/dx-download-agent.go:91 +0x64c
github.com/google/subcommands.(*Commander).Execute(0xc0000d0000, 0x92f640, 0xc000086000, 0x0, 0x0, 0x0, 0xc000040f50)
/go/src/github.com/google/subcommands/subcommands.go:209 +0x2f9
github.com/google/subcommands.Execute(...)
/go/src/github.com/google/subcommands/subcommands.go:492
main.main()
/go/src/github.com/dnanexus/dxda/cmd/dx-download-agent/dx-download-agent.go:214 +0x21e
It appears to create the files as if they were downloaded, but the files appear to be empty.
@jstnlvns could you please provide the dxda version and OS information?
Apologies, I had that info in there but must have removed it when I reformatted. I have this running on a server running Ubuntu 18.04.6 (64-bit) using Docker, and I am running the latest version of this repo. After reading your response to @kurt-mueller-osumc, I recloned the repo and rebuilt the Docker image using the provided Dockerfile and tried to rerun the download and received the same result.
I've been attempting to use the docker image to downlad files using a manifest... however, I keep getting a
panic: database is locked
error:Here's my
docker-compose.yml
file: