overture-stack / score

Secure Cloud Object REsource: file transfer microservice
https://www.overture.bio/products/score
GNU Affero General Public License v3.0
18 stars 10 forks source link

BUG - Resuming downloads not working #394

Open jmimico opened 7 months ago

jmimico commented 7 months ago

Only occurs if I have an incomplete download living in my download directory ie; /root/download_temp/.b7f6687e-bd7c-5f49-b836-857a5cf05e87

docker run --rm -e ACCESSTOKEN=<sanitized> -e METADATA_URL=https://song.azure.icgc.org -e STORAGE_URL=https://score.azure.icgc.org -e TRANSPORT_PARALLEL=13 -e TRANSPORT_MEMORY=1 -v /root/score-client/logs:/score-client/logs -v /root/download_temp/:/data --privileged ghcr.io/overture-stack/score:latest bin/score-client --profile azure download --object-id b7f6687e-bd7c-5f49-b836-857a5cf05e87 --output-dir /data --validate false
Downloading...
--------------------------------------------------------------------------------
[1/2] Downloading object: b7f6687e-bd7c-5f49-b836-857a5cf05e87 (PCAWG.b9bf94f6-dd37-11e4-971f-fdaac254ba06.TopHat2.v1.bam)
--------------------------------------------------------------------------------
ERROR: Command error: Could not parse presigned URL - missing expected expiry date parameters

Please check the log for detailed error messages

2023-12-12 21:34:07,911 [main] WARN  b.o.s.c.c.DownloadCommand - Space required: 4.5 GB (4543403991)  Space available: 616.0 GB (616036356096)
2023-12-12 21:34:07,955 [main] ERROR b.o.s.c.u.S3PresignedUrlValidator - Could not identify expected date parameters in request: sig=tLwgSqFWmzuzZCArvhcoj6s92u3hkaQlwDCI+5DFuqg= st=2023-12-12T21:18:57Z se=2023-12-13T21:33:57Z sv=2015-12-11 si=readonly sr=b
2023-12-12 21:34:07,957 [main] ERROR b.o.s.c.ClientMain - Unknown error:
java.lang.IllegalArgumentException: Could not parse presigned URL - missing expected expiry date parameters
    at bio.overture.score.client.util.S3PresignedUrlValidator.extractExpiryDate(S3PresignedUrlValidator.java:57)
    at bio.overture.score.client.util.PresignedUrlValidator.extractExpiryDate(PresignedUrlValidator.java:67)
    at bio.overture.score.client.util.PresignedUrlValidator.getExpiry(PresignedUrlValidator.java:58)
    at bio.overture.score.client.util.PresignedUrlValidator.isUrlExpired(PresignedUrlValidator.java:44)
    at bio.overture.score.client.util.PresignedUrlValidator.isUrlExpired(PresignedUrlValidator.java:40)
    at bio.overture.score.client.download.DownloadStateStore.getProgress(DownloadStateStore.java:98)
    at bio.overture.score.client.download.DownloadService.resumeIfPossible(DownloadService.java:158)
    at bio.overture.score.client.download.DownloadService.download(DownloadService.java:122)
    at bio.overture.score.client.command.DownloadCommand.downloadObjects(DownloadCommand.java:220)
    at bio.overture.score.client.command.DownloadCommand.execute(DownloadCommand.java:136)
    at bio.overture.score.client.ClientMain.run(ClientMain.java:137)
    at org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:779)
    at org.springframework.boot.SpringApplication.callRunners(SpringApplication.java:763)
    at org.springframework.boot.SpringApplication.run(SpringApplication.java:318)
    at org.springframework.boot.builder.SpringApplicationBuilder.run(SpringApplicationBuilder.java:139)
    at bio.overture.score.client.ClientMain.main(ClientMain.java:110)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:568)
    at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:47)
    at org.springframework.boot.loader.Launcher.launch(Launcher.java:86)
    at org.springframework.boot.loader.Launcher.launch(Launcher.java:50)
    at org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:51)
jmimico commented 7 months ago

Thanks to @UmmulkiramR when I specify the -e DEFAULTPROFILE=azure resuming downloads works!