search

ncbi / datasets

NCBI Datasets is a new resource that lets you easily gather data from across NCBI databases.
https://www.ncbi.nlm.nih.gov/datasets
Other
355 stars 39 forks source link

`--preview` option returns 'panic: send on closed channel' #374

Closed dtdoering closed 3 months ago

dtdoering commented 3 months ago

Before opening an issue, please:

Describe the bug A clear and concise description of what the bug is.

To Reproduce Please provide a reproducible example, with steps to reproduce the behavior and any relevant taxonomic, genome assembly, or gene identifiers.

Steps to reproduce the behavior:

  1. Update to latest version of datasets via conda: conda install -c conda-forge ncbi-datasets-cli

  2. Run commands:

    $ datasets download genome accession GCF_000009045.1 --preview
    See output ```txt Collecting 1 genome record [================================================] 100% 1/1 {"resource_updated_on":"2024-06-10T14:31:01Z","record_count":1,"estimated_file_size_mb":2,"included_data_files":{"all_genomic_fasta":{"file_count":1,"size_mb":1.1905632}}} panic: send on closed channel goroutine 1 [running]: github.com/gosuri/uiprogress.(*Progress).Stop(...) external/com_github_gosuri_uiprogress/progress.go:139 datasets_cli/v2/datasets.glob..func2(0xc00026d200?, {0xc0000e5140?, 0x1?, 0x2?}) apps/public/Datasets/v2/datasets/datasets/Download.go:104 +0x45 github.com/spf13/cobra.(*Command).execute(0xc00026d200, {0xc0000e5120, 0x2, 0x2}) external/com_github_spf13_cobra/command.go:960 +0xa2b github.com/spf13/cobra.(*Command).ExecuteC(0x1321920) external/com_github_spf13_cobra/command.go:1068 +0x3bc github.com/spf13/cobra.(*Command).Execute(...) external/com_github_spf13_cobra/command.go:992 datasets_cli/v2/datasets.Execute() apps/public/Datasets/v2/datasets/datasets/root.go:482 +0x25 main.main() apps/public/Datasets/v2/cmd/datasets/main.go:10 +0x17 ``` 
    $ datasets download genome accession GCF_000009045.1 --preview --debug
    See output ```txt 2024/06/10 13:03:07 POST /datasets/v2alpha/genome/dataset_report HTTP/1.1 Host: api.ncbi.nlm.nih.gov User-Agent: OpenAPI-Generator/1.0.0/go Content-Length: 766 Accept: application/json Content-Type: application/json Ncbi-Phid: 57798B214AAB11F180D9DC7A X-Datasets-Client: datasets-cli X-Datasets-Client-Arch: amd64 X-Datasets-Client-Cmd: download genome accession GCF_000009045.1 --preview --debug X-Datasets-Client-Os: linux X-Datasets-Client-Version: 16.20.0 Accept-Encoding: gzip {"accessions":["GCF_000009045.1"],"filters":{"assembly_source":"all","assembly_version":"all_assemblies","exclude_atypical":false,"exclude_multi_isolate":false,"has_annotation":false,"is_metagenome_derived":"METAGENOME_DERIVED_UNSET","is_type_material":false,"reference_only":false,"search_text":[],"type_material_category":"NONE"},"include_tabular_header":"INCLUDE_TABULAR_HEADER_FIRST_PAGE_ONLY","page_size":1,"returned_content":"ASSM_ACC","sort":[{"direction":"SORT_DIRECTION_ASCENDING","field":"organismName"},{"direction":"SORT_DIRECTION_DESCENDING","field":"isRefGenome"},{"direction":"SORT_DIRECTION_DESCENDING","field":"isRepGenome"},{"direction":"SORT_DIRECTION_DESCENDING","field":"isRefseq"},{"direction":"SORT_DIRECTION_ASCENDING","field":"accession"}]} 2024/06/10 13:03:07 HTTP/2.0 200 OK Content-Security-Policy: upgrade-insecure-requests Content-Type: application/json Date: Mon, 10 Jun 2024 20:03:07 GMT Grpc-Metadata-Via: h2 linkerd Ncbi-Phid: 57798B214AAB11F180D9DC7A.1.1 Server: Apache Strict-Transport-Security: max-age=31536000; includeSubDomains; preload X-Datasets-Version: 16.20.0 X-Ncbi-Next-Page-Token: eNrjColWSkpMzszJKS1WKC5NKsnMyQQzigv0kPglRXoKhmYWSjoKJUWlqToKaYk5xakwjpK7s1u8AQhYGpiY6hkqxQIA-JEa-A X-Ncbi-Total-Count: 1 X-Ua-Compatible: IE=Edge X-Xss-Protection: 1; mode=block 2024/06/10 13:03:07 POST /datasets/v2alpha/genome/dataset_report HTTP/1.1 Host: api.ncbi.nlm.nih.gov User-Agent: OpenAPI-Generator/1.0.0/go Content-Length: 769 Accept: application/json Content-Type: application/json Ncbi-Phid: 57798B214AAB11F180D9DC7A X-Datasets-Client: datasets-cli X-Datasets-Client-Arch: amd64 X-Datasets-Client-Cmd: download genome accession GCF_000009045.1 --preview --debug X-Datasets-Client-Os: linux X-Datasets-Client-Version: 16.20.0 Accept-Encoding: gzip {"accessions":["GCF_000009045.1"],"filters":{"assembly_source":"all","assembly_version":"all_assemblies","exclude_atypical":false,"exclude_multi_isolate":false,"has_annotation":false,"is_metagenome_derived":"METAGENOME_DERIVED_UNSET","is_type_material":false,"reference_only":false,"search_text":[],"type_material_category":"NONE"},"include_tabular_header":"INCLUDE_TABULAR_HEADER_FIRST_PAGE_ONLY","page_size":1000,"returned_content":"ASSM_ACC","sort":[{"direction":"SORT_DIRECTION_ASCENDING","field":"organismName"},{"direction":"SORT_DIRECTION_DESCENDING","field":"isRefGenome"},{"direction":"SORT_DIRECTION_DESCENDING","field":"isRepGenome"},{"direction":"SORT_DIRECTION_DESCENDING","field":"isRefseq"},{"direction":"SORT_DIRECTION_ASCENDING","field":"accession"}]} Collecting 1 genome record [------------------------------------------------] 0% 0/1 2024/06/10 13:03:07 HTTP/2.0 200 OK Content-Security-Policy: upgrade-insecure-requests Content-Type: application/json Date: Mon, 10 Jun 2024 20:03:07 GMT Grpc-Metadata-Via: h2 linkerd Ncbi-Phid: 57798B214AAB11F180D9DC7A.2.1 Server: Apache Strict-Transport-Security: max-age=31536000; includeSubDomains; preload X-Datasets-Version: 16.20.0 X-Ncbi-Total-Count: 1 X-Ua-Compatible: IE=Edge X-Xss-Protection: 1; mode=block 2024/06/10 13:03:07 POST /datasets/v2alpha/genome/download_summary HTTP/1.1 Host: api.ncbi.nlm.nih.gov User-Agent: OpenAPI-Generator/1.0.0/go Content-Length: 106 Accept: application/json Content-Type: application/json Ncbi-Phid: 57798B214AAB11F180D9DC7A X-Datasets-Client: datasets-cli X-Datasets-Client-Arch: amd64 X-Datasets-Client-Cmd: download genome accession GCF_000009045.1 --preview --debug X-Datasets-Client-Os: linux X-Datasets-Client-Version: 16.20.0 Accept-Encoding: gzip {"accessions":["GCF_000009045.1"],"hydrated":"FULLY_HYDRATED","include_annotation_type":["GENOME_FASTA"]} Collecting 1 genome record [================================================] 100% 1/1 2024/06/10 13:03:08 HTTP/2.0 200 OK Content-Security-Policy: upgrade-insecure-requests Content-Type: application/json Date: Mon, 10 Jun 2024 20:03:07 GMT Grpc-Metadata-Via: h2 linkerd Ncbi-Phid: 57798B214AAB11F180D9DC7A.3.1 Server: Apache Strict-Transport-Security: max-age=31536000; includeSubDomains; preload X-Datasets-Version: 16.20.0 X-Ua-Compatible: IE=Edge X-Xss-Protection: 1; mode=block Collecting 1 genome record [================================================] 100% 1/1 {"resource_updated_on":"2024-06-10T14:31:01Z","record_count":1,"estimated_file_size_mb":2,"included_data_files":{"all_genomic_fasta":{"file_count":1,"size_mb":1.1905632}}} panic: send on closed channel goroutine 1 [running]: github.com/gosuri/uiprogress.(*Progress).Stop(...) external/com_github_gosuri_uiprogress/progress.go:139 datasets_cli/v2/datasets.glob..func2(0xc00026d200?, {0xc000288600?, 0x1?, 0x3?}) apps/public/Datasets/v2/datasets/datasets/Download.go:104 +0x45 github.com/spf13/cobra.(*Command).execute(0xc00026d200, {0xc0002885d0, 0x3, 0x3}) external/com_github_spf13_cobra/command.go:960 +0xa2b github.com/spf13/cobra.(*Command).ExecuteC(0x1321920) external/com_github_spf13_cobra/command.go:1068 +0x3bc github.com/spf13/cobra.(*Command).Execute(...) external/com_github_spf13_cobra/command.go:992 datasets_cli/v2/datasets.Execute() apps/public/Datasets/v2/datasets/datasets/root.go:482 +0x25 main.main() apps/public/Datasets/v2/cmd/datasets/main.go:10 +0x17 ```

Expected behavior

Other info

OS: Rocky Linux v8.9 Conda version: 24.3.0

ericcox1 commented 3 months ago

Hi @dtdoering,

Thanks for opening this issue. I was able to reproduce this bug and we will look into it. I'll ping this thread with any updates.

Best, Eric

Eric Cox, PhD [Contractor] (he/him/his) NCBI Datasets NIH/NLM/NCBI eric.cox@nih.gov

ericcox1 commented 3 months ago

Hi @dtdoering,

This bug appears to be related to the progress bar. As a temporary workaround, you can use --no-progressbar:

datasets download genome accession GCF_000009045.1 --preview --no-progressbar
{"resource_updated_on":"2024-06-11T18:13:29Z","record_count":1,"estimated_file_size_mb":2,"included_data_files":{"all_genomic_fasta":{"file_count":1,"size_mb":1.1905632}}}

Best, Eric

ericcox1 commented 3 months ago

Hi @dtdoering,

This bug is now fixed in the latest version:

datasets version; datasets download genome accession GCF_000009045.1 --preview
datasets version: 16.22.1
Collecting 1 genome record [================================================] 100% 1/1
{"resource_updated_on":"2024-06-25T02:30:14Z","record_count":1,"estimated_file_size_mb":2,"included_data_files":{"all_genomic_fasta":{"file_count":1,"size_mb":1.1905632}}}

Thanks again for your report.

Best, Eric