QUERYNATOR_CIVICAPI: 'DataFrame' object has no attribute 'append'

HomoPolyethylen commented 3 months ago

Description of the bug

civic query fails with variantMTB test data, but not with querynator test data.
throws error 'DataFrame' object has no attribute 'append'
found DataFrame.append in Pandas 1.3 docs
querynator uses Pandas 1.5.3, DataFrame.append not found in docs

Command used and terminal output

nextflow run . -profile docker,test -resume --databases civic

executor >  local (1)
[-        ] NFCORE_VARIANTMTB:VARIANTMTB:TABIX_TABIX                       -
[80/129f89] NFCORE_VARIANTMTB:VARIANTMTB:TABIX_BGZIPTABIX (test_sample)    [100%] 1 of 1, cached: 1 ✔
[db/70bd26] NFCORE_VARIANTMTB:VARIANTMTB:BCFTOOLS_NORM (test_sample)       [100%] 1 of 1, cached: 1 ✔
[30/e6480a] NFCORE_VARIANTMTB:VARIANTMTB:QUERYNATOR_CIVICAPI (test_sample) [100%] 1 of 1, failed: 1 ✘
[-        ] NFCORE_VARIANTMTB:VARIANTMTB:CUSTOM_DUMPSOFTWAREVERSIONS       -
Execution cancelled -- Finishing pending tasks before exit
-[nf-core/variantmtb] Pipeline completed with errors-
ERROR ~ Error executing process > 'NFCORE_VARIANTMTB:VARIANTMTB:QUERYNATOR_CIVICAPI (test_sample)'

Caused by:
  Process `NFCORE_VARIANTMTB:VARIANTMTB:QUERYNATOR_CIVICAPI (test_sample)` terminated with an error exit status (1)

Command executed:

  # set path to civicpy cache
  export CIVICPY_CACHE_FILE=/home/zxmgc83/variantmtb/work/.civicpy/cache.pkl
  export MPLCONFIGDIR=/home/zxmgc83/variantmtb/work/.config/matplotlib

  # run querynator
  querynator query-api-civic \
      --vcf test_sample.normalized.vcf.gz \
      --outdir test_sample_civic \
      --genome GRCh37 \
      --filter_vep \
       \

  cat <<-END_VERSIONS > versions.yml
  "NFCORE_VARIANTMTB:VARIANTMTB:QUERYNATOR_CIVICAPI":
      querynator: $(echo $(querynator --version 2>&1) | sed 's/^.*querynator //; s/Using.*$//' ))
  END_VERSIONS

Command exit status:
  1

Command output:

                                             __ 
    ____ ___  _____  _______  ______  ____ _/ /_____  _____
   / __ `/ / / / _ \/ ___/ / / / __ \/ __ `/ __/ __ \/ ___/
  / /_/ / /_/ /  __/ /  / /_/ / / / / /_/ / /_/ /_/ / /
  \__, /\__,_/\___/_/   \__, /_/ /_/\__,_/\__/\____/_/
    /_/                /____/

Command error:
  2024-07-05 09:25:08,672 - Querynator - INFO - Start
  2024-07-05 09:25:08,683 - Querynator - INFO - Unzipping input file (test_sample.normalized.vcf.gz)
  2024-07-05 09:25:08,686 - Querynator - INFO - Filtering vcf file
  2024-07-05 09:25:08,690 - Querynator - INFO - Query the Clinical Interpretations of Variants In Cancer (CIViC)
  2024-07-05 09:25:08,690 - Querynator - INFO - Updating CIViCpy Cache
  2024-07-05 09:25:09,479 - Querynator - INFO - Querying
  Traceback (most recent call last):
    File "/usr/local/bin/querynator", line 10, in <module>
      sys.exit(run_querynator())
    File "/usr/local/lib/python3.10/site-packages/querynator/__main__.py", line 278, in run_querynator
      querynator_cli()
    File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1157, in __call__

                                             __ 
    ____ ___  _____  _______  ______  ____ _/ /_____  _____
   / __ `/ / / / _ \/ ___/ / / / __ \/ __ `/ __/ __ \/ ___/
  / /_/ / /_/ /  __/ /  / /_/ / / / / /_/ / /_/ /_/ / /
  \__, /\__,_/\___/_/   \__, /_/ /_/\__,_/\__/\____/_/
    /_/                /____/

      return self.main(*args, **kwargs)
    File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1078, in main
      rv = self.invoke(ctx)
    File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1688, in invoke
      return _process_result(sub_ctx.command.invoke(sub_ctx))
    File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1434, in invoke
      return ctx.invoke(self.callback, **ctx.params)
    File "/usr/local/lib/python3.10/site-packages/click/core.py", line 783, in invoke
      return __callback(*args, **kwargs)
    File "/usr/local/lib/python3.10/site-packages/querynator/__main__.py", line 450, in query_api_civic
      query_civic(candidate_variants, result_dir, logger, vcf, genome, cancer, filter_vep)
    File "/usr/local/lib/python3.10/site-packages/querynator/query_api/civic_api.py", line 652, in query_civic
      create_civic_results(access_civic_by_coordinate(coord_dict, logger, genome), out_path, disease, logger, filter_vep)
    File "/usr/local/lib/python3.10/site-packages/querynator/query_api/civic_api.py", line 544, in create_civic_results
      civic_result_df = civic_result_df.append(
    File "/usr/local/lib/python3.10/site-packages/pandas/core/generic.py", line 6299, in __getattr__
      return object.__getattribute__(self, name)
  AttributeError: 'DataFrame' object has no attribute 'append'. Did you mean: '_append'?

Work dir:
  /home/zxmgc83/variantmtb/work/30/e6480a29189618c823e3c63d5ef1bb

Tip: you can replicate the issue by changing to the process work dir and entering the command `bash .command.run`

 -- Check '.nextflow.log' file for details

System information

No response

HomoPolyethylen commented 3 months ago

debugging with *querynator test data:

Pandas version 1.5.3
DataFrame.append exists.
runs through without errors :heavy_check_mark:

debugging with variantMTB test data:

Pandas version 1.5.3
DataFrame.append exists
runs through without errors :heavy_check_mark: >:(

running the querynator 1.5.1 docker container in the nextflow workdir

on variantMTB test data :x:
on querynator test data :x: --> some docker issue?

HomoPolyethylen commented 3 months ago

version in container is different from qn-debug env: 2.2.2 instead of 1.5.3

ran container in nxf workdir and within running container, executed:
```
pip list | grep pandas
```

HomoPolyethylen commented 3 months ago

keep compatible with pandas 2.2.2

[ ] update pandas to 2.2.2 in qn-debug env
[ ] fix issues
[ ] replace deprecated DataFrame.append with pandas.concat

OR:

[x] pinn pandas to 1.5.3

HomoPolyethylen commented 3 months ago

civicpy also breaks with pandas 2.x --> pin pandas to 1.5.3

qbic-pipelines / querynator