search

cgat-developers / cgat-flow

cgat-flow repository
MIT License
13 stars 9 forks source link

Js rnaseqqc #115

Closed Acribbs closed 4 years ago

Acribbs commented 4 years ago

@jscaber the testing pipeline needs updating and I was thinking of making a separate branch to do this. However, I started on this branch. I will merge this and make a separate branch

jscaber commented 4 years ago

Great, do you want to wait, I’m refactoring this pipeline to change the various inputs and remove sailfish?

Acribbs commented 4 years ago

ok, if your refactoring this pipeline then we can wait to merge. Will set another branch called testing for making changes to testing pipeline

jscaber commented 4 years ago

Hi, can we add r-wasabi to the current conda build? It is a requirement for the rnaseqdiffexperssion pipeline. As part of removing some of the duplication of the code I decided to run the salmon quantifier from rnaseq.py in the ranseqqc pipeline. This gets around the problem that sailfish used to generate both transcript and gene counts whereas salmon no longer offers transcript2genemap conversions and therefore only outputs transcript counts (gene count instead done by tximport directly in R now). This is all for good reasons, however for the rnaseqqc pipeline depends on gene counts. The downside of using the rnaseq.py code is that this behemoth created by Tom many moons ago, allows for conversion between all types of transcript/gene abundance calculators and all types of differenential expression tools and therefore requires the conversion of salmon to kallisto format by default. R Wasabi is required for that.

Acribbs commented 4 years ago

Ah I hadn’t realised and it makes perfect sense to use r-wasabi. Just add it to the conda requirements in rnaseqdiffexpression.

Would we need a complete refactoring of rnaseqdiffexpressiin? Happy to help as I use this pipeline quite a lot

jscaber commented 4 years ago

It's already in rnaseqdiffexpression (and in the pipeline_rnaseqdiffexpression.yml, where it is actually hashed out). I would now need to add it to pipeline_rnaseqqc.yml?

I did try to install it and it completely changes the conda environment and results in 50-100 package updates and installs

Given that it's an integral part of running salmon in rnaseqdiffexpression.py and the pipeline fails without r-wasabi, how does travis get through ranseqdiffexpression tests (is salmon run in the tests?)?

sebastian-luna-valero commented 4 years ago

Hi,

I would like to have a look at this with more detail after work.

Travis do not run any pipeline tests. Those are performed by Jenkins, which I am troubleshooting at the moment.

I will add https://anaconda.org/bioconda/r-wasabi to the conda env asap.

Best regards, Sebastian

PS: As per your environment, please share what command you use to install it and the conda output.

jscaber commented 4 years ago

Yes, apologies I meant Jenkins, I still misremember which is which. conda install -c bioconda r-wasabi

jscaber commented 4 years ago

I did not install it because of the many conflicts.

sebastian-luna-valero commented 4 years ago

Did you install your cgat environment with the installer?

Could you please share the output of conda info?

jscaber commented 4 years ago 
     active environment : cgat-flow
    active env location : /XXX/cgat-developers-v2/conda-install/envs/cgat-flow
            shell level : 2
       user config file : ~/.condarc
 populated config files : ~/.condarc
          conda version : 4.7.12
    conda-build version : not installed
         python version : 3.7.3.final.0
       virtual packages : 
       base environment : /XXX/conda-install  (writable)
           channel URLs : https://conda.anaconda.org/conda-forge/linux-64
                          https://conda.anaconda.org/conda-forge/noarch
                          https://conda.anaconda.org/bioconda/linux-64
                          https://conda.anaconda.org/bioconda/noarch
                          https://conda.anaconda.org/cgat/linux-64
                          https://conda.anaconda.org/cgat/noarch
                          https://repo.anaconda.com/pkgs/main/linux-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/r/linux-64
                          https://repo.anaconda.com/pkgs/r/noarch
          package cache : /XXX/conda-install/pkgs
                          ~/.conda/pkgs
       envs directories : /XXX/conda-install/envs
                         ~/.conda/envs
               platform : linux-64
             user-agent : conda/4.7.12 requests/2.22.0 CPython/3.7.3 Linux/2.6.32-754.el6.x86_64 scientific/6.10 glibc/2.12
                UID:GID : 1400:1000
             netrc file : None
           offline mode : False
jscaber commented 4 years ago

Recent re-install, I do believe I made a minor change by updating libboost on cgat-flow but otherwise no changes to conda from the original install.

sebastian-luna-valero commented 4 years ago

Not sure you should have:

                          https://conda.anaconda.org/cgat/linux-64
                          https://conda.anaconda.org/cgat/noarch

in your .condarc file. Could you please share cat ~/.condarc

Anyway, could you please try instead:

conda install --dry-run -c conda-forge -c bioconda -c defaults r-wasabi

and share the output?

sebastian-luna-valero commented 4 years ago

Sorry, forgot a key flag --no-update-deps:

conda install --dry-run --no-update-deps -c conda-forge -c bioconda -c defaults r-wasabi
Acribbs commented 4 years ago

r-wasabi has very few dependancies, however it depends on bioconductor-rhdf5 and in the past I have had a lot of problems with installing conda packages that depend on this (Think it was the tximport or the one of the packages for Deseq2, I cant remember which ones now though).

jscaber commented 4 years ago

and share the output?

- conda-forge
- https://conda.anaconda.org/bioconda
- https://conda.anaconda.org/cgat
- defaults
sebastian-luna-valero commented 4 years ago

Thanks, please also share the output of:

conda install --dry-run --no-update-deps -c conda-forge -c bioconda -c defaults r-wasabi
jscaber commented 4 years ago

Thanks, please also share the output of:

conda install --dry-run --no-update-deps -c conda-forge -c bioconda -c defaults r-wasabi

Yes, coming, it's still solving... takes about 20 minutes.

sebastian-luna-valero commented 4 years ago

In the meantime, I suggest you have update your ~/.condarc with:

  - conda-forge
  - bioconda
  - defaults

Also, if you have a recent re-install, could you please share conda list rpy2

jscaber commented 4 years ago 
Collecting package metadata (current_repodata.json): done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.
Collecting package metadata (repodata.json): done
Solving environment: done

## Package Plan ##

  environment location: /ifs/projects/jakubs/cgat-developers-v2/conda-install/envs/cgat-flow

  added / updated specs:
    - r-wasabi

The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    bioconductor-annotate-1.60.1|           r351_0         2.1 MB  bioconda
    bioconductor-annotationdbi-1.44.0|           r351_0         4.9 MB  bioconda
    bioconductor-biobase-2.42.0|   r351h14c3975_1         2.3 MB  bioconda
    bioconductor-biocgenerics-0.28.0|           r351_1         695 KB  bioconda
    bioconductor-biocparallel-1.16.6|   r351h1c2f66e_0         1.1 MB  bioconda
    bioconductor-biomart-2.38.0|           r351_0         526 KB  bioconda
    bioconductor-biostrings-2.50.2|   r351h14c3975_0        13.8 MB  bioconda
    bioconductor-biovizbase-1.18.0|                1         2.2 MB  bioconda
    bioconductor-bsgenome-1.50.0|           r351_0         6.9 MB  bioconda
    bioconductor-cummerbund-2.12.1|           r351_1         2.8 MB  bioconda
    bioconductor-delayedarray-0.8.0|   r351h14c3975_0         1.6 MB  bioconda
    bioconductor-deseq-1.34.1  |   r351h14c3975_0         1.8 MB  bioconda
    bioconductor-deseq2-1.22.1 |   r351hf484d3e_0         2.5 MB  bioconda
    bioconductor-edger-3.26.0  |   r351hf484d3e_0         2.2 MB  bioconda
    bioconductor-genefilter-1.64.0|   r351h1c2f66e_1         1.7 MB  bioconda
    bioconductor-geneplotter-1.60.0|           r351_0         1.5 MB  bioconda
    bioconductor-genomeinfodb-1.18.1|           r351_0         3.8 MB  bioconda
    bioconductor-genomeinfodbdata-1.2.1|           r351_0           6 KB  bioconda
    bioconductor-genomicalignments-1.18.1|   r351h14c3975_0         2.2 MB  bioconda
    bioconductor-genomicfeatures-1.34.1|           r351_0         2.1 MB  bioconda
    bioconductor-genomicranges-1.34.0|   r351h14c3975_0         2.1 MB  bioconda
    bioconductor-gviz-1.14.2   |                0         4.0 MB  bioconda
    bioconductor-hpar-1.26.0   |           r351_0         7.2 MB  bioconda
    bioconductor-iranges-2.16.0|   r351h14c3975_0         2.3 MB  bioconda
    bioconductor-limma-3.40.0  |   r351h14c3975_0         2.6 MB  bioconda
    bioconductor-rhdf5-2.28.0  |   r351hf484d3e_0         2.4 MB  bioconda
    bioconductor-rhdf5lib-1.6.0|   r351h14c3975_0         3.6 MB  bioconda
    bioconductor-rsamtools-1.34.0|   r351hf484d3e_0         3.8 MB  bioconda
    bioconductor-rtracklayer-1.42.1|   r351h9d9f1b6_1         2.7 MB  bioconda
    bioconductor-s4vectors-0.20.1|   r351h14c3975_0         2.0 MB  bioconda
    bioconductor-summarizedexperiment-1.12.0|           r351_0         2.6 MB  bioconda
    bioconductor-variantannotation-1.28.3|   r351h14c3975_0         3.3 MB  bioconda
    bioconductor-xvector-0.22.0|   r351h14c3975_0         711 KB  bioconda
    bioconductor-zlibbioc-1.28.0|   r351h14c3975_0         112 KB  bioconda
    certifi-2019.9.11          |           py36_0         147 KB  conda-forge
    cloog-0.18.0               |                0         565 KB
    gcc-4.8.3                  |                1        66.5 MB  cgat
    gmp-6.1.2                  |    hf484d3e_1000         751 KB  conda-forge
    isl-0.21                   |       he80fd80_0         1.5 MB  conda-forge
    mpc-1.1.0                  |    h04dde30_1006         149 KB  conda-forge
    mpfr-4.0.2                 |       he80fd80_0         673 KB  conda-forge
    openssl-1.1.1d             |       h516909a_0         2.1 MB  conda-forge
    r-3.1.2                    |                2        18.5 MB
    r-acepack-1.4.1            | r35h9bbef5b_1004          69 KB  conda-forge
    r-assertthat-0.2.1         |    r35h6115d3f_1          69 KB  conda-forge
    r-backports-1.1.5          |    r35hcdcec82_0          67 KB  conda-forge
    r-base-3.5.1               |    h08e1455_1008        39.2 MB  conda-forge
    r-base64enc-0.1_3          | r35hcdcec82_1003          43 KB  conda-forge
    r-bh-1.69.0_1              |    r35h6115d3f_1        10.7 MB  conda-forge
    r-bit-1.1_14               |    r35hcdcec82_1         243 KB  conda-forge
    r-bit64-0.9_7              | r35hcdcec82_1001         477 KB  conda-forge
    r-bitops-1.0_6             | r35hcdcec82_1003          39 KB  conda-forge
    r-blob-1.1.1               |        r351_1001          27 KB  conda-forge
    r-catools-1.17.1.2         |    r35h0357c0b_1         223 KB  conda-forge
    r-checkmate-1.9.4          |    r35hcdcec82_1         647 KB  conda-forge
    r-cli-1.1.0                |    r35h6115d3f_2         177 KB  conda-forge
    r-cluster-2.1.0            |    r35h9bbef5b_2         551 KB  conda-forge
    r-colorspace-1.4_1         |    r35hcdcec82_1         2.5 MB  conda-forge
    r-crayon-1.3.4             | r35h6115d3f_1002         745 KB  conda-forge
    r-crosstalk-1.0.0          | r35h6115d3f_1002         596 KB  conda-forge
    r-curl-4.2                 |    r35hcdcec82_0         697 KB  conda-forge
    r-data.table-1.12.6        |    r35hcdcec82_0         1.8 MB  conda-forge
    r-dbi-1.0.0                | r35h6115d3f_1002         883 KB  conda-forge
    r-dichromat-2.0_0          |         r35_2001         151 KB  conda-forge
    r-digest-0.6.22            |    r35h0357c0b_0         192 KB  conda-forge
    r-dt-0.9                   |    r35h6115d3f_0         888 KB  conda-forge
    r-evaluate-0.14            |    r35h6115d3f_1          79 KB  conda-forge
    r-fansi-0.4.0              | r35hcdcec82_1001         192 KB  conda-forge
    r-fastcluster-1.1.25       | r35hc99cbb1_1002         191 KB  conda-forge
    r-foreign-0.8_72           |    r35hcdcec82_0         266 KB  conda-forge
    r-formatr-1.7              |    r35h6115d3f_1         165 KB  conda-forge
    r-formula-1.2_3            | r35h6115d3f_1002         192 KB  conda-forge
    r-futile.logger-1.4.3      | r35h6115d3f_1002         107 KB  conda-forge
    r-futile.options-1.0.1     | r35h6115d3f_1001          23 KB  conda-forge
    r-gdata-2.18.0             | r35h6115d3f_1002         1.1 MB  conda-forge
    r-ggplot2-2.2.1            |   r351h6115d3f_1         3.1 MB  conda-forge
    r-glue-1.3.1               |    r35hcdcec82_1         165 KB  conda-forge
    r-gmd-0.3.3                | r35h516909a_1002         505 KB  conda-forge
    r-gplots-3.0.1.1           |    r35h6115d3f_1         656 KB  conda-forge
    r-gridextra-2.3            | r35h6115d3f_1002         1.1 MB  conda-forge
    r-gsalib-2.1               |         r35_1001          29 KB  conda-forge
    r-gtable-0.3.0             |    r35h6115d3f_2         420 KB  conda-forge
    r-gtools-3.8.1             | r35hcdcec82_1003         324 KB  conda-forge
    r-highr-0.8                |    r35h6115d3f_1          47 KB  conda-forge
    r-hmisc-4.2_0              |    r35h9bbef5b_2         3.0 MB  conda-forge
    r-hms-0.4.2                |r351h6115d3f_1000          71 KB  conda-forge
    r-htmltable-1.13.2         |    r35h6115d3f_0         328 KB  conda-forge
    r-htmltools-0.4.0          |    r35h0357c0b_0         213 KB  conda-forge
    r-htmlwidgets-1.5.1        |    r35h6115d3f_0         674 KB  conda-forge
    r-httpuv-1.5.2             |    r35h0357c0b_1         855 KB  conda-forge
    r-httr-1.4.1               |    r35h6115d3f_1         485 KB  conda-forge
    r-jsonlite-1.6             | r35hcdcec82_1001        1006 KB  conda-forge
    r-kernsmooth-2.23_15       | r35h9bbef5b_1004         102 KB  conda-forge
    r-knitr-1.25               |    r35h6115d3f_0         1.3 MB  conda-forge
    r-labeling-0.3             | r35h6115d3f_1002          65 KB  conda-forge
    r-lambda.r-1.2.4           |    r35h6115d3f_0         118 KB  conda-forge
    r-later-1.0.0              |    r35h0357c0b_0         140 KB  conda-forge
    r-lattice-0.20_38          | r35hcdcec82_1002         1.1 MB  conda-forge
    r-latticeextra-0.6_28      | r35h6115d3f_1002         2.1 MB  conda-forge
    r-lazyeval-0.2.2           |    r35hcdcec82_1         163 KB  conda-forge
    r-locfit-1.5_9.1           | r35h516909a_1004         553 KB  conda-forge
    r-magrittr-1.5             | r35h6115d3f_1002         165 KB  conda-forge
    r-markdown-1.1             |    r35hcdcec82_0         143 KB  conda-forge
    r-mass-7.3_51.4            |    r35hcdcec82_1         1.1 MB  conda-forge
    r-matrix-1.2_17            |    r35hcdcec82_1         3.8 MB  conda-forge
    r-matrixstats-0.55.0       |    r35hcdcec82_0         904 KB  conda-forge
    r-memoise-1.1.0            | r35h6115d3f_1003          41 KB  conda-forge
    r-mime-0.7                 |    r35hcdcec82_1          51 KB  conda-forge
    r-munsell-0.5.0            | r35h6115d3f_1002         244 KB  conda-forge
    r-nnet-7.3_12              | r35hcdcec82_1003         130 KB  conda-forge
    r-openssl-1.1              |r351h3a9d887_1002         1.1 MB  conda-forge
    r-pillar-1.3.0             |   r351h6115d3f_0         152 KB
    r-pkgconfig-2.0.3          |    r35h6115d3f_0          24 KB  conda-forge
    r-plogr-0.2.0              | r35h6115d3f_1002          18 KB  conda-forge
    r-plotrix-3.7_6            |    r35h6115d3f_1         1.1 MB  conda-forge
    r-plyr-1.8.4               | r35h0357c0b_1003         814 KB  conda-forge
    r-prettyunits-1.0.2        | r35h6115d3f_1002          35 KB  conda-forge
    r-progress-1.2.2           |    r35h6115d3f_1          87 KB  conda-forge
    r-promises-1.1.0           |    r35h0357c0b_0         1.2 MB  conda-forge
    r-r6-2.4.0                 |    r35h6115d3f_2          62 KB  conda-forge
    r-rcolorbrewer-1.1_2       | r35h6115d3f_1002          59 KB  conda-forge
    r-rcpp-1.0.3               |    r35h0357c0b_0         1.9 MB  conda-forge
    r-rcpparmadillo-0.9.800.1.0|    r35h0357c0b_0         1.2 MB  conda-forge
    r-rcurl-1.95_4.12          |    r35hcdcec82_1         912 KB  conda-forge
    r-reshape-0.8.8            |    r35hcdcec82_1         173 KB  conda-forge
    r-reshape2-1.4.3           | r35h0357c0b_1004         129 KB  conda-forge
    r-rjson-0.2.20             | r35h0357c0b_1001         145 KB  conda-forge
    r-rlang-0.4.1              |    r35hcdcec82_0        1021 KB  conda-forge
    r-rmarkdown-1.16           |    r35h6115d3f_0         3.0 MB  conda-forge
    r-rpart-4.1_15             |    r35hcdcec82_1         737 KB  conda-forge
    r-rsqlite-2.1.2            |    r35h0357c0b_1         1.0 MB  conda-forge
    r-rstudioapi-0.10          |    r35h6115d3f_2         217 KB  conda-forge
    r-scales-1.0.0             | r35h0357c0b_1002         576 KB  conda-forge
    r-shiny-1.3.2              |    r35h6115d3f_1         4.2 MB  conda-forge
    r-snow-0.4_3               | r35h6115d3f_1001         122 KB  conda-forge
    r-sourcetools-0.1.7        | r35he1b5a44_1001          51 KB  conda-forge
    r-stringi-1.4.3            |    r35h0357c0b_2         766 KB  conda-forge
    r-stringr-1.4.0            |    r35h6115d3f_1         207 KB  conda-forge
    r-survival-2.44_1.1        |    r35hcdcec82_1         4.8 MB  conda-forge
    r-tibble-1.4.2             |r351h96ca727_1002         229 KB  conda-forge
    r-tinytex-0.17             |    r35h6115d3f_0         103 KB  conda-forge
    r-utf8-1.1.4               | r35hcdcec82_1001         158 KB  conda-forge
    r-venndiagram-1.6.20       | r35h6115d3f_1001         263 KB  conda-forge
    r-viridis-0.5.1            | r35h6115d3f_1003         1.8 MB  conda-forge
    r-viridislite-0.3.0        | r35h6115d3f_1002          63 KB  conda-forge
    r-wasabi-1.0.1             |           r351_0          31 KB  bioconda
    r-xfun-0.10                |    r35h6115d3f_0         182 KB  conda-forge
    r-xml-3.98_1.20            |    r35hcdcec82_1         2.0 MB  conda-forge
    r-xtable-1.8_4             |    r35h6115d3f_2         697 KB  conda-forge
    r-yaml-2.2.0               | r35hcdcec82_1002         113 KB  conda-forge
    rpy2-3.1.0                 |py36r35hc1659b7_2         305 KB  conda-forge
    ------------------------------------------------------------
                                           Total:       306.4 MB

The following NEW packages will be INSTALLED:

  bioconductor-rhdf5 bioconda/linux-64::bioconductor-rhdf5-2.28.0-r351hf484d3e_0
  bioconductor-rhdf~ bioconda/linux-64::bioconductor-rhdf5lib-1.6.0-r351h14c3975_0
  cloog              pkgs/main/linux-64::cloog-0.18.0-0
  gcc                cgat/linux-64::gcc-4.8.3-1
  gmp                conda-forge/linux-64::gmp-6.1.2-hf484d3e_1000
  isl                conda-forge/linux-64::isl-0.21-he80fd80_0
  mpc                conda-forge/linux-64::mpc-1.1.0-h04dde30_1006
  mpfr               conda-forge/linux-64::mpfr-4.0.2-he80fd80_0
  r                  pkgs/r/linux-64::r-3.1.2-2
  r-rjson            conda-forge/linux-64::r-rjson-0.2.20-r35h0357c0b_1001
  r-wasabi           bioconda/noarch::r-wasabi-1.0.1-r351_0

The following packages will be REMOVED:

  bioconductor-annotationfilter-1.8.0-r36_1
  bioconductor-ensembldb-2.8.0-r36_1
  bioconductor-protgenerics-1.16.0-r36_1
  bioconductor-rhtslib-1.16.1-r36hbcae180_1
  r-askpass-1.1-r36hcdcec82_1
  r-ellipsis-0.3.0-r36hcdcec82_0
  r-fastmap-1.0.1-r36h0357c0b_0
  r-mgcv-1.8_29-r36hcdcec82_0
  r-nlme-3.1_141-r36h9bbef5b_1
  r-sys-3.3-r36hcdcec82_0
  r-vctrs-0.2.0-r36hcdcec82_1
  r-withr-2.1.2-r36h6115d3f_1001
  r-zeallot-0.1.0-r36h6115d3f_1001

The following packages will be UPDATED:

  r-rcpp                                1.0.2-r36h0357c0b_0 --> 1.0.3-r35h0357c0b_0
  r-tinytex                              0.16-r36h6115d3f_0 --> 0.17-r35h6115d3f_0
  rpy2                              3.1.0-py36r36hc1659b7_1 --> 3.1.0-py36r35hc1659b7_2

The following packages will be SUPERSEDED by a higher-priority channel:

  bioconductor-cumm~ bioconda/noarch::bioconductor-cummerb~ --> bioconda/linux-64::bioconductor-cummerbund-2.12.1-r351_1
  bioconductor-gviz  bioconda/noarch::bioconductor-gviz-1.~ --> bioconda/linux-64::bioconductor-gviz-1.14.2-0
  ca-certificates    anaconda::ca-certificates-2019.10.16-0 --> conda-forge::ca-certificates-2019.9.11-hecc5488_0
  certifi                                          anaconda --> conda-forge
  openssl                anaconda::openssl-1.1.1-h7b6447c_0 --> conda-forge::openssl-1.1.1d-h516909a_0
  r-ggplot2          conda-forge/noarch::r-ggplot2-3.2.1-r~ --> conda-forge/linux-64::r-ggplot2-2.2.1-r351h6115d3f_1
  r-pillar           conda-forge/noarch::r-pillar-1.4.2-r3~ --> pkgs/r/linux-64::r-pillar-1.3.0-r351h6115d3f_0

The following packages will be DOWNGRADED:

  bioconductor-anno~                           1.62.0-r36_1 --> 1.60.1-r351_0
  bioconductor-anno~                           1.46.0-r36_1 --> 1.44.0-r351_0
  bioconductor-biob~                   2.44.0-r36h516909a_1 --> 2.42.0-r351h14c3975_1
  bioconductor-bioc~                           0.30.0-r36_1 --> 0.28.0-r351_1
  bioconductor-bioc~                   1.18.0-r36he1b5a44_1 --> 1.16.6-r351h1c2f66e_0
  bioconductor-biom~                           2.40.3-r36_0 --> 2.38.0-r351_0
  bioconductor-bios~                   2.52.0-r36h516909a_1 --> 2.50.2-r351h14c3975_0
  bioconductor-biov~                   1.32.0-r36h516909a_1 --> 1.18.0-1
  bioconductor-bsge~                           1.52.0-r36_1 --> 1.50.0-r351_0
  bioconductor-dela~                   0.10.0-r36h516909a_1 --> 0.8.0-r351h14c3975_0
  bioconductor-deseq                   1.36.0-r36h516909a_1 --> 1.34.1-r351h14c3975_0
  bioconductor-dese~                   1.24.0-r36he1b5a44_1 --> 1.22.1-r351hf484d3e_0
  bioconductor-edger                   3.26.5-r36he1b5a44_0 --> 3.26.0-r351hf484d3e_0
  bioconductor-gene~                   1.66.0-r36hc99cbb1_1 --> 1.64.0-r351h1c2f66e_1
  bioconductor-gene~                           1.62.0-r36_1 --> 1.60.0-r351_0
  bioconductor-geno~                           1.20.0-r36_1 --> 1.18.1-r351_0
  bioconductor-geno~                            1.2.1-r36_1 --> 1.2.1-r351_0
  bioconductor-geno~                   1.20.1-r36h516909a_0 --> 1.18.1-r351h14c3975_0
  bioconductor-geno~                           1.36.4-r36_0 --> 1.34.1-r351_0
  bioconductor-geno~                   1.36.0-r36h516909a_1 --> 1.34.0-r351h14c3975_0
  bioconductor-hpar                            1.26.0-r36_1 --> 1.26.0-r351_0
  bioconductor-iran~                   2.18.2-r36h516909a_0 --> 2.16.0-r351h14c3975_0
  bioconductor-limma                   3.40.2-r36h516909a_0 --> 3.40.0-r351h14c3975_0
  bioconductor-rsam~                    2.0.0-r36he1b5a44_1 --> 1.34.0-r351hf484d3e_0
  bioconductor-rtra~                   1.44.2-r36h516909a_1 --> 1.42.1-r351h9d9f1b6_1
  bioconductor-s4ve~                   0.22.0-r36h516909a_1 --> 0.20.1-r351h14c3975_0
  bioconductor-summ~                           1.14.0-r36_1 --> 1.12.0-r351_0
  bioconductor-vari~                   1.30.1-r36h516909a_0 --> 1.28.3-r351h14c3975_0
  bioconductor-xvec~                   0.24.0-r36h516909a_1 --> 0.22.0-r351h14c3975_0
  bioconductor-zlib~                   1.30.0-r36h516909a_1 --> 1.28.0-r351h14c3975_0
  r-acepack                          1.4.1-r36h9bbef5b_1004 --> 1.4.1-r35h9bbef5b_1004
  r-assertthat                          0.2.1-r36h6115d3f_1 --> 0.2.1-r35h6115d3f_1
  r-backports                           1.1.5-r36hcdcec82_0 --> 1.1.5-r35hcdcec82_0
  r-base                                   3.6.1-h8900bf8_2 --> 3.5.1-h08e1455_1008
  r-base64enc                        0.1_3-r36hcdcec82_1003 --> 0.1_3-r35hcdcec82_1003
  r-bh                               1.69.0_1-r36h6115d3f_1 --> 1.69.0_1-r35h6115d3f_1
  r-bit                                1.1_14-r36hcdcec82_1 --> 1.1_14-r35hcdcec82_1
  r-bit64                            0.9_7-r36hcdcec82_1001 --> 0.9_7-r35hcdcec82_1001
  r-bitops                           1.0_6-r36hcdcec82_1003 --> 1.0_6-r35hcdcec82_1003
  r-blob                                        1.2.0-r36_1 --> 1.1.1-r351_1001
  r-catools                          1.17.1.2-r36h0357c0b_1 --> 1.17.1.2-r35h0357c0b_1
  r-checkmate                           1.9.4-r36hcdcec82_1 --> 1.9.4-r35hcdcec82_1
  r-cli                                 1.1.0-r36h6115d3f_2 --> 1.1.0-r35h6115d3f_2
  r-cluster                             2.1.0-r36h9bbef5b_2 --> 2.1.0-r35h9bbef5b_2
  r-colorspace                          1.4_1-r36hcdcec82_1 --> 1.4_1-r35hcdcec82_1
  r-crayon                           1.3.4-r36h6115d3f_1002 --> 1.3.4-r35h6115d3f_1002
  r-crosstalk                        1.0.0-r36h6115d3f_1002 --> 1.0.0-r35h6115d3f_1002
  r-curl                                  4.2-r36hcdcec82_0 --> 4.2-r35hcdcec82_0
  r-data.table                         1.12.6-r36hcdcec82_0 --> 1.12.6-r35hcdcec82_0
  r-dbi                              1.0.0-r36h6115d3f_1002 --> 1.0.0-r35h6115d3f_1002
  r-dichromat                                2.0_0-r36_2001 --> 2.0_0-r35_2001
  r-digest                             0.6.22-r36h0357c0b_0 --> 0.6.22-r35h0357c0b_0
  r-dt                                    0.9-r36h6115d3f_0 --> 0.9-r35h6115d3f_0
  r-evaluate                             0.14-r36h6115d3f_1 --> 0.14-r35h6115d3f_1
  r-fansi                            0.4.0-r36hcdcec82_1001 --> 0.4.0-r35hcdcec82_1001
  r-fastcluster                     1.1.25-r36hc99cbb1_1002 --> 1.1.25-r35hc99cbb1_1002
  r-foreign                            0.8_72-r36hcdcec82_0 --> 0.8_72-r35hcdcec82_0
  r-formatr                               1.7-r36h6115d3f_1 --> 1.7-r35h6115d3f_1
  r-formula                          1.2_3-r36h6115d3f_1002 --> 1.2_3-r35h6115d3f_1002
  r-futile.logger                    1.4.3-r36h6115d3f_1002 --> 1.4.3-r35h6115d3f_1002
  r-futile.options                   1.0.1-r36h6115d3f_1001 --> 1.0.1-r35h6115d3f_1001
  r-gdata                           2.18.0-r36h6115d3f_1002 --> 2.18.0-r35h6115d3f_1002
  r-glue                                1.3.1-r36hcdcec82_1 --> 1.3.1-r35hcdcec82_1
  r-gmd                              0.3.3-r36h516909a_1002 --> 0.3.3-r35h516909a_1002
  r-gplots                            3.0.1.1-r36h6115d3f_1 --> 3.0.1.1-r35h6115d3f_1
  r-gridextra                          2.3-r36h6115d3f_1002 --> 2.3-r35h6115d3f_1002
  r-gsalib                                     2.1-r36_1001 --> 2.1-r35_1001
  r-gtable                              0.3.0-r36h6115d3f_2 --> 0.3.0-r35h6115d3f_2
  r-gtools                           3.8.1-r36hcdcec82_1003 --> 3.8.1-r35hcdcec82_1003
  r-highr                                 0.8-r36h6115d3f_1 --> 0.8-r35h6115d3f_1
  r-hmisc                               4.2_0-r36h9bbef5b_2 --> 4.2_0-r35h9bbef5b_2
  r-hms                                 0.5.2-r36h6115d3f_0 --> 0.4.2-r351h6115d3f_1000
  r-htmltable                          1.13.2-r36h6115d3f_0 --> 1.13.2-r35h6115d3f_0
  r-htmltools                           0.4.0-r36h0357c0b_0 --> 0.4.0-r35h0357c0b_0
  r-htmlwidgets                         1.5.1-r36h6115d3f_0 --> 1.5.1-r35h6115d3f_0
  r-httpuv                              1.5.2-r36h0357c0b_1 --> 1.5.2-r35h0357c0b_1
  r-httr                                1.4.1-r36h6115d3f_1 --> 1.4.1-r35h6115d3f_1
  r-jsonlite                           1.6-r36hcdcec82_1001 --> 1.6-r35hcdcec82_1001
  r-kernsmooth                     2.23_15-r36h9bbef5b_1004 --> 2.23_15-r35h9bbef5b_1004
  r-knitr                                1.25-r36h6115d3f_0 --> 1.25-r35h6115d3f_0
  r-labeling                           0.3-r36h6115d3f_1002 --> 0.3-r35h6115d3f_1002
  r-lambda.r                            1.2.4-r36h6115d3f_0 --> 1.2.4-r35h6115d3f_0
  r-later                               1.0.0-r36h0357c0b_0 --> 1.0.0-r35h0357c0b_0
  r-lattice                        0.20_38-r36hcdcec82_1002 --> 0.20_38-r35hcdcec82_1002
  r-latticeextra                    0.6_28-r36h6115d3f_1002 --> 0.6_28-r35h6115d3f_1002
  r-lazyeval                            0.2.2-r36hcdcec82_1 --> 0.2.2-r35hcdcec82_1
  r-locfit                         1.5_9.1-r36h516909a_1004 --> 1.5_9.1-r35h516909a_1004
  r-magrittr                           1.5-r36h6115d3f_1002 --> 1.5-r35h6115d3f_1002
  r-markdown                              1.1-r36hcdcec82_0 --> 1.1-r35hcdcec82_0
  r-mass                             7.3_51.4-r36hcdcec82_1 --> 7.3_51.4-r35hcdcec82_1
  r-matrix                             1.2_17-r36hcdcec82_1 --> 1.2_17-r35hcdcec82_1
  r-matrixstats                        0.55.0-r36hcdcec82_0 --> 0.55.0-r35hcdcec82_0
  r-memoise                          1.1.0-r36h6115d3f_1003 --> 1.1.0-r35h6115d3f_1003
  r-mime                                  0.7-r36hcdcec82_1 --> 0.7-r35hcdcec82_1
  r-munsell                          0.5.0-r36h6115d3f_1002 --> 0.5.0-r35h6115d3f_1002
  r-nnet                            7.3_12-r36hcdcec82_1003 --> 7.3_12-r35hcdcec82_1003
  r-openssl                             1.4.1-r36h9c8475f_0 --> 1.1-r351h3a9d887_1002
  r-pkgconfig                           2.0.3-r36h6115d3f_0 --> 2.0.3-r35h6115d3f_0
  r-plogr                            0.2.0-r36h6115d3f_1002 --> 0.2.0-r35h6115d3f_1002
  r-plotrix                             3.7_6-r36h6115d3f_1 --> 3.7_6-r35h6115d3f_1
  r-plyr                             1.8.4-r36h0357c0b_1003 --> 1.8.4-r35h0357c0b_1003
  r-prettyunits                      1.0.2-r36h6115d3f_1002 --> 1.0.2-r35h6115d3f_1002
  r-progress                            1.2.2-r36h6115d3f_1 --> 1.2.2-r35h6115d3f_1
  r-promises                            1.1.0-r36h0357c0b_0 --> 1.1.0-r35h0357c0b_0
  r-r6                                  2.4.0-r36h6115d3f_2 --> 2.4.0-r35h6115d3f_2
  r-rcolorbrewer                     1.1_2-r36h6115d3f_1002 --> 1.1_2-r35h6115d3f_1002
  r-rcpparmadillo                 0.9.800.1.0-r36h0357c0b_0 --> 0.9.800.1.0-r35h0357c0b_0
  r-rcurl                           1.95_4.12-r36hcdcec82_1 --> 1.95_4.12-r35hcdcec82_1
  r-reshape                             0.8.8-r36hcdcec82_1 --> 0.8.8-r35hcdcec82_1
  r-reshape2                         1.4.3-r36h0357c0b_1004 --> 1.4.3-r35h0357c0b_1004
  r-rlang                               0.4.1-r36hcdcec82_0 --> 0.4.1-r35hcdcec82_0
  r-rmarkdown                            1.16-r36h6115d3f_0 --> 1.16-r35h6115d3f_0
  r-rpart                              4.1_15-r36hcdcec82_1 --> 4.1_15-r35hcdcec82_1
  r-rsqlite                             2.1.2-r36h0357c0b_1 --> 2.1.2-r35h0357c0b_1
  r-rstudioapi                           0.10-r36h6115d3f_2 --> 0.10-r35h6115d3f_2
  r-scales                           1.0.0-r36h0357c0b_1002 --> 1.0.0-r35h0357c0b_1002
  r-shiny                               1.4.0-r36h6115d3f_0 --> 1.3.2-r35h6115d3f_1
  r-snow                             0.4_3-r36h6115d3f_1001 --> 0.4_3-r35h6115d3f_1001
  r-sourcetools                      0.1.7-r36he1b5a44_1001 --> 0.1.7-r35he1b5a44_1001
  r-stringi                             1.4.3-r36h0357c0b_2 --> 1.4.3-r35h0357c0b_2
  r-stringr                             1.4.0-r36h6115d3f_1 --> 1.4.0-r35h6115d3f_1
  r-survival                         2.44_1.1-r36hcdcec82_1 --> 2.44_1.1-r35hcdcec82_1
  r-tibble                              2.1.3-r36hcdcec82_1 --> 1.4.2-r351h96ca727_1002
  r-utf8                             1.1.4-r36hcdcec82_1001 --> 1.1.4-r35hcdcec82_1001
  r-venndiagram                     1.6.20-r36h6115d3f_1001 --> 1.6.20-r35h6115d3f_1001
  r-viridis                          0.5.1-r36h6115d3f_1003 --> 0.5.1-r35h6115d3f_1003
  r-viridislite                      0.3.0-r36h6115d3f_1002 --> 0.3.0-r35h6115d3f_1002
  r-xfun                                 0.10-r36h6115d3f_0 --> 0.10-r35h6115d3f_0
  r-xml                             3.98_1.20-r36hcdcec82_1 --> 3.98_1.20-r35hcdcec82_1
  r-xtable                              1.8_4-r36h6115d3f_2 --> 1.8_4-r35h6115d3f_2
  r-yaml                             2.2.0-r36hcdcec82_1002 --> 2.2.0-r35hcdcec82_1002
jscaber commented 4 years ago

In the meantime, I suggest you have update your ~/.condarc with:

  - conda-forge
  - bioconda
  - defaults

Also, if you have a recent re-install, could you please share conda list rpy2

Updated - by update I guess you mean to remove the existing depositories, which I have done.

sebastian-luna-valero commented 4 years ago

Sure, that's what I meant.

Ok, let me do some testing and I will come back to you.

Re: rpy2, I can see that you have version 3.1.0, which was installed with the installer but according to Jenkins that doesn't pass the tests. I can share details with you via PM.

jscaber commented 4 years ago

Aware of at least some of the rpy2 issues, recently pushed a compatibility fix to make both versions of rpy2 work. They appare to have moved the RRuntime library to another place.

jscaber commented 4 years ago

Hmm just a slightly more philosophical suggestion from me to tackle some of these recurrent problems - feel free to ignore.

That said, the R tximport package would be a must to be supported, as currently the salmon -> tximport -> deseq2 is stil a widely used and preferred rnaseqdiffexpression workflow?

sebastian-luna-valero commented 4 years ago

I agree with you, many thanks for sharing your thoughts!

Very keen on improving things as much as possible.

Re: r-wasaby. We do have separate conda environments for special tools that won't be updated but are required for some pipelines to work. We could do the same for r-wasabi if you think that's the best approach.

Acribbs commented 4 years ago
jscaber commented 4 years ago

Hi Sebastian, ran another fresh install in the background. Failed at the salmon stage.

 The code successfully installed!

 To activate the CGAT environment type: 
 $ source /ifs/projects/jakubs/cgat-developers-v0/conda-install/etc/profile.d/conda.sh
 $ conda activate base
 $ conda activate cgat-flow

 To deactivate the environment, use:
 $ conda deactivate

# install-devel.sh log | cgath1.anat.ox.ac.uk | Tue Nov 12 17:15:15 GMT 2019 | install pipeline deps 
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  1608  100  1608    0     0   8288      0 --:--:-- --:--:-- --:--:--  8246
Collecting package metadata (repodata.json): done
Solving environment: | 
Found conflicts! Looking for incompatible packages.
This can take several minutes.  Press CTRL-C to abort.
failed                                                                                                                                                      

UnsatisfiableError: The following specifications were found to be incompatible with each other:                                                             

Package llvm-meta conflicts for:
salmon -> libcxx -> llvm-meta[version='6.0.1|7.0.0']
sebastian-luna-valero commented 4 years ago

Exactly, that's the current issue.

When we leave rpy2 unpinned, version 3.1.0 is installed and tests fail with:

ImportError: cannot import name 'process_revents'

I haven't looked into why is that.

If we pin rpy2 to <= 2.9, then the installation fails with the error you just pasted. I think it's trying to install the latest salmon (e.g. version 1.0.0) and there is a conflict.

Either we solve the rpy2 version 3.1.0 issue, or we work with a previous version of salmon (e.g. <= 0.14)

Any preferences?

jscaber commented 4 years ago

Moving rpy2 to version 3 would be a significant rewrite of the codebase. Verison 3 included a complete rewrite of the interface. https://rpy2.github.io/doc/v3.0.x/html/changes.html

Example error: Pandas - we have lots of these https://stackoverflow.com/questions/55990529/module-rpy2-robjects-pandas2ri-has-no-attribute-ri2py https://rpy2.github.io/doc/v3.0.x/html/generated_rst/pandas.html

Acribbs commented 4 years ago

The only problem with installing earlier version of salmon would be that I have experienced segmentation faults when running on the cluster. However, I couldn't work out the exact reason for these seg faults.

jscaber commented 4 years ago

However, there is no reason to freeze the entire conda environment in the stone age because of one package. That would have a negative impact on research. The two options are:

jscaber commented 4 years ago

With respect to this branch I have fixed what I could. The pipeline now runs almost to the end again. Apart from the refactoring (completely changed geneset creation and removed sailfish) I have also detected a significant number of minor errors which resulted in errors in some of the minor tasks.

The remaining issues are:

Further testing is limited by my version of rpy2 (3.1.0).

sebastian-luna-valero commented 4 years ago

Hi,

I have been playing a little bit more with conda and https://github.com/cgat-developers/cgat-flow/commit/82791ccc780083821b0812f0dde6df6822d55207 seems to make the trick.

It now installs salmon version 1.0.0 and rpy2 2.9.4, which should be what we need.

@jscaber if you re-install, you should get the correct environment now. Instead of re-installing, you could also try conda install 'rpy2 < 3' and check whether that works.

Best regards, Sebastian

jscaber commented 4 years ago

Thanks Sebastian.

This is an example of what changes for rpy2 v3 compatibility might look like, not too terrible.

 from rpy2.robjects import pandas2ri
+import rpy2.robjects as ro
+from rpy2.robjects.conversion import localconverter

 import cgatcore.experiment as E
 import cgat.GTF as GTF
@@ -1420,15 +1425,17 @@ def plotTopGenesHeatmap(outfile):

-        plotHeatmap(pandas2ri.py2ri(intersection_pivot),
-                    pandas2ri.py2ri(factors_df))
+        with localconverter(ro.default_converter + pandas2ri.converter):
+            r_intersection_pivot = ro.conversion.py2rpy(intersection_pivot)
+            r_factors_df = ro.conversion.py2rpy(factors_df)
+        
+        plotHeatmap(r_intersection_pivot,
+                    r_factors_df)

     iotools.touch_file(outfile)
sebastian-luna-valero commented 4 years ago

Thanks, Jakub.

Indeed, it doesn't look terrible. However, I don't know how to quantify the effort for the remaining refactoring to migrate the code to rpy2 v3, and more importantly, I won't be able to help you with that particular task. So it is up to you guys (the end user of the production pipelines) to decide what's the best decision going forward.

Best regards, Sebastian

sebastian-luna-valero commented 4 years ago

Following up on the r-wasabi install, it depends on R version 3.5, and our main cgat-flow environment installs R version 3.6. Hence the large amount of changes required to install it.

If we want to keep r-wasabi as a dependency, I suggest we use it in a separate conda environment like other outdated dependencies.

Please let me know your thoughts.

jscaber commented 4 years ago

Hi,

Thank you @sebastian-luna-valero for all your hard work on this. I will speak to Adam about wasabi - but removing or creating a separate conda environment is the way to go.

Regarding rpy2, it looks like 2.9.4 is a little intermediate. While the pandas2ri.py2ri did not fail, the implicit conversions it induces were not tolerated. I think I could have overcome this with .activate(), but chose instead to use the explicit local converters: they have the same syntax as rpy2 v3 with one exception: ro.conversion.py2rpy in 3.1 is ro.conversion.py2ri 2.9.4! The mainers of rpy2 clearly love refactoring code.

I have fixed all the plot functions here and the only thing that does not work is runSalmonSaturation: this fails because the truncated bam of different sizes is recognised by salmon as corrupt. This may be something with my .bam though and I am not inclined to investgate.

@Acribbs could you test this branch? (I promise I will set up my own testing environment soon, too)

jscaber commented 4 years ago

Also, for the plotExpressionDensity I had to change the log2(TPM + 0.1) to log2(TPM + 1) to avoid negative values. Is there a reason it was 0.1 - I may be missing something?

jscaber commented 4 years ago

Hi,

I have been playing a little bit more with conda and 82791cc seems to make the trick.

It now installs salmon version 1.0.0 and rpy2 2.9.4, which should be what we need.

@jscaber if you re-install, you should get the correct environment now. Instead of re-installing, you could also try conda install 'rpy2 < 3' and check whether that works.

Best regards, Sebastian

Btw conda install 'rpy2 < 3' worked. No conflicts.

Acribbs commented 4 years ago

Hi @jscaber im going to be travelling for a few weeks but should be able to test this in-between trips. I will let you know outcome

sebastian-luna-valero commented 4 years ago

Hi,

Could we please recap what's needed to close this issue?

Best regards, Sebastian

Acribbs commented 4 years ago

I forgot about this, I will test this branch sing pipeline testing framework and get back to you.

Acribbs commented 4 years ago

Have tested this and all seems to be fine.

sebastian-luna-valero commented 4 years ago

ping @jscaber