preparation for 2.0 release

[bgruening]

Short check list for all things we need to do before and after we release.

• [x] tag a github release [Fidel]
• [x] upload to pypi [Fidel]
• [x] conda build [Bjoern or Devon]
• [x] Galaxy TTS and MTS update [Bjoern]
• [x] Docker Image update (ngs-preprocessing and deeptools image) [Bjoern]
• [x] migrate deeptools2 to deeptools [Devon]
• [x] usegalaxy.org installation [Bjoern]
• [x] automated internal testing of all tools [Fidel and Devon]
• [X] Update wiki to reference read the docs [Devon]
• [x] Screen-cast for the galaxy webpage [Fidel]

[bgruening]

Please feel to disagree and/or add remove items.

[dpryan79]

+1 We need to have an internal discussion about the last features that need to be added (my only suggestion would be remote file access, which is why I'm working on it now). I would also vote for automated internal testing, but I'm flexible on that one.

[bgruening]

Oh and a release date would be nice to coordinate this :)

[dpryan79]

We're shooting for 2 weeks until release.

[vivekbhr]

Two features that I think should to be added in near future:

1. A set of contrasting colours + alternating line types in bamFP, which will solve #80 . (maybe before 2.0 release)
2. A general GTF file parser, which will solve #71 #76 (for the next release).

[bgruening]

@vivekbhr I would create separate issues for this and tag them with a potential milestone. I wouldn't include any new features at this point. We can create a 2.1 release real quick if we like to, but we need to get this baby out, imho.

[dpryan79]

Agreed, these features should be in 2.1. This thing has been in development for 2 years without a release, that's more than long enough.

For what it's worth, I was going to assign #80 to myself, since that should hopefully be an easy fix. #71 is easy if we just use "transcript" entries. I think #76 would take rather more effort than it might appear, though I agree it would be nice.

[lelouar]

what you think of this feature for release 2.0 ? #108

[dpryan79]

@lelouar: It won't be done in time for 2.0. That's a good feature for 2.1 though.

[fidelram]

I agree with the list and I will add the renaming of bamCorrelate, bigwigCorrelate, bamFingerprint. For the last one the idea is to clarify that they make a plot and should be labelled accordingly like plotFingerprint. Since the input to make the bamFingerprint is virtually identical to the one used to plot the correlations we can simplify the process by requiring as input for plotFingerprint not a list of bam files but the output of bigwigCorrelate or bamCorrelate.

With respect to #80, an approach similar to what is done in plotProfile can be easily implemented which is basically to ask the user for the list of colors to be used.

[friedue]

can you update me on the renaming plans:

• do you mean that the tool names will be changed for the release of 2.0?
• what will the new names be?
• will the new names be implemented in Galaxy, too, and should I henceforth use the new names already when referring to the tools in the Galaxy documentation?

[dpryan79]

Bioconda is ready to go for both Linux and OSX. Just waiting on the release getting tagged and put on pypi.

[vivekbhr]

@friedue we are not changing the names in 2.0 .. probably in 2.1..

[bgruening]

@vivekbhr we changed the names in https://github.com/fidelram/deepTools/pull/220 and this is now in master, aka 2.0, isn't it? Now I'm confused as well :)

[dpryan79]

@bgruening Correct. It wasn't done for the initial Galaxy preview, but will for the actual release.

[vivekbhr]

@bgruening @dpryan79 Oh! I totally missed that thread..

[dpryan79]

@vivekbhr There are a lot of them, so that's unsurprising :)

[dpryan79]

We essentially have all of the internal testing scripts gathered together and run, so I've checked that off.

@fidelram @bgruening: I don't see anything preventing tagging a release. I'm having some issues with the galaxy wrappers not working, but that doesn't really prevent tagging 2.0.0 IMO. If we're all in agreement with that then perhaps it's time to tag. Thoughts?

[fidelram]

Let's do it. We already fulfilled all issues for the 2.0 milestone

[bgruening]

:+1: for me as well. Than I can fix the wrappers and the Docker Images.

[vivekbhr]

:+1:

[bgruening]

It's out! @fidelram awesome!

Thanks all for this great release!

[dpryan79]

I'm testing the bioconda recipe now.

[dpryan79]

I have the PR into bioconda: https://github.com/bioconda/bioconda-recipes/pull/646

[bgruening]

docker pull quay.io/bgruening/galaxy-deeptools has anything installed. Dockerhub makes trouble and is queueing my builds, but this will be resolved soonish hopefully. They are still migrating there services.

So this could be the final Docker release. I will make some smoke testing and release everything to the Main TS tomorrow if no one objects.

ping @dpryan79 @friedue

[dpryan79]

@bgruening Thanks Galaxy is working happily now with the updated wrappers and the tagged 2.0.0 release :)

[bgruening]

Awesome!

[dpryan79]

I just announced the deeptools.ie-freiburg.mpg.de migration for Friday next week.

[dpryan79]

We should discuss whether to keep the older versions after the migration. The normal default there would be, "but of course!", but since some of the names have changed, things like bamCorrelate will still show up and then be version 1.5.9. My proposal would be to just remove those old tools whose name changed. Thoughts?

[fidelram]

humm. we need some brain storming to detect unforeseen consequences.

In galaxy, I understand that if a user wants to run a pipeline or re-execute a job, the version of the program is kept. Thus, even if the tools are deleted from the menu, users reproducing results should be directed to the previous version. Is that the case?

Now, what would happen if a user needs the previous version of deeptools to keep results consistent? Can this person still access an earlier version of deepTools even if not in the galaxy menu?

[dpryan79]

DeepTools is now in bioconda and should work on OSX and Linux with it.

[dpryan79]

@fidelram: Yeah, I would just remove them from the menu. Normally they can select the version of the tool they would want to use (it defaults to the most recent, or at least the most recent installed). However, things like bamCorrelate wouldn't be there, so they couldn't run them. Alternatively, we could leave them there, but then people would continue using them.

I'm not entirely sure how Galaxy determines that some tools are different versions of others, perhaps there some trickery that could be done there.

[bgruening]

We changed all linage in deeptools 2.0 as the names changed. What we can do is to hide old deeptools from the menue, so they only work in old workflows or kind of. All new stuff is taking the new deeptools version.

[bgruening]

@dpryan79 do you think we can push to Main TS?

[dpryan79]

@bgruening: Yeah, we could just hide things, that'd be the simplest solution. Regarding pushing to the toolshed, I wan't to briefly test to make sure each tool runs. After that's done then I'll give a :+1:.

[dpryan79]

Missed opportunity: @vivekbhr just pointed out that the name of multiBamCoverage is confusing. In hindsight it should have been multiBamSummary, which would further distinguish it from bamCoverage. Maybe in 2.1...

[dpryan79]

Everything is getting a green light except plotPCA. I wonder if that's broken in the actual release...

[bgruening]

It's not to late from my side. Just tag a new release, it's not yet in the TS and changing name later is even more confusing if the paper uses the old name.

[dpryan79]

Yeah, plotPCA is broken in 2.0.0. I'll fix it real quick. If we want to change the name we can then tag 2.0.1 after doing so.

[dpryan79]

I'll try to have an answer by tonight regarding the renaming. plotPCA now works (at least for me, though it'd be nice if someone double checked...). Then we can proceed with tagging a release and pushing to the TS

[fidelram]

we can do another renaming. I agree that now is better than later. @friedue do you foresee any issues with the documentation?

[vivekbhr]

Yah I also agree we should do it sooner..

[dpryan79]

Shall we agree to just make multiBamCoverage multiBamSummary, to match multiBigwigSummary?

[dpryan79]

Truth be told, multiBigwigSummary can now take a single file, but I would ignore that since it's such an obscure use-case.

[bgruening]

+1 for renaming again.

[dpryan79]

@fidelram: so does "multiBamSummary" work for you? @thomasmanke: Thoughts?

I'm personally happy with the other names, though I guess if someone else wants another one renamed then it's sort of now or never...

[friedue]

well, if multiBigwigSummary can now take single files, that name is sort of a misnomer now.... :-P

[dpryan79]

@friedue Yeah, the biggest benefit to the "multi" prefix is that it emphasizes that more than one file should normally be entered. multiBamCoverage could also take a single file (I assume, if not it's another trivial fix). The only thing is that the resulting npz files aren't useful with any of our tools (the only useful thing is to write bedGraph files with the text output).

[dpryan79]

In fact, maybe they should print a warning if only a single file is provided and it isn't requested that the text file be written...

[friedue]

yes, I like that! print a warning that yes, the tools will run, but the output will be useless unless the --rawBlaOption is chosen