search

Bioconductor / Contributions

Contribute Packages to Bioconductor
134 stars 33 forks source link

(inactive) cellassign #1178

Closed kieranrcampbell closed 4 years ago

kieranrcampbell commented 5 years ago

Update the following URL to point to the GitHub repository of the package you wish to submit to Bioconductor

Confirm the following by editing each check box to '[x]'

I am familiar with the essential aspects of Bioconductor software management, including:

For help with submitting your package, please subscribe and post questions to the bioc-devel mailing list.

LiNk-NY commented 4 years ago

Hi Kieran, @kieranrcampbell Please add tensorflow to the listed Imports. We will be looking into updating the tensorflow version on the SPB / BBS. Best, Marcel

LiNk-NY commented 4 years ago

@kieranrcampbell Also, I'm getting a warning when checking the package: 

* checking data for non-ASCII characters ... WARNING
  Error loading dataset 'example_sce':
   Error in .requirePackage(package) : 
    unable to find required package 'splatter'

  The dataset(s) may use package(s) not declared in the DESCRIPTION file.
LiNk-NY commented 4 years ago

Hi Kieran, @kieranrcampbell It's great that you're using basilisk but basilisk hasn't been submitted to either CRAN or Bioconductor. I'm tagging @LTLA for an update on the status of basilisk. We would have to wait until it's submitted to be able to accept the package. Thanks, Marcel

LTLA commented 4 years ago

I am, in turn, waiting for someone from the core team to tell me that it is okay to use the SPB as a test system (Bioconductor/ZarrExperiment#1). And in addition to that, someone to help me fix the various difficulties raised in LTLA/basilisk#1.

LTLA commented 4 years ago

See #1318.

hpages commented 4 years ago

@kieranrcampbell Probably too early to depend on basilisk.

I guess it's time to give tensorflow 2 a try. Only on the build machines used for BioC devel of course. I just upgraded tensorflow from 1.14.0 to 2.0.0 on malbec2 (Linux build machine for BioC 3.11):

biocbuild@malbec2:~$ python3
Python 3.6.8 (default, Oct  7 2019, 12:59:55) 
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import tensorflow
>>> tensorflow.__version__
'2.0.0'

THIS IS EXPERIMENTAL! We'll carefully monitor the impact of this update on the other Bioconductor packages that rely on tensorflow and that this upgrade might break. Hopefully they will be easy to fix. If not, we might need to go back to tensorflow 1.14.0. If everything goes well, we'll upgrade on the Windows builder (tokay2). I need to check the status of python3/tensorflow on our Mac builder (celaya2) but given that we're still stuck with El Capitan there I'm not too optimistic.

LiNk-NY commented 4 years ago

Thanks Hervé for supporting TF2 on the builders. Any updates on using TF2 for cellassign? I think you supported it in the past. @kieranrcampbell

kieranrcampbell commented 4 years ago

Fantastic thanks. The current cellassign is written for tensorflow 1.x but if 2.x is installed sets a backwards compatibility flag to use the 1.x syntax. It probably needs properly updated at some point too.

kieranrcampbell commented 4 years ago

Hi folks, any updates on basilisk or tensorflow 2 ? Would like to get this accepted into bioc prior to next release so please let me know what needs done

Thanks

Kieran

lshep commented 4 years ago

Looking at the thread it looks like @hpages has tensorflow 2 on Malbec2. I don't know the status of the order builders off hand but we can look into it. I don't see a build report of your package after @hpages said tensorflow 2 was available.

LiNk-NY commented 4 years ago

Last I saw, basilisk was building on the SPB. Please follow the package here: https://github.com/Bioconductor/Contributions/issues/1318 @kieranrcampbell Bump the version for another build Best, Marcel

bioc-issue-bot commented 4 years ago

Received a valid push; starting a build. Commits are:

de51aec Bump version number to rebuild

bioc-issue-bot commented 4 years ago

Dear Package contributor,

This is the automated single package builder at bioconductor.org.

Your package has been built on Linux, Mac, and Windows.

On one or more platforms, the build results were: "skipped, ERROR". This may mean there is a problem with the package that you need to fix. Or it may mean that there is a problem with the build system itself.

Please see the build report for more details.

kieranrcampbell commented 4 years ago

Hi @LiNk-NY just pushed a build but got the basilisk not found error -- I think I may have misunderstood, is basilisk still not on the build server?

LiNk-NY commented 4 years ago

Hi Kieran, @kieranrcampbell

basilisk has not been accepted yet and you may want to consider using TensorFlow instead since there may be little to no support for basilisk on Windows. You've mentioned that you had existing code for TF 1.x and it may be easier to adapt this code to be compatible with the current version of TF on the build system (2.x) and backwards compatible to TF 1.x. Thanks.

LiNk-NY commented 4 years ago

Any updates Kieran? @kieranrcampbell

kieranrcampbell commented 4 years ago

Sorry working on it! Will keep you updated

bioc-issue-bot commented 4 years ago

Received a valid push; starting a build. Commits are:

2fcd188 Bump version #

bioc-issue-bot commented 4 years ago

Dear Package contributor,

This is the automated single package builder at bioconductor.org.

Your package has been built on Linux, Mac, and Windows.

On one or more platforms, the build results were: "WARNINGS, ERROR". This may mean there is a problem with the package that you need to fix. Or it may mean that there is a problem with the build system itself.

Please see the build report for more details.

bioc-issue-bot commented 4 years ago

Received a valid push; starting a build. Commits are:

c3eab81 Guess who forgot to bump the version number

bioc-issue-bot commented 4 years ago

Dear Package contributor,

This is the automated single package builder at bioconductor.org.

Your package has been built on Linux, Mac, and Windows.

On one or more platforms, the build results were: "WARNINGS, ERROR". This may mean there is a problem with the package that you need to fix. Or it may mean that there is a problem with the build system itself.

Please see the build report for more details.

kieranrcampbell commented 4 years ago

Okay @LiNk-NY update time! We have warnings / errors but I think we're close.

Windows

We get a bunch of warnings saying basilisk isn't found:

Warning in normalizePath(path.expand(path), winslash, mustWork) :
  path[1]="C:\Users\pkgbuild\AppData\Local\basilisk\basilisk\basilisk-0.99.34\anaconda/python.exe": The system cannot find the file specified

which is bizarre since there's no basilisk anywhere in the package now (grep -ri basilisk * returns nothing). The final error is not being able to find python.exe -- any pointers?

Linux

The warnings are mainly numpy deprecation errors that I think we can ignore, e.g. 

//usr/lib/python2.7/dist-packages/h5py/__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.
  from ._conv import register_converters as _register_converters

There's one BiocCheck warning about requiring R 4.0 that I will address right before we submit so I can still test locally without getting the error on my 3.6 build.

Thanks,

Kieran

LiNk-NY commented 4 years ago

Hi Kieran, @kieranrcampbell

Make sure you remove the docs folder from the source package. Perhaps the issue is related to the cache? Any ideas @lshep ?

Thanks, Marcel

kieranrcampbell commented 4 years ago

Thanks @LiNk-NY -- we're keen to use pkgdown for this, is there a recommended directory other than docs to host this in?

Thanks

Kieran

LiNk-NY commented 4 years ago

@kieranrcampbell Yes, we recommend that you create a branch and include all pkgdown related source files in that branch. -Marcel

LTLA commented 4 years ago

basilisk lives! https://bioconductor.org/packages/devel/bioc/html/basilisk.html

Should be able to throw together a conda environment with tensorflow=2.0.0 pretty easily; if that works, it should save @hpages some trouble of fiddling with the build system.

LiNk-NY commented 4 years ago

Thanks Aaron! @kieranrcampbell Kieran, Do you plan on using basilik? Please provide us with some updates. Thanks.

kieranrcampbell commented 4 years ago

Hi @LiNk-NY

It would be good to get some guidance on this -- I submitted a version that used basilisk, but then removed it since it was not yet accepted, but now it is accepted it would be good to know if the recommendation is to use it or not?

Thanks

LiNk-NY commented 4 years ago

Hi Kieran, @kieranrcampbell

AFAICT, basilisk is now part of the devel branch of Bioconductor. http://bioconductor.org/checkResults/devel/bioc-LATEST/basilisk/

You can go ahead and use it in your dependencies.

Best, Marcel

LiNk-NY commented 4 years ago

Hi Kieran, @kieranrcampbell Do you want the package to be part of the Bioconductor release? Please see the deadlines here: https://www.bioconductor.org/developers/release-schedule/ Best, Marcel

bioc-issue-bot commented 4 years ago

Received a valid push; starting a build. Commits are:

c38304c Add a probability threshold to celltypes(fit) 8504593 Beta now initialized based on genes

bioc-issue-bot commented 4 years ago

Dear Package contributor,

This is the automated single package builder at bioconductor.org.

Your package has been built on Linux, Mac, and Windows.

On one or more platforms, the build results were: "WARNINGS, ERROR". This may mean there is a problem with the package that you need to fix. Or it may mean that there is a problem with the build system itself.

Please see the build report for more details.

kieranrcampbell commented 4 years ago

Hi Marcel,

I'll work on integrating Basilisk today and then hopefully we're there.

Thanks

Kieran

LTLA commented 4 years ago

Just a heads-up that the BioC-devel basilisk is broken. The fix is up but will take a build cycle to propagate through, all going well. In the meantime, the GitHub repos:

https://github.com/LTLA/basilisk https://github.com/LTLA/basilisk.utils

contain the latest versions of the packages.

kieranrcampbell commented 4 years ago

Hi @LTLA

I'm getting an error when using tensorflow (of course) via this method along the lines of

OMP: Error #15: Initializing libiomp5.dylib, but found libomp.dylib already initialized. OMP: Hint This means that multiple copies of the OpenMP runtime have been linked into the program.

which sure enough seems to be the case:

(base) kierancampbell@Kierans-MacBook-Pro:~/Library/Caches/basilisk/0.99.58/cellassign-0.99.21/cellassign_env/lib$ ll | grep omp
lrwxr-xr-x    1 kierancampbell  staff    12B  7 Apr 11:23 libgomp.dylib@ -> libomp.dylib
lrwxr-xr-x    1 kierancampbell  staff    12B  7 Apr 11:23 libgomp.1.dylib@ -> libomp.dylib
lrwxr-xr-x    1 kierancampbell  staff    12B  7 Apr 11:23 libiomp5.dylib@ -> libomp.dylib
-rwxrwxr-x    1 kierancampbell  staff   639K  7 Apr 11:23 libomp.dylib*

One workaround frequently suggested online is to install nomkl which can be done via

cellassign_env <- BasiliskEnvironment("cellassign_env", pkgname="cellassign",
                            packages=c("nomkl=3.0",
                                       "python=3.7.4",
                                       "tensorflow=2.0.0",
                                       "tensorflow-probability=0.7.0"))

but this still gives the error ( another thing to note is that only tensorflow 2.0.0 is available for MacOS on conda compared to the official release which I think is 2.1.0 currently)

Finally after more googling I can across this thread which states

Solution 3: that restored performance was: Install pip in new conda env and use pip to install tensorflow. Using conda-forge also worked but version of tf is old. Apparently the new Intel-MKL optimizations in Anaconda are broken for OSX tensorflow.

so I'm wondering -- is it possible in basilisk to selectively install some packages with pip inside the conda environment? I think that might be the only way round this?

Thanks

LTLA commented 4 years ago

is it possible in basilisk to selectively install some packages with pip inside the conda environment?

This was, in fact, what I used to do. The conda environment was only used to provide me a working Python installation, and everything else was done with pip via virtual environments. This became a real pain; for starters, virtualenv wasn't supported on windows, so then I had to use conda environments anyway to get things to run on Windows. At that point, it was just easier to switch to conda environments for everything, so I stopped fighting it and let conda do its job.

One thing that came to mind is that the conda packages are probably linked to the conda-provided system libraries, while the pip wheels are linked to the actual system libraries (I assume). I don't know whether this would be a problem - maybe if Python packages have C/C++-level links to each other, and are passing objects with ABI incompatibilities? - but it was enough of a potential headache to give up on the conda/pip intermixing as a standard approach. I also assume you've read Anaconda's commentary on this topic, which is hardly encouraging either.

I guess I could provide an avenue for pip-installable packages, but I'm not going to take particular responsibility for whether the resulting environment is valid.

Edit: see the pip branch on the basilisk GitHub.

LTLA commented 4 years ago

@kieranrcampbell give me a yay or nay on the pip branch.

kieranrcampbell commented 4 years ago

Hi @LTLA are you proposing that cellassign would use a separate basilisk branch or that you would merge the pip branch as master? In the latter case I will install and check it works

LTLA commented 4 years ago

Test the branch locally and if it works I will merge it and push it into the build system.

kieranrcampbell commented 4 years ago

Success! This works (at least locally on my mac). If you're able to push to build system I'll push also and hopefully get them to build on other systems

LTLA commented 4 years ago

It is done, basilisk 0.99.59.

bioc-issue-bot commented 4 years ago

Received a valid push; starting a build. Commits are:

6ff6d4d Begin basilisk migration 354dd5c Initial go at basilisk 91de5fe Updated basilisk integration using pip branch; nee...

bioc-issue-bot commented 4 years ago

Dear Package contributor,

This is the automated single package builder at bioconductor.org.

Your package has been built on Linux, Mac, and Windows.

On one or more platforms, the build results were: "skipped, ERROR". This may mean there is a problem with the package that you need to fix. Or it may mean that there is a problem with the build system itself.

Please see the build report for more details.

kieranrcampbell commented 4 years ago 
Error in BasiliskEnvironment("cellassign_env", pkgname = "cellassign",  : 
  unused argument (pip = c("tensorflow==2.1.0", "tensorflow-probability==0.7.0"))

It is done, basilisk 0.99.59.

I guess the changes take some time to push through to the build server?

LiNk-NY commented 4 years ago

Hi Kieran, @kieranrcampbell You'd have to wait until tomorrow when the package build is propagated.

bioc-issue-bot commented 4 years ago

Received a valid push; starting a build. Commits are:

2917210 Bump version number to rebuild on bioc servers

bioc-issue-bot commented 4 years ago

Dear Package contributor,

This is the automated single package builder at bioconductor.org.

Your package has been built on Linux, Mac, and Windows.

On one or more platforms, the build results were: "skipped, ERROR, WARNINGS". This may mean there is a problem with the package that you need to fix. Or it may mean that there is a problem with the build system itself.

Please see the build report for more details.

bioc-issue-bot commented 4 years ago

Received a valid push; starting a build. Commits are:

71dd2e0 Does adding openssl solve the pip install problem?

bioc-issue-bot commented 4 years ago

Dear Package contributor,

This is the automated single package builder at bioconductor.org.

Your package has been built on Linux, Mac, and Windows.

On one or more platforms, the build results were: "WARNINGS, skipped, ERROR". This may mean there is a problem with the package that you need to fix. Or it may mean that there is a problem with the build system itself.

Please see the build report for more details.

bioc-issue-bot commented 4 years ago

Received a valid push; starting a build. Commits are:

6f8cd3d Does downgrading python version work? d026bfe gah version number