Closed kieranrcampbell closed 4 years ago
Hi Kieran, @kieranrcampbell
Please add tensorflow
to the listed Imports
.
We will be looking into updating the tensorflow
version on the SPB / BBS.
Best,
Marcel
@kieranrcampbell Also, I'm getting a warning when checking the package:
* checking data for non-ASCII characters ... WARNING
Error loading dataset 'example_sce':
Error in .requirePackage(package) :
unable to find required package 'splatter'
The dataset(s) may use package(s) not declared in the DESCRIPTION file.
Hi Kieran, @kieranrcampbell
It's great that you're using basilisk
but basilisk
hasn't been submitted to either CRAN or Bioconductor. I'm tagging @LTLA for an update on the status of basilisk
. We would have to wait until it's submitted to be able to accept the package.
Thanks,
Marcel
I am, in turn, waiting for someone from the core team to tell me that it is okay to use the SPB as a test system (Bioconductor/ZarrExperiment#1). And in addition to that, someone to help me fix the various difficulties raised in LTLA/basilisk#1.
See #1318.
@kieranrcampbell Probably too early to depend on basilisk.
I guess it's time to give tensorflow 2 a try. Only on the build machines used for BioC devel of course. I just upgraded tensorflow from 1.14.0 to 2.0.0 on malbec2 (Linux build machine for BioC 3.11):
biocbuild@malbec2:~$ python3
Python 3.6.8 (default, Oct 7 2019, 12:59:55)
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import tensorflow
>>> tensorflow.__version__
'2.0.0'
THIS IS EXPERIMENTAL! We'll carefully monitor the impact of this update on the other Bioconductor packages that rely on tensorflow and that this upgrade might break. Hopefully they will be easy to fix. If not, we might need to go back to tensorflow 1.14.0. If everything goes well, we'll upgrade on the Windows builder (tokay2). I need to check the status of python3/tensorflow on our Mac builder (celaya2) but given that we're still stuck with El Capitan there I'm not too optimistic.
Thanks Hervé for supporting TF2 on the builders.
Any updates on using TF2 for cellassign
? I think you supported it in the past. @kieranrcampbell
Fantastic thanks. The current cellassign is written for tensorflow 1.x but if 2.x is installed sets a backwards compatibility flag to use the 1.x syntax. It probably needs properly updated at some point too.
Hi folks, any updates on basilisk or tensorflow 2 ? Would like to get this accepted into bioc prior to next release so please let me know what needs done
Thanks
Kieran
Looking at the thread it looks like @hpages has tensorflow 2 on Malbec2. I don't know the status of the order builders off hand but we can look into it. I don't see a build report of your package after @hpages said tensorflow 2 was available.
Last I saw, basilisk
was building on the SPB. Please follow the package here: https://github.com/Bioconductor/Contributions/issues/1318
@kieranrcampbell Bump the version for another build
Best,
Marcel
Received a valid push; starting a build. Commits are:
de51aec Bump version number to rebuild
Dear Package contributor,
This is the automated single package builder at bioconductor.org.
Your package has been built on Linux, Mac, and Windows.
On one or more platforms, the build results were: "skipped, ERROR". This may mean there is a problem with the package that you need to fix. Or it may mean that there is a problem with the build system itself.
Please see the build report for more details.
Hi @LiNk-NY just pushed a build but got the basilisk not found error -- I think I may have misunderstood, is basilisk still not on the build server?
Hi Kieran, @kieranrcampbell
basilisk
has not been accepted yet and you may want to consider using TensorFlow instead since there may be little to no support for basilisk
on Windows. You've mentioned that you had existing code for TF 1.x and it may be easier to adapt this code to be compatible with the current version
of TF on the build system (2.x) and backwards compatible to TF 1.x. Thanks.
Any updates Kieran? @kieranrcampbell
Sorry working on it! Will keep you updated
Received a valid push; starting a build. Commits are:
2fcd188 Bump version #
Dear Package contributor,
This is the automated single package builder at bioconductor.org.
Your package has been built on Linux, Mac, and Windows.
On one or more platforms, the build results were: "WARNINGS, ERROR". This may mean there is a problem with the package that you need to fix. Or it may mean that there is a problem with the build system itself.
Please see the build report for more details.
Received a valid push; starting a build. Commits are:
c3eab81 Guess who forgot to bump the version number
Dear Package contributor,
This is the automated single package builder at bioconductor.org.
Your package has been built on Linux, Mac, and Windows.
On one or more platforms, the build results were: "WARNINGS, ERROR". This may mean there is a problem with the package that you need to fix. Or it may mean that there is a problem with the build system itself.
Please see the build report for more details.
Okay @LiNk-NY update time! We have warnings / errors but I think we're close.
We get a bunch of warnings saying basilisk isn't found:
Warning in normalizePath(path.expand(path), winslash, mustWork) :
path[1]="C:\Users\pkgbuild\AppData\Local\basilisk\basilisk\basilisk-0.99.34\anaconda/python.exe": The system cannot find the file specified
which is bizarre since there's no basilisk anywhere in the package now (grep -ri basilisk *
returns nothing). The final error is not being able to find python.exe
-- any pointers?
The warnings are mainly numpy deprecation errors that I think we can ignore, e.g.
//usr/lib/python2.7/dist-packages/h5py/__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.
from ._conv import register_converters as _register_converters
There's one BiocCheck warning about requiring R 4.0 that I will address right before we submit so I can still test locally without getting the error on my 3.6 build.
Thanks,
Kieran
Hi Kieran, @kieranrcampbell
Make sure you remove the docs
folder from the source package.
Perhaps the issue is related to the cache? Any ideas @lshep ?
Thanks, Marcel
Thanks @LiNk-NY -- we're keen to use pkgdown for this, is there a recommended directory other than docs
to host this in?
Thanks
Kieran
@kieranrcampbell Yes, we recommend that you create a branch and include all pkgdown related source files in that branch. -Marcel
basilisk lives! https://bioconductor.org/packages/devel/bioc/html/basilisk.html
Should be able to throw together a conda environment with tensorflow=2.0.0
pretty easily; if that works, it should save @hpages some trouble of fiddling with the build system.
Thanks Aaron! @kieranrcampbell Kieran, Do you plan on using basilik? Please provide us with some updates. Thanks.
Hi @LiNk-NY
It would be good to get some guidance on this -- I submitted a version that used basilisk, but then removed it since it was not yet accepted, but now it is accepted it would be good to know if the recommendation is to use it or not?
Thanks
Hi Kieran, @kieranrcampbell
AFAICT, basilisk
is now part of the devel branch of Bioconductor.
http://bioconductor.org/checkResults/devel/bioc-LATEST/basilisk/
You can go ahead and use it in your dependencies.
Best, Marcel
Hi Kieran, @kieranrcampbell Do you want the package to be part of the Bioconductor release? Please see the deadlines here: https://www.bioconductor.org/developers/release-schedule/ Best, Marcel
Dear Package contributor,
This is the automated single package builder at bioconductor.org.
Your package has been built on Linux, Mac, and Windows.
On one or more platforms, the build results were: "WARNINGS, ERROR". This may mean there is a problem with the package that you need to fix. Or it may mean that there is a problem with the build system itself.
Please see the build report for more details.
Hi Marcel,
I'll work on integrating Basilisk today and then hopefully we're there.
Thanks
Kieran
Just a heads-up that the BioC-devel basilisk is broken. The fix is up but will take a build cycle to propagate through, all going well. In the meantime, the GitHub repos:
https://github.com/LTLA/basilisk https://github.com/LTLA/basilisk.utils
contain the latest versions of the packages.
Hi @LTLA
I'm getting an error when using tensorflow (of course) via this method along the lines of
OMP: Error #15: Initializing libiomp5.dylib, but found libomp.dylib already initialized. OMP: Hint This means that multiple copies of the OpenMP runtime have been linked into the program.
which sure enough seems to be the case:
(base) kierancampbell@Kierans-MacBook-Pro:~/Library/Caches/basilisk/0.99.58/cellassign-0.99.21/cellassign_env/lib$ ll | grep omp
lrwxr-xr-x 1 kierancampbell staff 12B 7 Apr 11:23 libgomp.dylib@ -> libomp.dylib
lrwxr-xr-x 1 kierancampbell staff 12B 7 Apr 11:23 libgomp.1.dylib@ -> libomp.dylib
lrwxr-xr-x 1 kierancampbell staff 12B 7 Apr 11:23 libiomp5.dylib@ -> libomp.dylib
-rwxrwxr-x 1 kierancampbell staff 639K 7 Apr 11:23 libomp.dylib*
One workaround frequently suggested online is to install nomkl
which can be done via
cellassign_env <- BasiliskEnvironment("cellassign_env", pkgname="cellassign",
packages=c("nomkl=3.0",
"python=3.7.4",
"tensorflow=2.0.0",
"tensorflow-probability=0.7.0"))
but this still gives the error ( another thing to note is that only tensorflow 2.0.0 is available for MacOS on conda compared to the official release which I think is 2.1.0 currently)
Finally after more googling I can across this thread which states
Solution 3: that restored performance was: Install pip in new conda env and use pip to install tensorflow. Using conda-forge also worked but version of tf is old. Apparently the new Intel-MKL optimizations in Anaconda are broken for OSX tensorflow.
so I'm wondering -- is it possible in basilisk to selectively install some packages with pip inside the conda environment? I think that might be the only way round this?
Thanks
is it possible in basilisk to selectively install some packages with pip inside the conda environment?
This was, in fact, what I used to do. The conda environment was only used to provide me a working Python installation, and everything else was done with pip via virtual environments. This became a real pain; for starters, virtualenv wasn't supported on windows, so then I had to use conda environments anyway to get things to run on Windows. At that point, it was just easier to switch to conda environments for everything, so I stopped fighting it and let conda
do its job.
One thing that came to mind is that the conda packages are probably linked to the conda-provided system libraries, while the pip wheels are linked to the actual system libraries (I assume). I don't know whether this would be a problem - maybe if Python packages have C/C++-level links to each other, and are passing objects with ABI incompatibilities? - but it was enough of a potential headache to give up on the conda/pip intermixing as a standard approach. I also assume you've read Anaconda's commentary on this topic, which is hardly encouraging either.
I guess I could provide an avenue for pip-installable packages, but I'm not going to take particular responsibility for whether the resulting environment is valid.
Edit: see the pip
branch on the basilisk GitHub.
@kieranrcampbell give me a yay or nay on the pip
branch.
Hi @LTLA are you proposing that cellassign would use a separate basilisk branch or that you would merge the pip branch as master? In the latter case I will install and check it works
Test the branch locally and if it works I will merge it and push it into the build system.
Success! This works (at least locally on my mac). If you're able to push to build system I'll push also and hopefully get them to build on other systems
It is done, basilisk 0.99.59.
Dear Package contributor,
This is the automated single package builder at bioconductor.org.
Your package has been built on Linux, Mac, and Windows.
On one or more platforms, the build results were: "skipped, ERROR". This may mean there is a problem with the package that you need to fix. Or it may mean that there is a problem with the build system itself.
Please see the build report for more details.
Error in BasiliskEnvironment("cellassign_env", pkgname = "cellassign", :
unused argument (pip = c("tensorflow==2.1.0", "tensorflow-probability==0.7.0"))
It is done, basilisk 0.99.59.
I guess the changes take some time to push through to the build server?
Hi Kieran, @kieranrcampbell You'd have to wait until tomorrow when the package build is propagated.
Received a valid push; starting a build. Commits are:
2917210 Bump version number to rebuild on bioc servers
Dear Package contributor,
This is the automated single package builder at bioconductor.org.
Your package has been built on Linux, Mac, and Windows.
On one or more platforms, the build results were: "skipped, ERROR, WARNINGS". This may mean there is a problem with the package that you need to fix. Or it may mean that there is a problem with the build system itself.
Please see the build report for more details.
Received a valid push; starting a build. Commits are:
71dd2e0 Does adding openssl solve the pip install problem?
Dear Package contributor,
This is the automated single package builder at bioconductor.org.
Your package has been built on Linux, Mac, and Windows.
On one or more platforms, the build results were: "WARNINGS, skipped, ERROR". This may mean there is a problem with the package that you need to fix. Or it may mean that there is a problem with the build system itself.
Please see the build report for more details.
Update the following URL to point to the GitHub repository of the package you wish to submit to Bioconductor
Confirm the following by editing each check box to '[x]'
[x] I understand that by submitting my package to Bioconductor, the package source and all review commentary are visible to the general public.
[x] I have read the Bioconductor Package Submission instructions. My package is consistent with the Bioconductor Package Guidelines.
[x] I understand that a minimum requirement for package acceptance is to pass R CMD check and R CMD BiocCheck with no ERROR or WARNINGS. Passing these checks does not result in automatic acceptance. The package will then undergo a formal review and recommendations for acceptance regarding other Bioconductor standards will be addressed.
[x] My package addresses statistical or bioinformatic issues related to the analysis and comprehension of high throughput genomic data.
[x] I am committed to the long-term maintenance of my package. This includes monitoring the support site for issues that users may have, subscribing to the bioc-devel mailing list to stay aware of developments in the Bioconductor community, responding promptly to requests for updates from the Core team in response to changes in R or underlying software.
I am familiar with the essential aspects of Bioconductor software management, including:
For help with submitting your package, please subscribe and post questions to the bioc-devel mailing list.