Closed kieranrcampbell closed 4 years ago
Dear Package contributor,
This is the automated single package builder at bioconductor.org.
Your package has been built on Linux, Mac, and Windows.
On one or more platforms, the build results were: "skipped, ERROR, WARNINGS". This may mean there is a problem with the package that you need to fix. Or it may mean that there is a problem with the build system itself.
Please see the build report for more details.
Oh yeah, this SSL issue was another reason why I gave up on pip, I could never get that to work on Windows. Perhaps the following flags in the pip
invocation would help:
--trusted-host=pypi.python.org --trusted-host=pypi.org --trusted-host=files.pythonhosted.org
But I don't know enough to say that this is a good thing to do in general. Sounds kinda risky.
Thanks for keeping an eye out on this. CC'ing @kieranrcampbell
Received a valid push; starting a build. Commits are:
2d10235 CHange install based on OS
Dear Package contributor,
This is the automated single package builder at bioconductor.org.
Your package has been built on Linux, Mac, and Windows.
On one or more platforms, the build results were: "skipped, ERROR, WARNINGS". This may mean there is a problem with the package that you need to fix. Or it may mean that there is a problem with the build system itself.
Please see the build report for more details.
Well, at least the CondaVerificationError
s are the same as what Biocsklearn is eating:
http://bioconductor.org/checkResults/devel/bioc-LATEST/BiocSklearn/tokay2-buildsrc.html
CondaVerificationError: The package for scikit-learn located at C:\Users\BIOCBU~1\AppData\Local\me\basilisk\099~1.59\Cache\anaconda\pkgs\scikit-learn-0.22.2.post1-py37h7208079_0
appears to be corrupted. The path 'Lib/site-packages/sklearn/datasets/tests/data/openml/292/api-v1-json-data-list-data_name-australian-limit-2-data_version-1-status-deactivated.json.gz'
specified in the package manifest cannot be found.
This has been a persistent error for some time and is pretty cryptic to me; I'm just running reticulate::conda_install
, for crying out loud! I would ordinarily think that some corrupted version has been cached from some previous iteration and is being re-used, but that seems unlikely as basilisk destroys its conda
-related cache whenever its own version gets bumped. (Similarly, each client package will destroy its previous cached environment when its version is bumped.)
The next possibility is that the Windows builders (SPB and tokay2) are running out of disk space in the cache directory. Seems unlikely, but who knows. Maybe @lshep could take a look?
I would also mention that, thanks to some experiments from @vjcitn, we know that Biocsklearn has no inherent problems with running on Windows (well, none relative to the usual Windows-related shenanigans). So the CondaVerificationError
may be specific to the BioC build system.
Edit: Possibly many of these issues may be related to the fact that reticulate does not seem to activate the conda environment before doing stuff with it. This causes Windows to wander off and find any old thing in the PATH
, linking to the wrong SSL libraries and causing a variety of errors. I would further speculate that the error also depends on the state of the Windows system libraries, which is why you don't see it in clean VM instances but instead in long-running servers.
Edit 2: 0.99.60 now set CONDA_DLL_SEARCH_MODIFICATION_ENABLE=1
in an attempt to correct the path search, given that we can't really modify the bash
session via activation.
Edit 3: Unbelievably, that environment variable seemed to have worked; 0.99.60 builds on all systems and pip install
passes its tests, which implies that the SSL problems are fixed.
Wait. Wait. I realized what the problem was. Consider the full path of the first missing file:
C:\Users\pkgbuild\AppData\Local\me\basilisk\099~1.59\Cache\anaconda\pkgs\tensorflow-base-2.1.0-mkl_py37h230818c_0/Lib/site-packages/tensorflow-2.1.0.data/purelib/tensorflow_core/include/tensorflow_core/core/common_runtime/isolate_placer_inspection_required_ops_pass.h
Verily, it is beyond 260 characters, which is the file path limit on Windows! Ha!
Now, that is much, much harder to solve. The only part of that path that basilisk controls is:
me\basilisk\099~1.59\Cache\anaconda
Not much to cut off there. I could slim it to:
basilisk\099~1.59\0
... which gives us enough breathing space for the longest path (transposer_factory.h
).
However, I suspect that this just kicks the can down the road; one can be pretty sure that the environments will bump the path length back up again. The relevant section becomes:
basilisk\099~1.59\cellassign-<cellassign's version here>\<env name here>
I guess I could further change behavior on windows so that we instead get:
cellassign\<cellassign's version here>\<env name here>
... which is probably fine, so long as you keep your environment names short. Like, really short.
A "better" solution would be to put the files somewhere else other than the cache. But there's really nowhere to go from C:\Users\pkgbuild\AppData\Local
without hitting system dirs!
Edit: 0.99.61 now has shorter paths for the base directory. I haven't changed the environment path structure yet - though I add the client's name, version and environment, I think we also lose the Python package name (tensorflow-base-2.1.0-mkl_py37h230818c_0
) which should be a better-than-even trade in most cases. So let's see how it goes; I have a standby solution in the windows
branch of basilisk that mimics a registry of "installed" environments, but I'd rather not use it.
have you
run into this solution?
The Windows API has many functions that also have Unicode versions to permit an extended-length path for a maximum total path length of 32,767 characters. This type of path is composed of components separated by backslashes, each up to the value returned in the lpMaximumComponentLength parameter of the GetVolumeInformation https://docs.microsoft.com/en-us/windows/desktop/api/FileAPI/nf-fileapi-getvolumeinformationa function (this value is commonly 255 characters). To specify an extended-length path, use the "\?\" prefix. For example, "\?\D:*very long path*".
Note
The maximum path of 32,767 characters is approximate, because the "\?\" prefix may be expanded to a longer string by the system at run time, and this expansion applies to the total length.
On Mon, Apr 13, 2020 at 10:50 PM Aaron Lun notifications@github.com wrote:
Wait. Wait. I realized what the problem was. Consider the full path of the first missing file:
C:\Users\pkgbuild\AppData\Local\me\basilisk\099~1.59\Cache\anaconda\pkgs\tensorflow-base-2.1.0-mkl_py37h230818c_0/Lib/site-packages/tensorflow-2.1.0.data/purelib/tensorflow_core/include/tensorflow_core/core/common_runtime/isolate_placer_inspection_required_ops_pass.h
Verily, it is beyond 260 characters, which is the file path limit on Windows! Ha!
Now, that is much, much harder to solve. The only part of that path that basilisk controls is:
me\basilisk\099~1.59\Cache\anaconda
Not much to cut off there. I could slim it to:
basilisk\099~1.59\00
... which gives us enough breathing space for the longest path ( transposer_factory.h).
However, I suspect that this just kicks the can down the road; one can be pretty sure that the environments will bump the path length back up again. The relevant section becomes:
basilisk\099~1.59\cellassign\
A "better" solution would be to put the files somewhere else other than the cache. But there's really nowhere to go from C:\Users\pkgbuild\AppData\Local without hitting system dirs!
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/Bioconductor/Contributions/issues/1178#issuecomment-613197345, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABDI5QQA7ZAGDCCMVWVCZ2DRMPFOFANCNFSM4H6MYMTQ .
-- The information in this e-mail is intended only for the person to whom it is addressed. If you believe this e-mail was sent to you in error and the e-mail contains patient information, please contact the Partners Compliance HelpLine at http://www.partners.org/complianceline http://www.partners.org/complianceline . If the e-mail was sent to you in error but does not contain patient information, please contact the sender and properly dispose of the e-mail.
have registry on the build system edited to allow long paths and give advice to users on how to do it themselves
https://www.howtogeek.com/266621/how-to-make-windows-10-accept-file-paths-over-260-characters/
On Tue, Apr 14, 2020 at 5:09 AM Vincent Carey stvjc@channing.harvard.edu wrote:
This is from https://docs.microsoft.com/en-us/windows/win32/fileio/naming-a-file?redirectedfrom=MSDN#maximum_path_length -- have you
run into this solution?
The Windows API has many functions that also have Unicode versions to permit an extended-length path for a maximum total path length of 32,767 characters. This type of path is composed of components separated by backslashes, each up to the value returned in the lpMaximumComponentLength parameter of the GetVolumeInformation https://docs.microsoft.com/en-us/windows/desktop/api/FileAPI/nf-fileapi-getvolumeinformationa function (this value is commonly 255 characters). To specify an extended-length path, use the "\?\" prefix. For example, "\?\D:*very long path*".
Note
The maximum path of 32,767 characters is approximate, because the "\?\" prefix may be expanded to a longer string by the system at run time, and this expansion applies to the total length.
On Mon, Apr 13, 2020 at 10:50 PM Aaron Lun notifications@github.com wrote:
Wait. Wait. I realized what the problem was. Consider the full path of the first missing file:
C:\Users\pkgbuild\AppData\Local\me\basilisk\099~1.59\Cache\anaconda\pkgs\tensorflow-base-2.1.0-mkl_py37h230818c_0/Lib/site-packages/tensorflow-2.1.0.data/purelib/tensorflow_core/include/tensorflow_core/core/common_runtime/isolate_placer_inspection_required_ops_pass.h
Verily, it is beyond 260 characters, which is the file path limit on Windows! Ha!
Now, that is much, much harder to solve. The only part of that path that basilisk controls is:
me\basilisk\099~1.59\Cache\anaconda
Not much to cut off there. I could slim it to:
basilisk\099~1.59\00
... which gives us enough breathing space for the longest path ( transposer_factory.h).
However, I suspect that this just kicks the can down the road; one can be pretty sure that the environments will bump the path length back up again. The relevant section becomes:
basilisk\099~1.59\cellassign\
A "better" solution would be to put the files somewhere else other than the cache. But there's really nowhere to go from C:\Users\pkgbuild\AppData\Local without hitting system dirs!
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/Bioconductor/Contributions/issues/1178#issuecomment-613197345, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABDI5QQA7ZAGDCCMVWVCZ2DRMPFOFANCNFSM4H6MYMTQ .
-- The information in this e-mail is intended only for the person to whom it is addressed. If you believe this e-mail was sent to you in error and the e-mail contains patient information, please contact the Partners Compliance HelpLine at http://www.partners.org/complianceline http://www.partners.org/complianceline . If the e-mail was sent to you in error but does not contain patient information, please contact the sender and properly dispose of the e-mail.
Received a valid push; starting a build. Commits are:
ea5329e Bump version number
Dear Package contributor,
This is the automated single package builder at bioconductor.org.
Your package has been built on Linux, Mac, and Windows.
On one or more platforms, the build results were: "skipped, ERROR, WARNINGS". This may mean there is a problem with the package that you need to fix. Or it may mean that there is a problem with the build system itself.
Please see the build report for more details.
@vjcitn
Re. unicode paths, I didn't know that, but I see that the conda
folks run into this same problem and didn't see this recommendation anywhere, so I assume that there is a reason why this doesn't work in general. Maybe it relies on a too-recent version of the Windows API? I dunno.
Re. registry, sounds a bit dangerous to me. Definitely a last-ditch option.
@LTLA wrong Vince, friend.
Sorry, got mixed up between @vjcitn's GitHub and slack handles!
Nope, I just use BioC's servers for all of my CI needs.
Just to give an update: the path length modification was added to 0.99.61 but has not propagated, being held up by unpredictable failures on tokay2. I hope it will get through to give enough time for cellassign testing, but if not, you may consider not supporting Windows for now.
What is the status of this package?
I'm hoping now Basilisk is established on bioconductor we can have another go and decide on whether to support windows
Hi Kieran, @kieranrcampbell
I think you can have another go. Aaron has made progress on getting it to build successfully on all platforms: https://community-bioc.slack.com/archives/CEQ04GKEC/p1589834161387300
http://bioconductor.org/checkResults/devel/bioc-LATEST/basilisk/
Dear Package contributor,
This is the automated single package builder at bioconductor.org.
Your package has been built on Linux, Mac, and Windows.
On one or more platforms, the build results were: "WARNINGS, skipped, ERROR". This may mean there is a problem with the package that you need to fix. Or it may mean that there is a problem with the build system itself.
Please see the build report for more details.
Oh goody. At last, an error that isn't basilisk's fault.
Needless to say, I think you can get rid of the .onLoad
statement now, given that you are now in control of your own destiny with respect to tensorflow's presence on the system. (Putting the check in the .onLoad
doesn't seem like a great idea in the first place, but whatever.)
@kieranrcampbell Any updates?
Hi @LiNk-NY
I'll take a look at this shortly. Plan to have wrapped up for next bioc release now basilisk is in
Thanks
Kieran
Hi Keiran, @kieranrcampbell
Any updates on the package? Otherwise I'm forced to close the issue until it's ready. Thanks! -Marcel
This issue is being closed because there has been no progress for an extended period of time. You may reopen the issue when you have the time to actively participate in the review / submission process. Please also keep in mind that a package accepted to Bioconductor requires a commitment on your part to ongoing maintenance.
Thank you for interest in Bioconductor.
Update the following URL to point to the GitHub repository of the package you wish to submit to Bioconductor
Confirm the following by editing each check box to '[x]'
[x] I understand that by submitting my package to Bioconductor, the package source and all review commentary are visible to the general public.
[x] I have read the Bioconductor Package Submission instructions. My package is consistent with the Bioconductor Package Guidelines.
[x] I understand that a minimum requirement for package acceptance is to pass R CMD check and R CMD BiocCheck with no ERROR or WARNINGS. Passing these checks does not result in automatic acceptance. The package will then undergo a formal review and recommendations for acceptance regarding other Bioconductor standards will be addressed.
[x] My package addresses statistical or bioinformatic issues related to the analysis and comprehension of high throughput genomic data.
[x] I am committed to the long-term maintenance of my package. This includes monitoring the support site for issues that users may have, subscribing to the bioc-devel mailing list to stay aware of developments in the Bioconductor community, responding promptly to requests for updates from the Core team in response to changes in R or underlying software.
I am familiar with the essential aspects of Bioconductor software management, including:
For help with submitting your package, please subscribe and post questions to the bioc-devel mailing list.