Closed kevinrue closed 1 month ago
In #7269, the undefined symbol was in _imaging.so, so I do think that's different.
Allow me to try and understand - you've pasted the features/report info, showing the Pillow installation and the versions of its dependencies. So you are able to install Pillow successfully. What you're talking about is something else importing Pillow, and that triggers the error.
that doesn't build on Linux at the moment
So it did build on Linux previously at https://bioconductor.org/checkResults/3.18/bioc-LATEST/velociraptor/nebbiolo2-install.html, but now fails at https://bioconductor.org/checkResults/devel/bioc-LATEST/velociraptor/nebbiolo2-buildsrc.html.
Any idea what triggered this change?
to run some Python code internally.
From a quick browse of https://github.com/kevinrue/velociraptor, is scvelo where this becomes a Python matter? So was this triggered by the recent https://github.com/kevinrue/velociraptor/pull/71?
Ultimately, I'm not sure that we'll be able to fix this from our end. It sounds like the error is in how libtiff was built, and we don't control the build process of conda-forge pillow - that is maintained over at https://github.com/conda-forge/pillow-feedstock. Or maybe it is even a task for https://github.com/conda-forge/libtiff-feedstock, since they will willing to deal with https://github.com/conda-forge/libtiff-feedstock/issues/78
Allow me to try and understand
I think you've summarised it beautifully.
Any idea what triggered this change? / was this triggered by the recent https://github.com/kevinrue/velociraptor/pull/71?
Yes. The main thing that changed is the Conda environment, i.e. package versions. At least the OS is the same, and the Bioconductor core team who maintains the build system hasn't flagged any obvious change in the system. They seem as puzzled as I am.
You will see in the last commit before the PR that - when things worked - we used to pin a lot less dependencies (https://github.com/kevinrue/velociraptor/blob/dbc6491d830ec195ebe7ea14412157c7e3fb3f74/R/basilisk.R#L2-L72). The motivation for reviewing the dependencies was that:
is scvelo where this becomes a Python matter?
Yes
It sounds like the error is in how libtiff was built / Or maybe it is even a task for https://github.com/conda-forge/libtiff-feedstock
Sounds like I better open an issue there, linking to this one, and see what they say. Thanks a lot for your time and thorough review!
If you just run
from PIL import Image
im = Image.new("RGB", (1, 1))
im.save("out.tif", compression="jpeg")
does the error occur?
Our Linux cluster is offline for maintenance today and tomorrow. I'll test this on Friday if everything is back online. Thanks!
Actually, while waiting for the cluster to come back online, I've used GitHub Action to run the code you gave above. The error does not occur:
https://github.com/kevinrue/velociraptor/actions/runs/9594803654/job/26458166745#step:18:29
Just for the record, proof that I've run that command in the environment that raises the error undefined symbol: jpeg12_write_raw_data, version LIBJPEG_8.0
:
https://github.com/kevinrue/velociraptor/actions/runs/9594803666/job/26458166748#step:17:1966
Did the information above help?
Having a GitHub Actions reproduction is definitely helpful. I haven't had as much time as I would like to investigate this, but here are my thoughts so far.
The fact that you're able to install and run Pillow code that uses libtiff without an error points to this not actually being a Pillow problem. I understand that the error message mentions Pillow, but I'm not sure this will be able to be replicated without scvelo.
Even if you can't replicate it without scvelo, it would be helpful if you could trigger the problem starting from Python, rather than from velociraptor.
Even if you can't replicate it without scvelo, it would be https://github.com/conda-forge/libtiff-feedstock/issues/104#issuecomment-2176006361 if you could trigger the problem starting from Python, rather than from velociraptor.
Good point. I've reproducing the error the only I know how, but I'll see if I can figure out a way to trigger the problem directly from Python. (I'm also short on time)
Many thanks for your time!
This micromamba environment and Python script simulate the workflow but do not trigger the issue from Python while it still occurs from velociraptor: https://github.com/kevinrue/velociraptor/issues/78#issuecomment-2314867022
In the end I think there's nothing wrong with pillow.
The issue seems to relate to a difference in environment between the R CMD build
and R CMD check
commands.
This command fails
remotes::install_local("/project/sims-lab/albrecht/git/kevinrue/velociraptor", dependencies = TRUE, repos = BiocManager::repositories(), build_vignettes = TRUE, upgrade = TRUE, force = TRUE, lib = "/project/sims-lab/albrecht/scvelo-test/test-velociraptor/bioc_320")
While this one works, and subsequently runs the velociraptor code fine in an interactive session.
remotes::install_local("/project/sims-lab/albrecht/git/kevinrue/velociraptor", dependencies = TRUE, repos = BiocManager::repositories(), build_vignettes = FALSE, upgrade = TRUE, force = TRUE, lib = "/project/sims-lab/albrecht/scvelo-test/test-velociraptor/bioc_320")
The only difference being build_vignettes = TRUE|FALSE
meaning that for some reason the libtiff symbol is not found during R CMD build
.
This SO thread points to the plausible reason, but isn't enough for me to fix my situation.
Thanks for your help. I think you can close this issue now.
Thanks very much for your further investigation.
For what it's worth, I've finally sidestepped the issue by adding anaconda
(first) in the channels.
Interestingly, it's reducing the number of conda packages in the environment by 57.
Not trying to reopen this issue, but in case you're interested you can compare the environment that causes me problems (without anaconda) https://github.com/kevinrue/velociraptor/blob/devel/inst/envs/Linux_x86_64.yaml vs the one that finally got me going again (with anaconda) https://github.com/kevinrue/velociraptor/blob/devel/inst/envs/Linux_x86_64_anaconda.yaml
The version of pillow is the same in both cases. No clue what's causing/fixing the issue.
What did you do?
I am debugging an issue for a Bioconductor R package that doesn't build on Linux at the moment: https://bioconductor.org/checkResults/devel/bioc-LATEST/velociraptor/nebbiolo2-buildsrc.html
I have replicated the issue on a HPC at work. However, the HPC does not provide the version of R that I need, I have followed IT guidelines and used Apptainer to work in a container that provides everything I need to replicate the issue.
The issue arose when I ran
R CMD build
on the Bioconductor R package that uses a Conda environment to run some Python code internally.What did you expect to happen?
The package used to pass
R CMD build
, e.g. https://bioconductor.org/checkResults/3.18/bioc-LATEST/velociraptor/nebbiolo2-install.htmlWhat actually happened?
The R function that creates the Conda environment and runs the Python code crashes with the error
Which I have seen in the previous issue https://github.com/python-pillow/Pillow/issues/7269 but cannot find a way to fix in my case.
What are your OS, Python and Pillow versions?