search

jump-cellpainting / 2024_Chandrasekaran_NatureMethods

BSD 3-Clause "New" or "Revised" License
55 stars 11 forks source link

Conda list of this environment available? #69

Closed bethac07 closed 1 year ago

bethac07 commented 1 year ago

Does the exact environment used to generate these profiles still exist on an extant machine somewhere, and if so, could we conda list it? Others/future-us might need to tease out version-related issues, and so if you can't use an old conda recipe due to certain pinned versions no longer available (#67), it might be nice to have.

I'm guessing not, based on the later comments in #67, but maybe start doing so for the new environment? I've also added it to our list of recipe wishes

niranjchandrasekaran commented 1 year ago

It seems like the environment that was used for running the notebooks is broken. But the one used to create the profiles is still working.

bethac07 commented 1 year ago

It is not working on my machine (almost certainly an M1 issue), and not for at least one other user (no idea the configuration). https://github.com/cytomining/profiling-recipe/issues/35

bethac07 commented 1 year ago

And even if it is working on other machines, to my understanding, environments are not 100% deterministic for packages that aren't pinned - ie, if pandas==1.2.0 requires numpy>=1.21, when you build it on X date vs Y date, if numpy 1.23 vs 1.24 is what's available, users building on different dates might get different packages from the resolver. Is my understanding correct? I believe if we wanted always-identical environments, we'd need to use conda list --explicit or link out to a Docker container.

niranjchandrasekaran commented 1 year ago

Ah thanks for linking to that issue on profiling-recipe (I seem to have missed it). The only reason those specific versions were used was because those were the ones that worked nicely with each other. Of course, there are likely newer versions of each of the package that may also work with each other but I haven't had the chance to check them.

And even if it is working on other machines, to my understanding, environments are not 100% deterministic for packages that aren't pinned - ie, if pandas==1.2.0 requires numpy>=1.21, when you build it on X date vs Y date, if numpy 1.23 vs 1.24 is what's available, users building on different dates might get different packages from the resolver. Is my understanding correct?

That's my understanding as well.

bethac07 commented 1 year ago

Cool, so that then goes back to my question - is the exact environment used to make these profiles still in existence somewhere, either on your personal machine or a VM? If so, might be nice to add a conda list --explicit of what was in it in a file or in this issue, for future tracing purposes. If not, we should close here and maybe discuss in the recipe wishes issue.

niranjchandrasekaran commented 1 year ago

Cool, so that then goes back to my question - is the exact environment used to make these profiles still in existence somewhere, either on your personal machine or a VM? If so, might be nice to add a conda list --explicit of what was in it in a file or in this issue, for future tracing purposes.

Ah, I see. I don't believe that the original environment exists anywhere. It has probably been rewritten as I have processed newer datasets.

If not, we should close here and maybe discuss in the recipe wishes issue.

Sounds good.