CZI Proposal Spending of Funds

[CJ-Wright]

@conda-forge/core This vote falls under the Spending of funds policy, please vote and/or comment on this issue. This vote needs 50% of core to vote yea to pass. To vote please leave :+1: (yea) or :-1: (nay) responses on this comment. If you would like changes to the current language please leave a comment. This vote will end on August 4th 2020 at noon Eastern.

@ocefpaf has put together a proposal for funding to

Improve the packaging ecosystem around conda by developing a robust open-source alternative to the proprietary hosting service operated by Anaconda, and furthering the development of mamba, the fast drop-in replacement for conda.

It would be good if we could decide on if we want to spend funds along these lines so we do not run into a situation where we have funds but don't allocate them.

[CJ-Wright]

On a related but somewhat separate note, this is this first time this kind of vote has come up. We do not have procedures for voting on grant submissions/proposals. I would like to get some sense as to if this approach is appropriate and to help build a precedent to cover these cases (or if a precedent can not be built, spur the modification of the governance document). I would appreciate it if members of core could please vote on this comment (:+1: :-1:) if they think this precedent (calling a Spending of funds vote before the submission of the grant) is good and proper.

[CJ-Wright]

Just for clarification, the budget vote is the first comment, the precedent vote is on the second comment.

[wolfv]

@CJ-Wright I think you missed a link to the proposal in question:

https://docs.google.com/document/d/1dEXQsrXM5kBW6ARLpsVGRQltVxQVbVkjmXbBh8LnHac/edit

This proposal has been up for criticism & change proposals since a month and we had only a few remarks. Before people vote >NO< here I'd be happy if they take the time and criticize the proposal as well :)

[wolfv]

Also, the deadline of the vote coincides with the proposal submission deadline. If I could motivate all of @conda-forge/core to vote as soon as possible that would be awesome so we can take the appropriate actions.

@beckermr you have voted on the meta-question, but not on the question itself? @CJ-Wright you have opened the initial question but not voted yourself, could you do that please?

[marcelotrevisani]

I believe that this voting part which @CJ-Wright is proposing is a bit late for this CZI at least. Since the document is public and it was mentioned several times in other conda-forge meetings and it was/is open for criticism and if anybody was against it could just spoke up about it. I don't see this current voting as beneficial as the proposal was published for about a month or so (as mentioned by @wolfv ). Not that I am against it, but I didn't get the point here.

But we can do it for the next iterations along with the document published. I mean, when the person publishes the first version of the document we will have time to review, make suggestions, or reject the proposal and we can add a deadline for reviewers as well.

[marcelotrevisani]

and also voting as :+1: in two comments, I personally do not see that as a straight forward solution for a voting system. For me it seems to be a bit confusing :grimacing:

[CJ-Wright]

@CJ-Wright you have opened the initial question but not voted yourself, could you do that please?

I thought this would inject some unneeded bias to vote first, but I have voted now if that is helpful.

This proposal has been up for criticism & change proposals since a month and we had only a few remarks. Before people vote >NO< here I'd be happy if they take the time and criticize the proposal as well :)

I don't think this is strictly necessary. Some people may disagree with the proposal on existential grounds, where no level of critique may address their issues.

Also, the deadline of the vote coincides with the proposal submission deadline.

I thought the deadline was the 5th so I attempted to schedule it to end before hand.

I believe that this voting part which @CJ-Wright is proposing is a bit late for this CZI at least.

That's a fair criticism, I wasn't certain if this would be needed or if the proposing parties would call it themselves as part of due diligence. As I mentioned at the core meeting I'm not certain if this is needed and proposed this out of an abundance of caution.

if anybody was against it could just spoke up about it.

As mentioned above I think there is a strong difference between modifying the language of the proposal and being in favor of spending funds in this direction.

Not that I am against it, but I didn't get the point here.

The purpose of this is to give a strong mandate to allocate and spend the funds. We could do this voting after the proposal has won and we have been given the funds, but this runs the risk of the vote not passing. In that case we would have the funds but wouldn't be able to spend them, which I think is a bad outcome. This vote would help to address these issues by getting the determination of if conda-forge wants to spend funds in this direction out of the way.

[CJ-Wright]

and also voting as +1 in two comments, I personally do not see that as a straight forward solution for a voting system. For me it seems to be a bit confusing grimacing

I happy to move this to a better voting platform if one such exists and doesn't require too much setup.

[msarahan]

I'm voting no on the proposal because I support Quetz as an open source package repo, but "replacing conda/conda-build" is fractious and disrespectful of the time and effort that Anaconda has put into maintaining an open project. It is a hostile fork of sorts. Is it hard to work with a large project with lots of code smell? Yes. Is it impossible? No. Can you use Febreze on smelly repos? Maybe. Have Mamba and Boa primarily served to avoid social problems with open source? Seems likely. We should not aim to fund the avoidance of social problems. Given the recent developments of the conda-tools organization and the overall willingness of Anaconda to come to the table, I would only support "mamba as a drop in replacement" if that was the direction that Anaconda wanted for conda.

[pkgw]

I guess I'm unclear on the policies for when we use Helios for voting and when we use something else like GitHub reactions (anonymous vs. not?) but that's what I get for not being very plugged in to the governance activities. (I have a standing weekly meeting that conflicts with the core meetings, unfortunately.)

I voted "no" on the second "precedent" question just because I think it would be good to take the time to think things through for a bit longer. Deciding to submit a proposal is certainly related to a "spending of funds" question, but they're not the same thing. If nothing else, this is a good flag that the governance documents should explicitly describe a process here — I'd rather not hack it in as an overload of "spending of funds".

[sodre]

I voted no as I agree with @msarahan. Is there any chance of scoping down the proposal to only include Quetz?

[SylvainCorlay]

"replacing conda/conda-build" is fractious and disrespectful of the time and effort that Anaconda has put into maintaining an open project.

Developers of mamba & quetz are conda fans who have been pushing for conda publicly and behind closed doors at companies. The "serve as a drop-in replacement" is meant to emphasize our intent to be compatible with the existing tools, even for the command-line interface, and it was certainly not meant in a disrespectful way.

In the early days of the mamba projects, we reached out to Anaconda about a possible collaboration to integrate our developments upstream, and it did not happen unfortunately. Mamba's codebase is separated from conda, but I see this more as a reboot than an hostile fork, as we have been reaching to everyone to get involved.

[ocefpaf]

I voted no as I agree with @msarahan. Is there any chance of scoping down the proposal to only include Quetz?

While I agree with Mike too it is important to note that mamba is part of our stack and many communities are already using it. So it is important to foster it's development. Bare in mind that this is not deliberate! There was multiple attempts to work with conda and conda-build before. Also the time does not does allow for huge scope changes like that.

[msarahan]

I edited the doc to change the language around the conda/conda-build relationship. If the changes are accepted, I think I can vote for the proposal.

[SylvainCorlay]

I edited the doc to change the language around the conda/conda-build relationship. If the changes are accepted, I think I can vote for the proposal.

This is awesome. I accepted all your suggestions (and I saw that you approved my minor edits).

[ocefpaf]

@sodre do you still have any comments that needs to be addressed?

[sodre]

I changed my vote to yes.

With that being said, I would like to point out that Sonatype Nexus is open source and supports proxying conda packages. There is an open PR sonatype/nexus-public#50 to support hosting packages as well. I suggest we include that in the proposal.

[ocefpaf]

I changed my vote to yes.

:+1:

With that being said, I would like to point out that Sonatype Nexus is open source and supports proxying conda packages. There is an open PR sonatype/nexus-public#50 to support hosting packages as well. I suggest we include that in the proposal.

Wow! I had no idea! I'm not sure we can add it this late but we definitely should contact them and work out some sort of collaboration!

[ocefpaf]

@sodre I'm reading the Sonatype stuff and I would like to contact them. Do you know someone in there or you just found out about it online? If the former can you put me in touch with them?

PS: you are counting as voting twice (+1 and -1)! You need to remove the previous vote (-1) by clicking on it again.

[sodre]

@sodre I'm reading the Sonatype stuff and I would like to contact them. Do you know someone in there or you just found out about it online? If the former can you put me in touch with them?

PS: you are counting as voting twice (+1 and -1)! You need to remove the previous vote (-1) by clicking on it again.

@ocefpaf, I fixed the downvote issue.

Unfortunately, I don't have any contacts at Sonatype. However, I have been using their repo for about 5months together with a very simplistic conda server I wrote for myself.

[ocefpaf]

Unfortunately, I don't have any contacts at Sonatype. However, I have been using their repo for about 5months together with a very simplistic conda server I wrote for myself.

We should talk then ;-p This should be in our sights seems it is something that works already. I'll do more research on this. Thanks!

[scopatz]

Hey all! Reading through the proposal, it seems like this is something that QuantStack or SnakePit should put in. What am I missing? Why is this a conda-forge propsal?

Also - in the past conda-forge has been discouraged from putting in CZI proposals because of a couple of points from the impact:

• Demonstrated scientific impact of the software project,
• Alignment of the software project to areas currently prioritized by CZI Science in accordance with our mission (e.g. imaging, cell biology, genomics).

So not to rain on anyone's parade, but if conda/conda-forge has had a hard time justifying ourselves to CZI, it seems like mamba/boa/quetz are going to have an even harder time.

I haven't voted yet.

[ocefpaf]

Hey all! Reading through the proposal, it seems like this is something that QuantStack or SnakePit should put in. What am I missing? Why is this a conda-forge propsal?

It is a conda-forge proposal b/c it is a community effort to improve the tools conda-forge is using and wants to adopt in the long run. We are already using it on the bot and quetz may be crucial in the future if we want to host packages. That is in the proposal BTW.

Also - in the past conda-forge has been discouraged from putting in CZI proposals because of a couple of points from the impact: So not to rain on anyone's parade, but if conda/conda-forge has had a hard time justifying ourselves to CZI, it seems like mamba/boa/quetz are going to have an even harder time.

That is not true. I met with CZI folks who encourage us to submit. I also read past proposal and our impact is as strong as many others that got funding. Last but not least, I acted a a reviewer to proposal that got funded and had no direct impact.

I haven't voted yet.

Please do. Also, if you have any suggestions on how to strengthen our proposal make comments and edits on the document. It is open for a month now and many core members already chimed-in! You are not too late. We plan to submit at the deadline.

[scopatz]

Thanks @ocefpaf

[scopatz]

My main point is that may be crucial in the future does not seem to align with Demonstrated scientific impact of the software project. Mamba is the only one of these that I think is even close to demonstrating impact to conda-forge, and it is isn't really production ready in terms of the number of eyes & installs relative to conda itself.

I'll comment on the document

[scopatz]

Don't get me wrong, I want conda-forge to have more funds and see more development. But I am still having a hard time seeing how this affects the things that we do in terms of packaging, since this is wholly based on improving tools. So in my mind it would be best if a tools org took this and put it in.

[scopatz]

Actually, I can't even comment on the document

[ocefpaf]

Actually, I can't even comment on the document

You have to request access. We did sent an invite on the core meetings for everybody. Sorry that you missed.

[wolfv]

In the first cycle, pip got funding and fundamentally that doesn't sound like a project that's far away from what we're proposing here (although obviously it has many more users). https://chanzuckerberg.com/eoss/proposals/improving-user-experience-and-debuggability-of-pip-for-all-python-users/

Also we got extremely encouraging feedback especially from the bioconda community. Mamba is advertised as the "installation method of choice" for snakemake, which seems to be a very used tool https://snakemake.readthedocs.io/en/stable/getting_started/installation.html You can also check the core-x-conda-forge channel on gitter.

So I think your points are unfair, @scopatz

Also, isn't that up to the CZI jury to decide? :) Would be cool if you help us to make the proposal as strong as possible.

[scopatz]

Yeah, I mean fundementally it is up to CZI to decide. I mean, I am asking these questions to make it the best proposal possible. Again I would love to see these efforts have more funding.

It does would be stronger outside of conda-forge. It sounds like even bioconda has a stronger support case here. Conda-forge (against my personal wishes) can't even ship an miniforge installer with mamba yet. And that is an example of the intersection of mamba and conda-forge that is missing from the current propsal. I am not trying to be a downer here, it is just that we decided that conda-forge is for packaging and tools live elsewhere. As written this has little to do with actual packages and more to do with tools. Other ways to imporve (since I don't have edit access still, though I did request it):

• Update staged-recipes to use mamba/boa optionally
• Work on conda-smithy to use mamba/boa optionally
• Integrate mamba etc with bot
• Figure out how to host quetz for conda-forge
• Generally cut down on the negative language. It is better to phrase things positively and in terms of improvements.

Does this make sense in terms of building this out as a packaging & conda-forge proposal?

[wolfv]

it seems like this is something that QuantStack or SnakePit should put in. What am I missing? Why is this a conda-forge propsal?

The entire idea of this proposal is that we get money to develop these tools, and conda-forge would have the oversight over the money so that the PM of the project & the conda-forge community could weigh in on what get's developed. I think there is also room to diverge from the proposal language to the actual implementation of features, and we'd concentrate to develop those features that'll make sense to conda-forge. E.g. there would probably be monthly invoicing and lots of control for cf.

So in fact it would be "free money" + "control" over these tools that have a demonstrated positive potential for the ecosystem. Where is the downside?

Some other work we've done lately: we have already forked conda smithy to enable uploading with the quetz-client here: https://github.com/TheSnakePit/conda-smithy I've been using boa to do huge rebuilds of the ROS distro successfully (in fact I think that uncovered the recent bug in the linux-sysroot package: https://github.com/conda-forge/linux-sysroot-feedstock/pull/32) You can find the boa-build logs for hundreds of ROS packages here: http://robostack.net:8000/RoboStack/vinca/56/1/2 This is also an effort to stabilize boa to a point where we can discuss it further in the conda-forge community.

I also had a really long and great discussion with @CJ-Wright on Friday and I'll try to address some of his ideas on how to make the proposal stronger on Monday. I am also really excited about some of the ideas that we discussed for the bot and would be very happy to see what we can do i.e. with quetz for this -- one thing in particular seems to be creating a database of symbols that are stored in .so files or Python function signatures and trying to automate version updates against those. If we have a quetz server, we could upload packages to it and index this metadata to take the bot to the next level :) These kind of ideas are things that we -- as a community -- have the possibility to explore next year if we can get funding.

[ocefpaf]

That would expand the goal extensively and we should keep our goals feasible. I'm not saying no but we should be aware of the deliverables. Also, most of that seems to be work for after we have a better working stack of tools, not before. Let's improve mamba first, make it a first class citizen in the community, then we can write more grants to work on those integrations.

[wolfv]

What I am saying is that we could have hooks in quetz to extract further metadata from packages and store it in a database for bot consumption, as an example.

[scopatz]

I think that is all fine well and good. There is no problem with that. I am all for making these tools first class citizens in general. It is just hard for me to see this as a conda-forge proposal then, without the conda-forge components. Maybe once these tools are first class citizens it would be better to revist an integration proposal. What is the reason for putting it in in conda-forge rather than elsewhere? Anyone can put in a proposal, so why here? I understand the justifications that you are making on the technical side on why this proposal should be funded in general.

What I am saying is that we could have hooks in quetz to extract further metadata from packages and store it in a database for bot consumption, as an example.

For example, this isn't connecting for me. If this is a goal, why not do that AND connect it to conda-forge's bots as part of the work here?

That would expand the goal extensively and we should keep our goals feasible. I'm not saying no but we should be aware of the deliverables.

Agreed, so maybe it would be better to focus on just one of these tools and develop the full vertical integration with conda-forge as the main goal. You can always add stretch goals of what we would do (ie the other projects) if there was money left over or as needed incidentally. That would be a much stronger proposal in my opinion.

[ocefpaf]

Agreed, so maybe it would be better to focus on just one of these tools and develop the full vertical integration with conda-forge as the main goal. You can always add stretch goals of what we would do (ie the other projects) if there was money left over or as needed incidentally. That would be a much stronger proposal in my opinion.

We disagree and that is why we wrote it to foster the 3 main bases that we need before the "verticalization." It is stronger this way in the opinion of all those who reviewed and contribute to it in the past month.

[scopatz]

OK - yeah, I think we disagree. All I need to be able to vote for this is some change (anything really) that directly touches conda-forge

[scopatz]

Thanks @SylvainCorlay for adding the conda-forge parts to the proposal. I have voted for this now.

[wolfv]

cool!

[mbargull]

I've finally re-read the current version of the proposal. Have to say, I'm very glad the wording has been changed in ways of

The long-term goal is to bring Conda and Mamba together, so that there is a unified set of community tooling that is as efficient as possible.

which nicely reflects the "Conda community" effort that has been started in the meantime.

I am also glad that this will be submitted under "conda-forge". Conda-forge has stronger/directer tie-ins with Anaconda and QuantStack than, say, Bioconda, which I am sure will facilitate this undertaking a lot. Plus, IMHO, the largest potential of improvement can be found at (as in during the guided development of the tools) and implemented at conda-forge not alone due to the sheer size of the project. But also other projects that are effectively "downstream" of conda-forge, e.g., Bioconda (as also mentioned in the proposal with "Conda-forge is at the foundation of the Bioconda project"), will benefit by proxy from more efficient processes at conda-forge, or even directly by adopting some of the tools in their pipelines.

[CJ-Wright]

As of 12:04 Eastern there are 15 yeas and 2 nays. This vote has passed and should the grant be funded the finance subteam is instructed to disperse funds accordingly.

[CJ-Wright]

As of 12:11 Eastern there are 6 yeas and 2 nays for forming a precedent around this approach for submitting grants. While there are no formal requirements for this vote as it is non-binding it does seems that a majority of those voting are in favor.

[CJ-Wright]

Thank you all who voted, commented and otherwise provided input to this process.