search

nextflow-io / nextflow

A DSL for data-driven computational pipelines
http://nextflow.io
Apache License 2.0
2.76k stars 629 forks source link

Nextflow contingency plan #1508

Closed grst closed 4 years ago

grst commented 4 years ago

This might be a bit of an uncomfortable question and I hope I don't offend anyone, but as Nextflow is gaining popularity this is becoming important.

What happens to Nextflow if @pditommaso

From the contributors chart it doesn't look like anybody else has a solid knowledge of the codebase, except for, maybe, @ewels.

drpatelh commented 4 years ago

We need to wrap @pditommaso in cotton wool and hope for the best i.e. he doesn't come into close contact with :fire:

gets bitten by a Velociraptor

:rofl: As far as I know he and Evan haven't been invited to talk at Jurassic World. Delays reopening mainly :roll_eyes:

pditommaso commented 4 years ago

Interesting "issue". What do you think it should be done to have more core contributors?

grst commented 4 years ago

In the end this boils down to funding...

Some random thoughts:

olgabot commented 4 years ago

Interesting points, @grst. I think the same argument/concern could be brought up about other widely used open source projects such as C Python and pandas, which at different points in time were completely dominated by the contributions of a single person.

pandas

Screen Shot 2020-02-28 at 9 23 46 AM

CPython

Screen Shot 2020-02-28 at 9 24 00 AM

Not too sure by what is meant by "investing a lot in Bioinformatics infrastructure" here:

czbiohub (ping @olgabot) is investing a lot in Bioinformatics infrastructure and also uses nextflow for their projects. Maybe they would dedicate some of their time to nextflow.

We are indeed using Nextflow pipelines for many projects, but we don't currently have any Groovy developers who could contribute to Nextflow itself.

I'm not familiar with how a project becomes a member of Apache Software Foundation, or whether it introduces any restrictions in how one's project can be run, but it seems more industry/production focused than science. If being part of a nonprofit software foundation is important, another option is NumFOCUS, which has strong ties to ContiuumIO, the company behind Anaconda.

Sequera Labs is an excellent way to go as Software-as-a-Service (SAAS) models have been successful in the past, especially Open SAAS like WordPress which by some calculations makes up ~60% of the internet.

grst commented 4 years ago

Hi @olgabot,

thanks for your input!

Not too sure by what is meant by "investing a lot in Bioinformatics infrastructure" here:

I was referring to the EOSS grants. But of course, funding is something different than providing developers.

I think the same argument/concern could be brought up about other widely used open source projects such as C Python and pandas, which at different points in time were completely dominated by the contributions of a single person.

Then the real question is, how did these projects make the step from a single developer to a core team?

olgabot commented 4 years ago

I was referring to the EOSS grants. But of course, funding is something different than providing developers.

Ah yes, that is a common misconception. "Chan Zuckerberg Biohub (@czbiohub)" != "Chan Zuckerberg Initiative (@chanzuckerberg)". CZ Biohub (https://www.czbiohub.org/) is a nonprofit, 501(c)3 biomedical research institute that does not give out international grants. CZI (https://chanzuckerberg.com/) is a separately run organization, with many initiatives in Healthcare, Education, Criminal Justice Reform, and one initiative is Science.

pditommaso commented 4 years ago

Thanks for the interesting discussion. This is indeed a recurrent question about the state of the project.

Olga is right mentioning that is quite common that on OSS projects most of the dev is done by one of a very few maintainers eg. https://twitter.com/Sydonahi/status/991797172877381632

At the same time, I understand the worry and I agree that having more recurrent contributors would improve the perceived health status of the project.

This was also one of the main reason why some years ago the license from GPL to Apache, ie. to encourage people and orgs to contribute code without having to worry to much about the legal implications of GPL licensed code. After this change, the number of users has actually increased, but the impact of contributors was little.

Then, what could be done to improve this situation:

Still, the ideal solution would be more individual developers and organisation step-in to provide any kind of code contributions.

Maybe a way to promote NF contributions could be to include in the future nf-core hackathons a NF core (name collision!!) development and hacking session. How to code is organised, how to build, test and debug it, etc.

What do think @drpatelh @ewels ?

ewels commented 4 years ago

Maybe a way to promote NF contributions could be to include in the future nf-core hackathons a NF core (name collision!!) development and hacking session. How to code is organised, how to build, test and debug it, etc.

I think this would be excellent! I think it would be particularly well suited to remote hackathons, as I suspect that people with the technical knowhow to get into the guts of Nextflow may be fairly well spread. But if we have very concrete tasks with plenty of help then that could be a really good way to get people over the hurdle of their first contribution. Even if some tasks are just related to improving documentation, that would be something.

I think that part of the "problem" here is that @pditommaso does such a good job of maintaining and developing Nextflow! It makes it a little difficult to justify the effort of getting deep into the codebase when you know that @pditommaso will do a much better job in a fraction of the time if you just ask him nicely 😆 But I agree that it would be good for the health of the project to get a wider spread of people with in-depth knowledge of the codebase (or at least parts of it).

It's worth noting that the CZI EOSS is only one of a large number of grants / projects that have been applied for which have at least some aspect of Nextflow development. The hope is that some dedicated personnel funding may start to materialise in the coming months / years.

drpatelh commented 4 years ago

Yes, I agree with @ewels thoughts. @pditommaso is doing such a great job in developing NF and making sure that it just works through his meticulous screening of what actually gets into the code base. I agree that much of this concern will hopefully be alleviated when Seqera Labs matures and acquires adequate seed funding - something which obviously takes time.

Having a nextflow/nf-core Hackathon back to back is a great idea and would definitely push more people to contribute directly to the code base. Even if we were to address issues that we find ourselves saying "We should add this functionality directly to Nextflow and not nf-core!". However, I suspect this would need to be a week long Hackathon if run back to back or run separately over the course of a few days.

Unfortunately, I guess the biggest obstacle is knowing how to be Groovy with your contributions but as @ewels suggested improving the existing documentation would be a good start. I think we can probably discuss this in a little more detail tomorrow at the Hackathon 👍 😎

stale[bot] commented 4 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.