Refactor BHMM core into separate repo from spectroscopic analysis using BHMMs?

jchodera commented 9 years ago

It sounds like @franknoe would like to use the BHMM model estimation core in multiple projects that have nothing to do with spectroscopic analysis, but are instead pure machine learning projects.

The best way to do this is to separate out the BHMM model estimation core into a separate repository (or extract the single-molecule spectroscopy parts into another project).

This could optimally be made compatible with scikit-learn, which seems to be the modern way new machine learning codes are written.

franknoe commented 9 years ago

Splitting the sklearn question off to #42

jchodera commented 9 years ago

I'm finally back to working on this again.

Did we want to proceed with keeping BHMM-only functionality in this repo and splitting out force spectroscopy stuff into, say, bhmm-force-spectroscopy? Would you be OK with that?

If so, I can do that today.

franknoe commented 9 years ago

Yes please!

Thank you.

Am 23/06/15 um 16:20 schrieb John Chodera:

I'm finally back to working on this again.

Did we want to proceed with keeping BHMM-only functionality in this repo and splitting out force spectroscopy stuff into, say, |bhmm-force-spectroscopy|? Would you be OK with that?

If so, I can do that today.

— Reply to this email directly or view it on GitHub https://github.com/bhmm/bhmm/issues/40#issuecomment-114521455.

Prof. Dr. Frank Noe Head of Computational Molecular Biology group Freie Universitaet Berlin

Phone: (+49) (0)30 838 75354 Web: research.franknoe.de

Mail: Arnimallee 6, 14195 Berlin, Germany

franknoe commented 9 years ago

I'll start doing this separation, because we need it for pyemma.

jchodera commented 9 years ago

Sorry about the delay here. Are you doing this now? Otherwise, I can tackle it right now.

My plan was to just peel out the force spectroscopy stuff into a separate repo bhmm-force-spectroscopy. The main stuff would stay in this repo.

jchodera commented 9 years ago

Or did you mean that you needed some other components that aren't here already for pyemma?

franknoe commented 9 years ago

OK, if you can do it please go ahead. I am not very familiar how to split git histories.

Apparently bhmm is currently so large (several hundreds of MB - there is manuscript stuff, reference pdfs and matlab stuff) that it is prohibitively huge for travis, so we would need a pure code package to be used as a part of pyemma. That should also only include the relevant part of the git history.

Everything else can be split off into bhmm-force-spectroscopy (I have actually created a forcespec repo because I though - and _ are discouraged in package names, but I'm happy to use bhmm-force-spectroscopy).

Thanks!

Am 08/07/15 um 19:26 schrieb John Chodera:

Sorry about the delay here. Are you doing this now? Otherwise, I can tackle it right now.

My plan was to just peel out the force spectroscopy stuff into a separate repo |bhmm-force-spectroscopy|. The main stuff would stay in this repo.

— Reply to this email directly or view it on GitHub https://github.com/bhmm/bhmm/issues/40#issuecomment-119670470.

Prof. Dr. Frank Noe Head of Computational Molecular Biology group Freie Universitaet Berlin

Phone: (+49) (0)30 838 75354 Web: research.franknoe.de

Mail: Arnimallee 6, 14195 Berlin, Germany

jchodera commented 9 years ago

OK, I'll take care of this in a few min.

jchodera commented 9 years ago

Do you think the size of the commit history for bhmm will cause problems if we keep this repo? If so, I can rename bhmm to bhmm-manuscript and then split off just the critical BHMM bits into a new repo called bhmm and just the force spectroscopy tools into a new repo called bhmm-force-spectroscopy.

franknoe commented 9 years ago

yes, better! I think there's something like git subtree to do that (and keep the history), but I've never used it.

Am 08/07/15 um 19:47 schrieb John Chodera:

Do you think the size of the commit history for |bhmm| will cause problems if we keep this repo? If so, I can /rename/ |bhmm| to |bhmm-manuscript| and then split off just the critical BHMM bits into a new repo called |bhmm| and just the force spectroscopy tools into a new repo called |bhmm-force-spectroscopy|.

— Reply to this email directly or view it on GitHub https://github.com/bhmm/bhmm/issues/40#issuecomment-119675088.

Prof. Dr. Frank Noe Head of Computational Molecular Biology group Freie Universitaet Berlin

Phone: (+49) (0)30 838 75354 Web: research.franknoe.de

Mail: Arnimallee 6, 14195 Berlin, Germany

jchodera commented 9 years ago

OK, stay tuned.

jchodera commented 9 years ago

First try didn't work. Trying a new approach based on this: http://stackoverflow.com/questions/359424/detach-subdirectory-into-separate-git-repository

jchodera commented 9 years ago

OK, it finally worked, using this strategy: http://jimmy.schementi.com/splitting-up-a-git-repo/

franknoe commented 9 years ago

Great, thank you!!!

The new repo bhmm-force-spectroscopy-manuscript still contains the bhmm sources. I guess the easiest thing to do would be to make a new release of the bhmm core package, then import bhmm in the examples of bhmm-force-spectroscopy-manuscript, and then delete the bhmm subdirectory.

Am 09/07/15 um 06:31 schrieb John Chodera:

OK, it finally worked, using this strategy: http://jimmy.schementi.com/splitting-up-a-git-repo/

— Reply to this email directly or view it on GitHub https://github.com/bhmm/bhmm-force-spectroscopy-manuscript/issues/40#issuecomment-119814675.

Prof. Dr. Frank Noe Head of Computational Molecular Biology group Freie Universitaet Berlin

Phone: (+49) (0)30 838 75354 Web: research.franknoe.de

Mail: Arnimallee 6, 14195 Berlin, Germany

jchodera commented 9 years ago

I think so. I think I can expunge the bhmm/ history from that repo so we can keep them separate.

I agree that we should make a new bhmm release and the import the bhmm package into the force spectroscopy code, but I would like to put the force spectroscopy specific code into a module to make it easier for people to perform their own analyses and avoid code duplication. I can do the tinkering to make this work.

I struggled go try to get Travis working again for the bhmm repo, so there may still be things we need to do to get everything working again.

franknoe commented 9 years ago

Am 09/07/15 um 12:24 schrieb John Chodera:

I think so. I think I can expunge the bhmm/ history from that repo so we can keep them separate.

I agree that we should make a new bhmm release Is it easy for you to trigger a conda release? Actually I need to learn how to do this, but maybe it's just a click for you... and the import the bhmm package into the force spectroscopy code, but I would like to put the force spectroscopy specific code into a module to make it easier for people to perform their own analyses and avoid code duplication. I can do the tinkering to make this work. Yes, exactly, that's what I thought too. Maybe later we can have a force-spectroscopy package which only contains the examples and the force-spectroscopy specific code - I guess you don't want to distribute the paper and matlab sources as well in the same repo, or do you?

I struggled go try to get Travis working again for the bhmm repo, so there may still be things we need to do to get everything working again. Unfortunately I don't know anything about this, but if e.g. Martin can help let us know.

— Reply to this email directly or view it on GitHub https://github.com/bhmm/bhmm-force-spectroscopy-manuscript/issues/40#issuecomment-119900882.

Prof. Dr. Frank Noe Head of Computational Molecular Biology group Freie Universitaet Berlin

Phone: (+49) (0)30 838 75354 Web: research.franknoe.de

Mail: Arnimallee 6, 14195 Berlin, Germany

jchodera commented 9 years ago

Is it easy for you to trigger a conda release? Actually I need to learn how to do this, but maybe it's just a click for you

Conda release package builds occur when PRs are committed into the omnia-md/conda-recipes repo. You just need to create a PR to do this. If we want to build from the same release, we have to increment the build number.

If we want to cut a new release, we first have to make a GitHub release, then update the conda recipe release information in a PR. Merging the PR (after travis tests to ensure we didn't break package resolution) will cause a package to be built and deployed to binstar. (Make sure not to delete the old packages! We need them for historical reproducibility!)

Unfortunately I don't know anything about this, but if e.g. Martin can help let us know.

Still struggling. Not sure if travis is confused by the renaming of repos.

jchodera commented 9 years ago

Hm. It looks like we somehow lost the release history of bhmm in the split: https://github.com/bhmm/bhmm/releases

I'm wondering if I should go back and try to re-do this to try to preserve the release history as well, or if it's OK if I just cut a new release?

franknoe commented 9 years ago

I don't think it matters at this point. No problem for me if you loose the release history.

Am 09/07/15 um 13:08 schrieb John Chodera:

Hm. It looks like we somehow lost the release history of bhmm in the split: https://github.com/bhmm/bhmm/releases

I'm wondering if I should go back and try to re-do this to try to preserve the release history as well, or if it's OK if I just cut a new release?

— Reply to this email directly or view it on GitHub https://github.com/bhmm/bhmm-force-spectroscopy-manuscript/issues/40#issuecomment-119912757.

Prof. Dr. Frank Noe Head of Computational Molecular Biology group Freie Universitaet Berlin

Phone: (+49) (0)30 838 75354 Web: research.franknoe.de

Mail: Arnimallee 6, 14195 Berlin, Germany

franknoe commented 9 years ago

Am 09/07/15 um 13:06 schrieb John Chodera:

Is it easy for you to trigger a conda release? Actually I need to
learn
how to do this, but maybe it's just a click for you
Conda release package builds occur when PRs are committed into the |omnia-md/conda-recipes| https://github.com/omnia-md/conda-recipes repo. You just need to create a PR to do this. If we want to build from the same release, we have to increment the build number https://github.com/omnia-md/conda-recipes/blob/master/bhmm/meta.yaml#L11.

If we want to cut a new release, we first have to make a GitHub release, then update the conda recipe release information https://github.com/omnia-md/conda-recipes/blob/master/bhmm/meta.yaml#L1-L7 in a PR. Merging the PR (after travis tests to ensure we didn't break package resolution) will cause a package to be built and deployed to binstar. (Make sure not to delete the old packages! We need them for historical reproducibility!)

awesome. This is really easy :)
Unfortunately I don't know anything about this, but if e.g. Martin
can help let us know.
Still struggling. Not sure if travis is confused by the renaming of repos.

— Reply to this email directly or view it on GitHub https://github.com/bhmm/bhmm-force-spectroscopy-manuscript/issues/40#issuecomment-119911572.

Prof. Dr. Frank Noe Head of Computational Molecular Biology group Freie Universitaet Berlin

Phone: (+49) (0)30 838 75354 Web: research.franknoe.de

Mail: Arnimallee 6, 14195 Berlin, Germany

jchodera commented 9 years ago

OK, travis build is working now (though we will have to see if it passes) and conda recipe for 0.4.0 created. Will update soon.

We have to move issues over to the bhmm repo manually. Can you guys carry over any important issues with TODOs not related to force spectroscopy?

franknoe commented 9 years ago

Yes, sure. Thanks a lot for your help.

Am 09/07/15 um 14:17 schrieb John Chodera:

OK, travis build is working now (though we will have to see if it passes) and conda recipe for 0.4.0 created. Will update soon.

We have to move issues over to the |bhmm| repo manually. Can you guys carry over any important issues with TODOs not related to force spectroscopy?

— Reply to this email directly or view it on GitHub https://github.com/bhmm/bhmm-force-spectroscopy-manuscript/issues/40#issuecomment-119936939.

Prof. Dr. Frank Noe Head of Computational Molecular Biology group Freie Universitaet Berlin

Phone: (+49) (0)30 838 75354 Web: research.franknoe.de

bhmm / legacy-bhmm-force-spectroscopy-manuscript