search

glotzerlab / signac-flow

Workflow management for signac-managed data spaces.
https://signac.io/
BSD 3-Clause "New" or "Revised" License
48 stars 37 forks source link

Proposal: A library for project templates #9

Closed csadorf closed 6 years ago

csadorf commented 7 years ago

Original report by Carl Simon Adorf (Bitbucket: csadorf, GitHub: csadorf).

Proposal for a template library

Right now there is one maintained project template that serves to provide some generic example and guidance for users on how to setup a project. For that purpose it needs to remain rather abstract and simple.

With a library of different templates we could cater much better towards the specific needs of individual workflows. With that in mind we would need some way to manage and distribute these templates.

Library Format

I propose to use the anaconda style, where a git repo may contain one or multiple templates (recipes). That would mean a repo would have for example the following directory structure:

#!bash
flow-template-library:
./minimal/
./minimal/project.py
./hoomd/
./hoomd/project.py
./hoomd/operations.py
./gromacs/
./gromacs/project.py
./gromacs/operations.py
...

Installation

With the above format we would have the following installation methods:

Manual Installation

The user just clones the whole repo and copies the files into their project.

#!bash
git clone git@bitbucket.org:glotzer/flow-template-library.git
cp flow-template-library/hoomd/* my_hoomd_project/

Guided Installation

The user would use the flow interface, for example like this:

#!bash
git clone git@bitbucket.org:glotzer/flow-template-library.git
cd my_hoomd_project
flow init --src=../flow-template-library -t hoomd

or directly with the remote source:

#!bash
cd my_hoomd_project
flow init --src=git@bitbucket.org:glotzer/flow-template-library.git -t hoomd

It would be possible to configure a repository like a channel:

#!bash
signac config add flow.templates.url git@bitbucket.org:glotzer/flow-template-library.git

cd my_hoomd_project
flow init -t hoomd

Versioning

There should be standardized way to indicate what flow version a template has been written/ tested for. For example, by adding the following line to the project.py file:

#!python
flow.required_version("0.5.0")

Integrated Testing

Templates should have simple routines for the initialization and execution of simple operations. Not all operations need to be testable.

csadorf commented 7 years ago

Original comment by Carl Simon Adorf (Bitbucket: csadorf, GitHub: csadorf).

@PaulDodd @vyasr @bdice I would appreciate your input.

csadorf commented 7 years ago

Original comment by Vyas Ramasubramani (Bitbucket: vramasub, GitHub: vyasr).

I like this idea. I imagine that this is something we could populate easily from people's existing projects. Also, it would be useful to have an easy way to get the list of available templates. We'd have to do some thinking on the implementation side, but at this level the proposal sounds good to me.

csadorf commented 7 years ago

Original comment by Bradley Dice (Bitbucket: bdice, GitHub: bdice).

This sounds great. I agree that a set of templates would be more usable than the single template that already exists. There are features that I have yet to use in signac-flow because I've kept so closely to the example template. Seeing additional examples where these features are used would encourage users to borrow some features from multiple templates to better suit their needs.

csadorf commented 7 years ago

Original comment by Carl Simon Adorf (Bitbucket: csadorf, GitHub: csadorf).

I created a first shot at the template library and implemented a flow-init script. Please feel free to test it and let me know if you have any questions.

csadorf commented 7 years ago

Original comment by Vyas Ramasubramani (Bitbucket: vramasub, GitHub: vyasr).

Some thoughts from my initial test:

csadorf commented 7 years ago

Original comment by Carl Simon Adorf (Bitbucket: csadorf, GitHub: csadorf).

@vyasr , thank you for your feedback.

  1. Yes, that was an oversight. Fixed now.
  2. Yes, we should carefully assess which extra dependencies are actually needed. Using subprocess for git might not be platform independent though. One option is to make it a soft dependency for when you actually want to clone from git repos. It is probably better to use a download path pointing at the corresponding tar archive anyways: https://bitbucket.org/csadorf/flow-project-templates/get/master.zip. That would also allow us more easily clone specific branches without extra API arguments. This should work with private repos, but I have not thoroughly tested that.
  3. Let me know if you were able to resolve those issues or just copy & paste the error output here.

In general we need to be careful not to reimplement git here...

csadorf commented 7 years ago

Original comment by Carl Simon Adorf (Bitbucket: csadorf, GitHub: csadorf).

While prototyping and iterating over this proposal, I realized that what we really need is a tool that just copies a given project without the data. Whether that is some kind of existing project that we want to clone and modify or whether that is a complete or incomplete example does not really matter.

That said, I think that providing a specific tool for copying source code is not critical for the main goal of this proposal. We should therefore focus on setting up the library as soon as possible and worry about the tool later.

Unless there are any objections, I'm going to do exactly that and migrate everything "clone/copy tool" related to a new issue.

csadorf commented 7 years ago

Original comment by Carl Simon Adorf (Bitbucket: csadorf, GitHub: csadorf).

I put all the minimal examples into the example-projects branch into the signac-examples repository.

This is work in progress and I would like to have at least one HOOMD-blue example before announcing this publicly.

csadorf commented 6 years ago

Original comment by Carl Simon Adorf (Bitbucket: csadorf, GitHub: csadorf).

The example library can be found at:

http://bitbucket.org/glotzer/signac-examples