Reproducibility section for apps

[samumantha]

How about adding a reproducibility section for Apps (where it makes sense)?

This would include how the app was installed by CSC (some package manager or built from source, using tykky, via container, etc) and then also link to environment.yml, Dockerfiles or Singularity definition files, which could be collected somewhere public.

I understand that only in some cases this helps the users, but I am also thinking here about for example other HPC centers.

environment, requirements, dockerfiles and definition files could for example be collected in a github repository under csc-training together with sharing course related computing environment setups (internally suggested in customer-training:

Collecting course environment information

Many of us create some kind of environment for our courses (eg own module on Puhti or some custom image for CSC Notebooks):
Shouldn't we collect information on them in some public space?

This would serve at least 3 purposes:

    We/Our colleague know next year how we set up that course environment
    The Dockerfiles, definition and module files would be version controlled
    Customers that want to set up their own environment would have examples on how it's done and could copy and adapt our environments.

Maybe there could be a new repository under https://github.com/csc-training ?

[samumantha]

Totally forgot about it, but this already exists: https://github.com/CSCfi/singularity-recipes

[samumantha]

But according to the name, conda and pip files would not fit here. :thinking:

[mvsjober]

This kind of exists already, it should just be maintained better - used more frequently by CSC'ers: http://env-docs.rahtiapp.fi/

[samumantha]

Yes, but only for CSCers. Now thinking about users. It would need to be thought what kind of information they could find useful. But just linking to the recipe in https://github.com/CSCfi/singularity-recipes would already be a first step

[mvsjober]

I think the stuff in env-docs is not so far from what users would need? Seems a bit of duplication of effort to make yet another almost similar page? Or is the page behind the firewall?

For example the Python data page: http://env-docs.rahtiapp.fi/apps/python-data/ - it could just have links to those yaml files a bit better, and anyone could use them. (I also realise that page is a bit out of date...)

[samumantha]

I agree that it would be better to use what we have.

But still think that there is too much information (partly of things that users cannot even do without access to login13) there. I'd also NOT create yet another page, but put some short description on the applications docs page. "This application was done using tykky with . You can adapt your own module by following instructions on ." For example.

If users can see the env guide files, then of course the linked environment.yml file could be linked from there. :thinking:

[samumantha]

But it seems external cannot see anything of env-guide?

[mvsjober]

OK, apparently it is available to CSC internal network only. This is maybe something to discuss inside CSC, should it be public - or should we have a separate public one... without making too much effort and complexity to keep everything updated.

[mvsjober]

... even now the csc-env-guide is not being updated very regularly, so adding another one that also needs to be updated might be too much. But maybe making csc-env-guide public and rewriting it to be more user friendly could be one option?

[samumantha]

That is true. @Nortamo suggested a meeting in august/september with relevant people. So let's get back to it then :)

Could also think then if singularity-recipes needs to be separate anymore if envguide would be public.

[mvsjober]

Yes, ideally there would be one place that could serve all needs. I already have trouble remembering to put my singularity recipies in BOTH csc-env and singularity-recipes...

OK, let's have a meeting then! :+1:

[samumantha]

Some minimal install info per app could look like the ones in GIS applications here: https://github.com/CSCfi/csc-user-guide/pull/1723